js controller 2.0

Hello ioBroker community,

In line with reaching the 30,000 active installations mark a few days ago, we would now like to introduce the new js-controller 2.0 to you. This is now available in the Latest Repository and on npm.

In an internal test and a very extensive beta test in the community, we have already tested this major update of the js controller very intensively. Big thanks to @Arteck, @sigi234, @SBorg, @opossum, @e-s, @e-i-k-e, @Yetiberg, @Jan1, @Einstein67, @Dr. Bacterius and many more. That was a great collaboration!

Above all, some fundamental changes have been made "under the hood" that justify the change to a new main release number. More on that below.

The js-controller 2.0 is generally compatible with all existing ioBroker systems. It can be upgraded to 2.0 version from any previous version. Only the Node.js version must be upgraded to at least 8.x, better still 10.x, before the update! However, it is still a bit too early for Node.js 12, as some adapters are still not compatible.

Furthermore, the ioBroke own file area (usually under /iobroker-data/files/...) is now treated more strictly and files stored/copied there manually or by script (fs.write) may be .can no longer be displayed in visualizations! Scripts must be adapted (use of writeFile) or the files must be stored in officially defined adapter base directories (e.g. vis.0, iqontrol.meta, etc.). It is best to use the official uploader via Vis or iqontrol so that these files are correctly registered. This change was also implemented to increase security! The positive side effect is that the files end up in the backup, which was not the case before!

Installation

Before installation As with any update of this kind: Please make a backup! iobroker backup or copying the iobroker-data directory is sufficient in itself in case of doubt (stop ioBroker beforehand, of course). Please don't just copy the node_modules directory, otherwise symbolic links may break, leading to bigger problems afterwards.

Necessary adapter updates

The following adapters must be updated to the minimum version numbers mentioned, otherwise they will not work with the js-controller 2.0. It is best to carry out these updates beforehand, because all the versions mentioned also work with the old js-controller versions.

  • simple-api 2.1.2 or higher
  • email 1.0.5 or higher
  • pushover 1.1.1 or higher
  • hue 1.2.4 or higher
  • node-red 1.10.1 or higher
  • vis 1.2.1 or higher
  • iqontrol 0.2.6 or higher
  • socketio 2.1.2 or higher
  • radar2 1.0.9 (GitHub version 1.2.0 needs to be adjusted manually, see FAQ!)
  • broadlink2 (see FAQ)

ATTENTION: SLAVE SYSTEMS FIRST!

In a multi-host system, when updating to version 2.0, it is very important to update the slave systems first. The master will be updated last!

If this order is not followed, the slave systems cannot connect to the master and the update must be carried out manually (see FAQ post for details).

Windows

On systems set up with the new Windows Installer, the js controller must not be updated with npm. There will be a new version of the Windows Installer that will allow the js controller to be updated with just a few mouse clicks. We'll update this in this thread.

For all "old manual" installations, the same ioBroker-specific iobroker upgrade self command applies as usual.

Linux

As usual, the update is carried out via iobroker upgrade self.

In the event of errors: If errors occur during the Linux installation due to missing access rights, it is best to use the installation fixer and repeat the installation.

curl -sL https://iobroker.net/fix.sh | bash -

If there are still errors after this, please try the installation again using sudo -H -u iobroker npm install iobroker.js-controller. Please report such cases here in the thread.

After installation

After the installation, start ioBroker again (e.g. using iobroker start).

If everything works, you won't notice any difference apart from the higher version number in the host view in the admin. Everything continues to work as before. All adapter instances start and work. If that's the case, everything worked. The big changes are all hidden "under the hood".

You will find information further down in this text about what all the new features bring you. New functions as a basis for further developments were carefully integrated and some existing problems were specifically corrected.

With iobroker help a list of possible command line commands is displayed, which has become a few commands longer with version 2.0.

What has changed, what to watch/test in particular?

One of the bigger changes is that ioBroker's own States and Objects databases have been completely rewritten. A TCP-based and Redis-compatible protocol is now used for communication in the ioBroker system. In particular, "Reconnection from DB" errors should now be a thing of the past. Based on these changes, we are planning some interesting innovations for the future.

As a result of this change, the logs sometimes say "connected to redis" even though you are not using Redis at all. However, you can still tell from the port if it is ioBroker's own database (usually ports 9000 and 9001). First tests have shown that the CPU load of the adapter processes and the js-controller is lower than in the old version, since the new protocol is much leaner. It's still more flexible and robust - as long as the network connection doesn't break down. But even in such a case, an automatic reconnect should take place and, if the interruption is not too long, all changes from the time without connections should be sent again. So if you've been plagued by "Reconnect to DB" messages and effects in the past, your report is of great interest to us.

Also the infamous "Error 7", which appears in the log when an adapter process is already running, but a new one is to be started, has been improved. If a new process is to be started, any processes that are still running should now automatically terminate themselves and generate a one-off message in the log.

As said before, many changes took place behind the scenes. Here's a spoiler summary for those interested:

2.0 - Release Bella

Breaking changes

Minimum requirement for js-controller 2.0 is node.js 8.x Files in iobroker-data/files are only supported in officially registered directories New user features

  • Add Compact Mode and compact groups (Technology Preview)
  • Add build-in Alias handling for Objects/States (Technology Preview)
  • Add support to also use Redis for Objects and Files
  • Add Redis sentinel support
  • Allow dynamic change of Loglevel for adapter instance and js-controller hosts processes
  • Add optional migration for State and/or Objects values when using setup custom
  • Add monitoring for event-Loop-Lag as host and adapter objects
  • Add possibility to validate backup files
  • Support command "iobroker logs"
  • Support command "cert create"
  • Remember installation location for reinstallations
  • Use Remembered installation location for automatic adapter installs
  • Log Process-ID for all adapter log messages
  • Enhance some CLI commands like iobroker status
  • New adapter developer features

Streamline redis vs file States handling which was different also before controller 2.0:

  • not set states will always return null now
  • States will set to null completely (not only value) when they expire
  • States will also be published to onChanged handlers when states are in Redis
  • Add adapter.supportsFeature('NAME') method to check if a certain feature exists. #244
  • Ability to define secured objects in io-pack access only via own adapter and admin. #287
  • Added getObjectView and getObjectViewAsync on adapter object
  • Added getObjectList and getObjectListAsync on adapter object
  • Allow the deletion of multiple objects with wildcard
  • setObject/setObjectNotExists now also sets default value of state after object creation
  • Allow getPort to check for the port optionally on a certain host/IP

Further changes

  • Rewrite InMem databases (States & Objects) to TCP (redis compatible) protocol and deprecate socket.io version; will be removed approx. in v2.1
  • Add adapter handling to prevent "error 7" (adapters will stop themself as soon as PID is not as expected)
  • Upgrade all dependencies
  • Don't chmod 777 after controller upgrade
  • Refactoring of many CLI commands
  • Add possibility to return zip file as a link and not as base64
  • Standardize error codes
  • Root should always npm install with --unsafe-perm
  • Enable gzip to read repositories
  • Read hash of sources.json online before downloading the whole file
  • Add some information about user-agent
  • Verify the version of node.js by start of the instance
  • Hide cmd window on windows
  • Include certificate creation in setup first
  • Suppress warning by npm install
  • Allow optional dependencies being installed
  • Optimize setup custom command and add more user guidance
  • Add Feature overview to README
  • Forward upload console outputs from slave to master
  • Make sure to upload and upgrade all relevant objects on installations and updates of adapters
  • Always upgrade instance objects after successful installs or upgrades
  • Optimize adapter start processes, especially when combined with needed automatic installations of adapters
  • After 2 installation tries with "last-installedFrom" use the installedVersion field to try to install from npm
  • Hhosts now ignore object changes when the affected instances is still in installQueue
  • Code refactoring and optimizations in various places
  • Randomize Certificate Serial numbers
  • delay parallel start of scheduled instances to prevent system overload scenarios (same rules asd for adapter start, basically 4s delay)

Bug fixes

  • Log scheduled restarts as info only (fixes #315)
  • Fixed #340 to maintain restartSchedule on updates
  • Fixed a bug where it was possible to set "ack" to any value via cli
  • Enable ESLint and fix most issues
  • Optimize multi host upload
  • Restart stopped adapters at the end of the upload and not before to make sure to not have two adapters restarts on upgrade cases
  • Enhance checks for failed installations in cli and controller
  • Also update adapter instance statistical objects when no instanceObejcts are defined
  • ".alive" state values are only checked on adapter start if ack=true to allow to start a process if not running
  • Fix for mutlihost detection
  • Fix backup of states
  • Make sure also VIS global CSS is included in backup and restored
  • Many more fixes in various places

More details about the changes and bug fixes can be found in the changelog.

How to report bugs?

If you are unsure whether there is an error, it is best to describe the problem here in the thread. This way we can all try to understand the problem and narrow it down if necessary.

As soon as an error occurs that ends in an error message or a crash with error details in the log or on the command line, it is best to open a GitHub issue directly in the js-controller project and also post it here in the thread. The more detailed the information in the issue is (exact error messages/logs, information on the DB constellation used (file(file, file/redis, redis/redis ...), information on the OS and Node.js environment and precise steps for reproducing the problem), the faster we can isolate and fix errors.

Overview of some of the new features

1.Compact Mode and Compact Groups

One of the great things about ioBroker is that each adapter runs as its own process. This makes the system very stable - problems only affect one adapter and not the entire system. On the other hand, this approach also requires a bit more RAM. For systems with little available RAM (e.g. Raspi Nano or Raspi 1 with 512MB RAM), which are often used as slave systems, the number of adapters is thus limited.

The compact mode solves this problem by running several adapters together in one process and thus the RAM requirement is significantly lower (about 20-30MB are saved per adapter instance). However, this is at the expense of stability, since a faulty adapter also affects all other adapters in the same process and these may also restart themselves.

To spread the risk somewhat, adapter instances can be divided into several groups. Each group starts its own process, in which all instances of this group are then executed. Group 0 is special. Being a member here means that the adapter in question is running in the main js-controller process. This results in the greatest RAM savings - but also the greatest risk, since a faulty adapter can negatively affect the js-controller. By default, instances run in Group 1 when Compact mode is activated for the corresponding instance.

Whether an adapter supports the compact mode depends on the respective adapter. This information will be included in the adapter list in the future. Currently, only adapters that are running as daemons are also started in compact mode (i.e. no scheduled adapters). Even if the adapter generally supports compact mode, use must be activated individually for each instance!

There is currently no support in the admin for configuring the compact mode. The configuration is carried out by calling the command line. The most important commands are:

iobroker compact enable to generally activate the Compact mode for the current js-controller host. ioBroker must be restarted for the change to take effect.

iobroker list instances now also displays the status of the compact mode of the adapter instances.

iobroker compact <adaptername>.<instance> status indicates the compact mode status of the instance.

iobroker compact <adaptername>.<instance> enable 1 activates execution in Compact mode in group "1". Only the adapter will be restarted. This configuration can be done while the ioBroker is running.

If there are problems (for example, an adapter does not run clean or "hang" while stopping), please open an issue with the adapter. Otherwise, please post here in the thread, so we see what it is.

2. Installation source of adapters is saved

ioBroker adapters are typically installed from npm's latest or stable repository. If an adapter is moved to another host or the system needs to be reinstalled, npm will attempt to reinstall the same version. Normally that works too. However, if an adapter was installed from GitHub as a test, that level or version may not be available on npm. This means that the same version cannot be reinstalled automatically. This is now changing.

For all new adapter installations after the update, ioBroker remembers the exact GitHub status of a custom installation and can then reinstall this status.

The adapter installation itself, the upload and the like have also been revised and optimized. Fixed some special cases where parts of the upload logic were not executed correctly. Again, your tests will help.

3. Redis support now also for objects and files

By default, objects and states are managed in an ioBroker proprietary storage database and stored in JSON files. With this proprietary solution, no additional software is required.

For some time it has been possible to alternatively store states in an optimized Redis database. From a certain number of status changes per second, the use of Redis can reduce the overall system load or spread it over several systems. However, Redis also involves additional work, since this software needs to be installed, managed and backed up so that there is no data loss during updates or in the event of a problem.

With js-controller 2.0, ioBroker now also allows objects and files that are currently stored in the file system on the master system to be managed in the Redis database.

Important: Above all, moving files to the database can lead to it becoming quite large (several hundred MB). Since Redis always keeps all data in RAM, this option is only suitable for systems that have enough RAM resources available. The CPU load will also be higher with a redis/redis system because data has to be processed differently.

The relocation of files into the database leads to significantly greater flexibility. Due to this change, there is virtually no more local data in the file system. All in all, the Redis database becomes the central data storage of the ioBroker system, as it contains all data. All js controllers and adapters then connect to this central database.

With the js-controller 2.x, even a Redis-Sentinel installation can be used to increase system availability (quasi a Redis-HA cluster). This is an initial basis for providing a highly available ioBroker system that can compensate for failures of individual server components. However, you should only test this option if you know what you are doing 😉 More on that in later updates.

With the introduction of the various storage options for files, objects and states, the iobroker setup custom command has been revised. For one, it displays more information. In addition, he is now also able to migrate the data in all directions when there is a change in data management. A change is possible at any time.

4. "Alias" feature

One of the advantages of ioBroker is that there are a lot of adapters. However, it has turned out that each adapter implements individual structures regarding the storage of the provided data points, e.g. depending on the connected systems. This entails certain challenges, especially with your own scripts, but also with visualization and the cloud/iot adapters. When exchanging devices between different manufacturers, you sometimes have to adapt scripts due to changes in the data points. Visualization adapter and the iot adapter try to recognize the type of devices based on the roles of data points, for example, in order to display them correctly or report them to Amazon or Google. The adapters are sometimes lost because certain adapters have information about roles and the like. cannot deliver at all - especially MQTT, modbus, etc. are affected here.

The alias feature, which is anchored directly in the js-controller, provides the new namespace "alias.0" for objects. The feature allows to create devices with a stable structure and clean roles. This is now first done manually, later also e.g. using the upcoming "Devices" adapter, which is currently under development.

This feature also does not have full support in the admin yet, but that will come.

After defining the alias object, the new area common.alias can be used to define the ID of the source object in the respective adapter. From then on all data will be synchronized in both directions between the objects. In addition, interestingly, a read and write function can be defined to perform simple conversions (e.g., Wh <--> kWh).

We have published further (technical) details under Alias Information in the js-controller README.