Data recording

A data point only knows its current value. Anyone wanting to know how warm it was last night or how much electricity was consumed last week needs an adapter that records data. Three are available, and the decision should be made early because switching later requires extra work.

Not to be confused with the two internal databases for objects and states. Those maintain the current state of the system and are handled under Redis . This concerns the history.

A data point only knows its current value. Anyone needing the historical data can activate recording and choose where it should be recorded.

What the course is and what it isn't

A state has exactly one value: its current one. When a new one occurs, the old one disappears. The state database is a piece of paper that always contains the latest status, not a booklet in which the pages remain.

A recording adapter is attached alongside: It listens to the data points you specify and writes down each new value along with a timestamp. Only then does a trend emerge that can be plotted in a graph.

This leads to four things that regularly cause surprises:

• Recording begins from the moment the device is switched on. There is no retrospective recording, not even from yesterday.
• The data is recorded at each individual point. The system itself is not switched on; instead, each individual value that you want to view later is recorded.
• The history is not stored in the internal databases. Objects and states represent the current state; the history is located elsewhere, see Redis .
• The ioBroker backup does not automatically include it. It backs up objects, states, and configurations. The recorded values are a separate item in BackItUp .

In the system settings, under Default History , you can see which instance is suggested when a dialog or chart asks for the source. This is a default setting, not a recorded history: the source will still be selected per data point.

Which adapter

The history function stores data in two stages: the values are first stored in RAM and only written to files when a predefined threshold is reached. This protects the card, but also means that the most recently collected values are lost in the event of a hard power outage.

The files are located in a folder below/opt/iobroker/iobroker-data , without providing their own information inhistory , and within that, one subfolder for each day. An absolute path like/mnt/history This is also possible, for example, on an attached storage device. The location of the data is important for backup purposes; see below.

For starters, history . It doesn't need a second service, and switching to InfluxDB is possible later, see below.

Recording means writing, and writing uses up an SD card. Anyone who regularly records large amounts of data should not do so on an SD card, but rather on an SSD or in a database on another computer.

Turn on

The adapter is installed, an instance is created, and then a decision is made for each data point whether it is recorded. This is done in the Objects tab via the gear icon at the end of the line.

The instance configuration contains the default settings that apply to each newly activated data point. These settings can be overridden at the data point itself.

Recording begins from the moment the device is switched on . There is no retrospective recording.

The step-by-step process is described under "Recording Values" .

The settings that matter

The complete description of all fields can be found in the documentation of the respective adapter: history , influxdb , sql .

What should be recorded and what shouldn't

The most common reason for a system becoming sluggish after a year is not insufficient computing power, but that someone has written everything down because they might need it someday.

• Useful : Temperatures, consumption, fill levels, switching states, where you want to be able to read later when something happened.
• Not useful : internal data points of the adapters, counters that already contain a history, and anything you will never look at.

Switching from the history adapter to a database

The history adapter includes scripts for this purpose, which are located in the directory/opt/iobroker/node_modules/iobroker.history/converter lie and withnode to be called up. The recommended procedure:

1. Set up and run the new target. Configure the new adapter and enable the same data points there. Verify that the values are being received. During this time, data will be written twice: once to the history and once to the new target. This is intentional and the reason why nothing is lost during the migration.

2. Analyze the existing data. The analysis script determines which data already exists in the target and saves the results in JSON files. It is called in the converter directory:

cd /opt/iobroker/node_modules/iobroker.history/converter
node analyzeinflux.js influxdb.0 info --deepAnalyze

For an SQL database, accordingly:

node analyzesql.js sql.0 info

The first parameter is the target instance, the second is the protocol level.--deepAnalyze Additionally, it records which values already exist for each day. Without this information, only the earliest value is determined. The difference is relevant if there are already gaps in the target data that need to be filled.

3. Stop and convert the history adapter.

node history2db.js

The script reads the JSON files from step 2 and only transfers what is not already present. It then continues writing the files, so a second run usually doesn't create duplicates. It can also be called without prior analysis; in this case, a start date must be specified as a parameter, and everything before that date will be converted. This process can take a long time.

4. Only then should you clean up. Once the values in the target are complete and the logs confirm this: delete the history data and deactivate the adapter.

Create a backup before migrating. Only delete the old data once the new data is demonstrably complete, and to verify this, check a graph that goes back a long way.

The complete parameter list for the three scripts can be found in the documentation for the history adapter .

What happens during a fuse

There's a common misconception here that can be costly: an ioBroker backup does not contain the recorded values. It saves objects, states, and the file storage—that is, the current state of the system. The history is stored elsewhere, and this applies to all three adapters.

• In the history section, the files are indeed located below...iobroker-data However, these are not part of the ioBroker backup. With an absolute path, they are located outside the backup anyway.
• With InfluxDB and SQL, the data is stored in a separate database, often even on a different computer.

BackItUp therefore lists them as separate backup types , created in addition to the ioBroker backup: History Data , InfluxDB , MySQL , PostgreSQL , and SQLite3 . These switches are not set by default.

Anyone recording data must also activate the appropriate switch on the backup adapter. Otherwise, after a restore, a fully configured system will be present in which all diagrams are empty.

View

Recorded values are evaluated as a diagram, usually withecharts The route there is shown under Diagrams .