LogAndStream Firmware User Manual

Preview: this manual is being updated to cover Shimmer3R. Notes marked TODO are open questions our team is still confirming.

1. Introduction

Shimmer3R: This manual covers LogAndStream on both Shimmer3 and Shimmer3R. The two are built from one shared code base and behave the same way; where Shimmer3R differs, a Shimmer3R: note says so. The Shimmer3R firmware is released as LogAndStream for Shimmer3R (LogAndStream_Shimmer3R_vx.yy.zzz). Its version numbers are independent of the Shimmer3 firmware’s and cannot be compared with them. A Shimmer3R reports hardware version 10, against 3 for Shimmer3; both report firmware identifier 3 (LogAndStream).

This document is an accompaniment to the LogAndStream Firmware for Shimmer3. No previous development experience is required.

LogAndStream firmware is a complete data recording solution that merges features from our previous Shimmer3 firmware releases. The LogAndStream firmware facilitates logging of data from a Shimmer3 to the on‑board SD card while also providing the ability to simultaneously stream data over wireless connection to a Bluetooth-enabled PC. The firmware allows for full user configuration of the Shimmer3 using a configuration file stored on the SD card, or over Bluetooth using the provided ConsensysBASIC PC application.

It is recommended that the LogAndStream firmware be used in conjunction with the ConsensysBASIC or ConsensysPRO software applications (available for download from the Shimmer website¹).

Whenever the warning symbol appears throughout this document, it denotes a new, modified or deprecated feature in v0.4.0 (or later).

2. Changes in latest release

Note: This chapter describes LogAndStream v0.11.x for Shimmer3. Release notes for every later release are published with the firmware: https://github.com/ShimmerResearch/shimmer3-firmware/releases for Shimmer3 and https://github.com/ShimmerResearch/shimmer3r-firmware/releases for Shimmer3R.

TODO(S3R): Decide whether to keep this chapter, or replace it with the links above.

This release is a major update to v0.10.0 with the following updates/improvements:

2.1. Bug Fixes

Sensing stopped when docked

This release fixes a bug in SetupDock() whereby the shimmer unit is forced to stop sensing if “docked”.

2.2. New Features

SR47-4 (New ExG build) support

New SR47-4 (ExG) devices now supported. ADS1292R clock lines are reconfigured for this device such that both ADS chips share the same clock line. Interrupt for Chip 2 is disabled such that both chips are serviced by chip 1’s data ready pin interrupt routine alone. Default configuration bytes now also cater for SR47-4 (and newer) Shimmer ExG devices. Checks EEPROM info and applies ADS clock chip configuration bit accordingly.

An additional 1 second time-out is now applied in firmware- see page 63 of ADS1292R datasheet for more info.

3. Scope of this User Manual

The purpose of this User Manual is to guide the user through the features of the LogAndStream firmware and to provide the required instructions to configure the data logging options, configure the data streaming options and to parse the recorded and received data. The User Manual does not provide an extensive explanation of the source code for the firmware.

4. Pre-Requisites

The LogAndStream firmware can be used with any Shimmer3 device that has a connected microSD card. For Shimmer3 compatibility, the chosen microSD card must be less than 32 GB in capacity, not be an XC (eXtended Capacity) card and it must support 1‑bit SPI mode. Please refer to the Shimmer3 MicroSD Media Guide on our website² for more information.

A Shimmer Dock or Consensys Base is required to allow access the SD card on the Shimmer from the PC for configuration of logging preferences and data transfer. Please note that legacy (black) Shimmer docks, sold with Shimmer2r and earlier hardware, are not suitable for this purpose.

Note: Please see LogAndStream_Shimmer3_v0.11.0 for full release notes and accompanying firmware.

For a wireless connection to the Shimmer3, a Bluetooth enabled PC and a software application like ConsensysBASIC or ConsensysPRO is required to interface with Shimmer3 and receive the streamed data. The ConsensysBASIC and ConsensysPRO applications allow a user to fully configure the Shimmer3 whilst providing the ability to save streamed data locally to the PC in real-time.

For more information on the Shimmer3 platform, please refer to the Shimmer User Manual which is available for download at www.shimmersensing.com.

5. Installation

Shimmer3R: Consensys installs LogAndStream for Shimmer3R over USB, through the Shimmer3R’s built-in USB bootloader; see Programming a Shimmer in the Shimmer User Manual.

Install the LogAndStream firmware v0.11.0 image (LogAndStream_Shimmer3_v0.11.0.txt) onto a Shimmer3 device, using the ConsensysBASIC or ConsensysPRO software applications, available on our website1.

6. Firmware Features

As stated previously, the LogAndStream firmware provides a seamless integration of logging Shimmer3 data to an on-board microSD card and the ability to stream Shimmer3 data to “host” device via Bluetooth. The LogAndStream firmware has been designed such that a user can still utilise the Shimmer3 in a purely Bluetooth streaming or SD-logging mode if they desire. What sets the LogAndStream firmware apart from previous firmware releases is its ability to perform both SD-logging and Bluetooth streaming operations simultaneously.

A Shimmer3 programmed with LogAndStream firmware can be in one of five states: Idle, BT Connected, BT Streaming, BT Streaming + SD Logging or SD Logging – as shown in Figure 6‑1. The active states form two operational branches from which the user can choose to operate the Shimmer3 device – one initiated by a Bluetooth connection (blue shaded area in Figure 6‑1) and the other based on SD Logging operation (orange shaded area in Figure 6‑1).

The key feature of the LogAndStream firmware can be viewed at the intersection of the two active branches (green shaded area in Figure 6‑1). From a “host” application, the user can select to stream Shimmer3 sensor data over a Bluetooth connection at the same time as saving it locally to the on‑board microSD card. An instance where this is of key benefit is the case where, in the process of data collection, a Shimmer3 might travel out of range from the “host” device. In this circumstance, the Shimmer3 will continue to save data to the microSD card such that there will be no break in the data recording. Similarly, during normal SD logging operation, continuous Bluetooth streaming is not always desirable or required. By combining Bluetooth support with SD logging capability, the new LogAndStream firmware now facilitates the ability for the user to check the recording activity of a Shimmer3 at any stage during SD logging operation – without interrupting the data recording. It should be noted, however, that the Bluetooth radio on the Shimmer will remain active, regardless of the connection state, so in use cases where battery life is an important factor, SDLog should be chosen over LogAndStream firmware.

6.1. General operation

State 1 – Idle

Note: To use the Bluetooth features of the LogAndStream firmware, the Shimmer3 device must first be paired with a “host” device (e.g., a PC), as outlined in the Shimmer3 User Manual.

When a Shimmer3, programmed with LogAndStream firmware, is first powered on or reset, it enters the Idle state. How long it will remain in the Idle state depends on the configuration of a parameter called “Undock Start” (see Section 7.2.3 for more details).

The ‘paths’ referred to below are indicated in Figure 6‑1.

1. To enter BT Connected state, a Bluetooth connection must be initiated by a “host” side application (e.g., ConsensysBASIC). A subsequent disconnection of the Bluetooth connection will send the Shimmer3 back to the Idle state.

10. There are two ways of reaching the SD Logging state from the Idle state:

    1. If Undock Start is disabled, the device will remain in the Idle state until either a connection is made over the Bluetooth link (i.e., by opening a serial connection), or the user instigates a single button press which will initiate SD logging mode. Similarly, another short button press will terminate SD Logging mode and return the Shimmer3 to Idle state.

    2. If “Undock Start” is enabled, the Shimmer3 will progress directly from Idle mode into SD Logging mode after a power-on or reset condition.

State 2 – BT Connected

In the BT Connected state, the Shimmer3 can process various commands to configure its sensors and sampling parameters, set calibration parameters, send configuration settings back to the “host” (PC, mobile or other) and start sampling. When the Shimmer3 is in the BT Connected, BT Streaming or BT Streaming + SD Logging states, there can be active communication between the Shimmer3 and the host over the Bluetooth serial connection. Packets of bytes are sent in both directions and these can consist of commands, responses or data.

2. The host application (e.g., ConsensysBASIC) allows the user to fully configure the Shimmer3 and choose between either BT Streaming or BT Streaming + SD Logging recording modes.

State 3 – BT Streaming

When a command to start BT Streaming is received, the Shimmer3 enters the BT Streaming state and starts sampling data from its sensors and sending that data over the Bluetooth link. This state will continue until one of the following events occurs:

1. A command to stop logging is received, whereupon the Shimmer3 returns to the BT Connected state.

2. The serial connection is closed, sending the Shimmer3 back into the idle state.

3. The Shimmer3 goes out‑of‑range from the “host” device for a certain period (≈ 150 s), whereupon the Shimmer3 will also return to Idle state.

State 4 – BT Streaming + SD Logging

The BT Streaming + SD Logging state can be initiated by using the “host” application (e.g., ConsensysBASIC). When the Shimmer3 enters the BT Streaming + SD Logging state, the Shimmer3 will start sampling data from its sensors and send this data over the Bluetooth link as well as store it locally in raw file format to the microSD card (see Section 8).

The Shimmer3 will continue in this mode until one of the following occurs:

1. A stop logging command is received from the “host”. The Shimmer3 will return to the BT Connected state.

2. In the event of either a Bluetooth disconnection, the serial port being closed or the Shimmer3 travelling out of range for a certain period (≈ 150 s), the Shimmer3 will automatically enter the SD Logging state (as illustrated by “Path (d)” in Figure 6‑1).

State 5 – SD Logging

In the SD Logging state, the Shimmer3 will continuously read data from its sensors and store this data locally to a raw data file in its on-board microSD card (for more information see Section 8). The Shimmer3 will remain open to a Bluetooth connection by a host device, whereupon the Shimmer3 will automatically switch to the BT Streaming + SD Logging state without interruption to the SD logging data stream.

6.2. Real-time clock

Shimmer3R: Shimmer3R keeps the real-world clock across a reset; Shimmer3 loses it at every reset. A Shimmer3R loses the clock only when it loses power completely, and then restarts from 2000-01-01, so host software should still set the clock on every connection. The timekeeping reference describes both generations.

In LogAndStream v0.3.0, the capability to save the ‘real-world’ time from a PC to the Shimmer was introduced. This feature allows simple synchronisation with external events or third party devices by providing a common clock timestamp at the beginning of a logging session. The firmware indicates if the real-time clock has been set on the Shimmer. A blue/green LED indicator on the Shimmer will intermittently flash to indicate to the user that the real time clock has not been set (see Section 6.5 for LED indicators).

The recommended method to set the real-time clock from a PC to a Shimmer device is to use Shimmer ConsensysBASIC or ConsensysPRO software. The time is set via the UART interface with Shimmer Dock or Consensys Base or over Bluetooth in the Consensys software applications. In the case of a user who wishes to set the time outside of a Shimmer application, more details should be sought in the source code for LogAndStream Firmware on the Shimmer github repository³.

6.2.1. Important considerations for real-time clock

Please note the following factors that will affect the reliability of the real-time clock timestamps.

1. The real-time clock is saved in volatile memory. This means that, if the Shimmer is powered off or reset, the real-world time will be lost.

2. The real-time clock setting is implemented as a once-off operation that relates the timestamp on the Shimmer’s local clock to the timestamp on the PC. This setting cannot measure or correct for drift of the Shimmer’s internal clock. It is recommended that a data collection session should begin as soon as possible after the real-time clock has been set on the Shimmer. Similarly, if the PC time is being set on multiple devices (Shimmer or third party devices), this should be done on all devices within as short an interval as possible to reduce the effect of clock drift.

Note: It is not critical for the real-time clock setting to be present on the Shimmer in order to record data to the Shimmer’s SD card. The LED sequence will change to the “logging” mode after undock start or push button start and will return to the error LED sequence when logging ceases (see section 6.3 for LED indicators).

6.3. Time synchronisation

Each Shimmer device has its own independent on-board clock which is reset to zero whenever the device is powered on or reset and which is subject to drift, caused by component-specific differences and environmental factors, like temperature. It is important to note that LogAndStream firmware does not support time synchronisation, due to conflicts between the requirements for host communication and Shimmer-Shimmer communication. Users who require synchronisation (e.g., if the duration of a data collection session is too great to accept the clock drift) are advised to use the SDLog firmware and Shimmer software applications that enable sync (e.g ConsensysPRO)

6.4. Start/Stop Logging

6.4.1. Dock (automatic start)

This is the default way for the Shimmer3 to start and stop logging data to the SD card and is enabled if the “Undock Start” option is enabled from ConsensysBASIC or ConsensysPRO or ShimmerCapture (i.e., configuration file line entry: “userbutton=0”). With this configuration, if the Shimmer is powered on or reset while it is not on the dock, the following process will begin immediately:

1. The Shimmer unit will go into standby mode for up to 3 seconds.

2. The Shimmer unit will read the configuration file and create the required directories on the SD card.

3. The Shimmer unit will start logging.

Alternatively, if the Shimmer is powered on or reset while it is on the dock, the three steps above will begin as soon as the Shimmer is removed from the dock.

In either case, logging will continue until the Shimmer unit is reset, powered off, replaced in the dock or the battery runs out.

Repeatedly resetting the Shimmer unit will result in multiple logging sessions on the SD card.

Warning: Please note that it is not recommended to dock the Shimmer unit while it is being configured by the firmware. Ideally, the Shimmer unit should either be powered off or in standby mode whenever it is being placed on the dock.

6.4.2. User button (manually start/stop logging)

This option relies on the orange user button on the Shimmer3 enclosure to start/stop logging.

With this setting, undocking the Shimmer unit or powering it on when it is off the dock will trigger the same steps 1 and 2 as above (i.e., standby for 3 seconds followed by configuration). However, logging will not start until the user button is pressed by the user.

If the user button is pressed while the Shimmer unit is on the dock or Base or connected over Bluetooth, it will have no effect on logging. Otherwise, if the user button is pressed while the Shimmer unit is undocked (and not logging), logging will start immediately.

Logging will continue until one of the following occurs: the user button is pressed, the Shimmer unit is reset, powered off, replaced in the dock, the battery runs out or the Shimmer is told to stop by a host application, whichever happens soonest.

Repeatedly pressing the user button (with Push Button Start enabled) or docking and undocking the Shimmer3 unit (with Undock Start enabled) will result in multiple logging sessions on the SD card.

6.5. LED indicators

Shimmer3R: Shimmer3R has one RGB LED in each location instead of five single-colour LEDs, with the same patterns. “RTC not set” is a single cyan LED rather than green and blue together, and a purple flash on the lower LED, which Shimmer3 cannot show, means the Shimmer3R is entering its bootloader. The LED feedback reference lists every state for both generations.

TODO(S3R): The current firmware shows steady red while charging, where the tables below say yellow; see the LED feedback reference, section 4.4. Confirm from which version and update the tables.

The Shimmer3 has five LEDs in two locations: the lower LED location (location A) which contains the green (b), yellow⁴ and red LEDs; and the upper LED location (location B) which contains the green (a) and blue LEDs.

The two upper LEDs are used to indicate state of operation for the LogAndStream firmware. The status of each depends on whether the Shimmer3 is in a docked or undocked state, as shown in Table 6‑1 below.

Note: The Shimmer unit should never be placed in the dock while the operational LED’s indicate that it is configuring as this may cause a file-system error. Once configuration has begun, you must power off or reset the Shimmer unit before docking.

Note: It is not recommend to place the Shimmer3 unit in the dock while it is logging data as this can cause SD card access problems. Once logging has begun, you must power off or reset the Shimmer3 unit before docking.

Note: The above SD error LED sequence is present in LogAndStream_v0.9.0 and later. Older versions of LogAndStream couple both SD and RTC errors into a 0.1s Blue/0.1s Green LED flashing sequence.

Note: There are a couple of reasons for the Shimmer to enter “Error” mode.

1. If the real-world time has not been set on the Shimmer.

   1. It is not critical for the real-world time setting to be present on the Shimmer in order to record data to the Shimmer’s SD card. The LED sequence will change to the “logging” mode after undock start or push button start. LED sequence for real-time clock status only in LogAndStream v0.6.0 or later.

2. If no configuration file is present on the SD card.

3. If the Shimmer is unable to access the SD card for some reason e.g. if the SD card is corrupt (try reformatting the SD card but note that this will delete all of the data from the SD card)

The lower three LEDs are exclusively used across all Shimmer3 firmware as a battery charge indicator and will show the colour corresponding to the charge status. As above, the status of each LED depends on whether the Shimmer3 is in a docked or undocked state. If the Shimmer3 is on a dock or multi-charger and is powered on, the battery charge status indicator will show a solid LED. If the Shimmer3 is undocked, the corresponding LED will flash, as described below in Table 6‑2.

7. Using the Firmware

7.1. Bluetooth Streaming

LogAndStream is controlled over Bluetooth with one-byte commands, each followed by any parameters it needs. The Shimmer acknowledges every command it receives. There are four kinds:

• Set commands write configuration: enabled sensors, sampling rate, sensor ranges and rates, calibration parameters and so on.
• Get commands read configuration and status back.
• Action commands start and stop streaming and logging (for example START_STREAMING_COMMAND, START_SDBT_COMMAND), or toggle an LED.
• In-stream commands (GET_STATUS_COMMAND, GET_DIR_COMMAND, GET_VBATT_COMMAND) can be sent while the Shimmer is streaming or logging; their responses arrive among the data packets.

START_STREAMING_COMMAND starts streaming only, and works only while the Shimmer is neither logging nor streaming. START_SDBT_COMMAND starts or resumes streaming and logging together; use it to stream from a Shimmer that is already logging.

Both generations use the same protocol. The complete, current reference, with every command, its bytes, the packet structure, the in-stream responses and the firmware versions that introduced each feature, is the Shimmer3 / Shimmer3R Bluetooth Communication Protocol:

• Opcode table
• Status, ACK/NACK and in-stream responses
• Streaming data flow, including how to separate data packets from in-stream responses
• Command reference
• Connection session workflow

Most users never need these details: Consensys and the Shimmer APIs (Java/Android, C#, Python, MATLAB, LabVIEW) implement the protocol.

Shimmer3R: A Shimmer3R accepts the same commands over both Classic Bluetooth (serial port profile) and Bluetooth Low Energy. It also has commands Shimmer3 ignores, to list, read and delete the files on its SD card over Bluetooth; see the SD file transfer reference.

7.1.1. Bluetooth Latency

Delays due to Bluetooth transmission should be taken into account when streaming data, particularly if the data is to be synchronised on the receiver side. Our lab tests have shown up to 100 ms of latency with considerable variation (> 50 ms). These measures result from multiple FIFOs in the data path, as expected in wireless data acquisition systems using conventional computing devices for the data end-points. Actual performance is strongly impacted by end-point system configuration and load.

7.2. Configuration

To use this firmware image, the user must provide the desired configuration parameters of each Shimmer. It is recommended that Shimmer software applications (e.g. ConsensysBASIC or ConsensysPRO) are always used for configuration of sensors for logging, in order to avoid errors introduced in the development of third party software. However, the configuration parameters can be sent to the Shimmer outside of the provided applications, by following the guidelines throughout this section.

Since v0.4.0, there are three options available for configuring the Shimmer, as outlined in the following section.

7.2.1. Configuring a device

Configuration via Bluetooth

The recommended method of writing the calibration parameters to a Shimmer device, programmed with LogAndStream firmware is to connect via the ConsensysBASIC or ConsensysPRO software application (available from the Shimmer website1) and use the graphical interface to configure the device via a Bluetooth connection. Assuming that LogAndStream v0.4.0 (or later) firmware is programmed on the devices, the new configuration parameters will automatically be written to the appropriate file on the SD card, following the file structure outlined later in this section.

Configuration via UART

Configuration via UART was a new feature introduced in v0.4.0.

Configuration parameters can be written directly to the non-volatile memory (InfoMem) on the Shimmer3 via the UART interface, using a Shimmer Dock or Consensys Base.

To write the configuration parameters via UART, the following must be specified:

• The memory address of the first byte.

• The number of bytes to be written.

• The values to be written to the relevant bytes.

Please refer to the Section 9.1 in the Appendices of this document for a description of InfoMem contents and refer to the information in the following subsections for more details regarding the meaning of and valid values for each parameter.

If the configuration parameters are modified via UART, the ‘InfoMem changed’ flag should be set to true to notify the firmware that the configuration file on the SD card (see below) and the InfoMem do not hold the same configuration parameters. See Section 9.1 for a description of which bit in InfoMem contains this flag.

Configuration by file

The configuration parameters can be saved in a configuration file, named sdlog.cfg, on the Shimmer’s on-board SD card, from which they will be loaded by the Shimmer at initialisation. Configuration parameters are read from the sdlog.cfg file or from the InfoMem according to the following:

• If the ‘InfoMem changed’ flag (see UART method) is set, the firmware will read the configuration from InfoMem and will replace any existing sdlog.cfg file.

• If the ‘InfoMem changed’ flag is not set, the firmware will look for a configuration file on the SD card. Assuming that the file exists, the configuration will be read and copied to InfoMem.

• If the ‘InfoMem changed’ flag is not set and no configuration file exists on the SD card, the firmware will check that there is a valid configuration on the InfoMem. Assuming that there is a valid configuration, a new configuration file will be written with the configuration parameters from InfoMem.

• If the ‘InfoMem changed’ flag is not set, no configuration file exists on the SD card, and there is not a valid configuration on the InfoMem, the firmware will indicate an error (see Section 6.5 for LED indicators).

Thus, to change the configuration via the sdlog.cfg file, simply modify the sdlog.cfg file and reboot the Shimmer (there is no need to reprogram).

The configuration file must be saved in the top level directory of the SD card (i.e. not within a subfolder). This file allows configuration of all firmware features, as described throughout the following subsections. Each new line of the configuration file contains a single configuration parameter command. Users should note that the commands should be written exactly as they appear in the following sections so that they will be successfully parsed by the firmware. Any extra whitespaces will cause parsing errors.

If the configuration file has a glitch, such that the firmware cannot read it, a red/yellow LED indicator on the Shimmer will intermittently flash to warn the user of the error (see Section 6.5 for LED indicators). Note that this error will not occur in the event of parsing errors (e.g. typographical errors, extra whitespace, and repeated commands); it is the user’s responsibility to ensure that the sdlog.cfg file is well-written. Commands with typographical errors and similar will be ignored by the firmware.

Note: The above SD error LED sequence is present in LogAndStream_v0.9.0 and later. Older versions of LogAndStream couple both SD and RTC errors into a 0.1s Blue/0.1s Green LED flashing sequence. See Section 6.5 for full details.

It is recommended to use the ConsensysBASIC or ConsensysPRO application for configuration of Shimmer devices, to avoid errors in writing the configuration file. However, the configuration file can be written in any text editor and saved to the SD card without using Shimmer software, by following the guidelines throughout this section.

7.2.2. Enabling sensors

The user may choose which sensors are enabled and disabled on the Shimmer for a particular experiment. These can be enabled or disabled by setting the value of the appropriate bits in InfoMem or by modifying the configuration file. The configuration file should contain a new line for each sensor enable/disable command. If any sensor is not included in the configuration file, it will be assumed to be disabled.

The currently available sensors and their corresponding sdlog.cfg entries are listed in Table 7‑8 for Shimmer3.

Note: To enable the ExG Test Signal to verify the correct operation of the Shimmer3 ECG or EMG module, see the ECG User Guide or the EMG User Guide, available on our website⁵.

Certain combinations of sensors cannot be enabled simultaneously due to hardware or firmware restrictions. A table listing the Shimmer3 sensor conflicts is given in Table 9‑6 in Appendix 9.2.

7.2.3. Sensor parameters

See Table 7‑9 for sensor configuration parameters and corresponding configuration file commands.

Table 7‑9: Sampling configuration options for Shimmer3

Note: See ECG User Guide or EMG User Guide for ExG gain and rate settings.

7.2.4. Experiment settings

In addition to the sensor-specific parameters listed in the previous subsections, configuration parameters for an experiment or data collection can be configured using the commands in Table 7‑10. Further details describing these parameters are provided below the table.

Table 7‑10: Logging configuration options.

The Sampling rate is the frequency at which sampling should be performed. It should be noted that the actual sampling rate may not be exactly equal to this parameter because the firmware requires the sampling period to be an integer multiple of 1/32768 s. For example, a desired sampling frequency of 500 Hz requires a sampling period of 2 ms which is equal to 65.536*(1/32768) s. In the firmware, this sampling period will be rounded up to 66*(1/32768) = 2.014 ms, so the true sampling rate will be 496.48 Hz. The actual sampling rate that will be used (True Fs) can be calculated from the specified sampling rate (Fs) using the following formula:

True Fs = 32768/(round(32768/Fs))

where the round() function means rounding to the nearest integer.

The Baud rate setting stores the baud rate at which the Shimmer’s microcontroller communicates with the on‑board Bluetooth module and consequently, determines the rate at which data can be communicated to a host device. The allowable values are listed in Table 9‑4.

Number of Shimmers and Shimmer ID number are used to describe the total number of Shimmers in an experiment and the ID number of each individual Shimmer. These parameters do not have any explicit function in firmware.

The Shimmer name parameter allows the user to specify a meaningful name for each device. It is used by the firmware to name the data directories on the SD card. For more information on the directory structure, see Section 8.1.

The Experiment ID parameter allows the user to specify a meaningful name for an experiment. It is used by the firmware to name the data directories on the SD card. For more information on the directory structure, see Section 8.1.

The Configuration ID is a numeric identifier for the experiment. It typically refers to the date and/or time at which the configuration file was created (but users may choose any valid numeric value that is meaningful to them⁶). It is appended to the experiment ID name in the data directories to allow the user to identify different experiments in the case that the same experiment ID name is used multiple times. It is vital that this parameter is equal for all Shimmers in the experiment if synchronisation or single-touch start is enabled.

The User button refers to the button on the baseboard which is accessible by pressing the circular orange button on the Shimmer3 enclosure. If the button is enabled, then logging will begin when the user button is pressed. Logging will finish when any one of the following happens: the user button is pressed again; the Shimmer battery runs out; the Shimmer is placed on the Shimmer Dock.

The SD Error is a flag (introduced in LogAndStream_v0.9.0) which enables/disables the SD-error LED sequence (red/yellow) which flashes should the Shimmer have issues accessing the SD card or if there is no SD card present.

7.3. Calibration parameters

Calibration information bytes, including the calibration parameters and the timestamp when the sensors are calibrated, are not used in this firmware image to provide calibrated data, but they are saved in the configuration header of the raw data files on the microSD card. This is so that the data can be calibrated after data collection.

7.3.1. Providing calibration information

As with configuration parameters, there are a number of ways to provide calibration information in LogAndStream firmware. The recommended method is to use the Shimmer 9DoF Calibration application, as outlined below.

Writing calibration information to InfoMem via Bluetooth

The recommended method of writing the calibration information to the Shimmer is to program the device with the latest LogAndStream firmware and perform calibration using the Shimmer 9DoF Calibration application, which can be downloaded from the Shimmer website⁷. Assuming that LogAndStream firmware is programmed on the devices during calibration⁸, the new calibration information will automatically be written to appropriate files on the SD card, following the file structure outlined later in this section.

Writing calibration information to file

Users should note that this is not a recommended method of providing calibration parameters for the Shimmer when using LogAndStream v0.4.0.

To supply calibration information by file, the user should create a folder called “Calibration” or “calibration” in the top-level SD card directory and should save the calibration file within that folder. The format of the file should match the output given by the Shimmer 9DoF Calibration application’s “Save to file” function.

Since LogAndStream v0.7.0 and later the Shimmer firmware uses a new format of calibration file, called the “Calibration Dump”. This file stores all calibration information, including all ranges of all kinematic sensors, in the same file in hex format. The file name is “calib_xxxx” where xxxx is the BT Radio ID of the Shimmer device (located on the sticker on the back of the Shimmer). Backwards compatibility has NOT been included, i.e. the users have to re-calibrate the shimmers before the new calibration system can be functional.

Calibration information is read from the calibration folder or from the InfoMem according to the following:

• If the calibration dump file is available, the firmware will read the calibration information from the file, and update the RAM, the InfoMem and the configuration header.

• If the calibration dump file is not available, the firmware will generate a default calibration dump in the RAM. If the InfoMem is available, the firmware will replace the corresponding ranges of the sensors in the RAM with the InfoMem values. Then the firmware will create a new calibration dump file “calibration\calib_xxxx” and update the configuration header with the RAM values.

• If when the shimmer is docked and the user send new calibration information via Bluetooth, the firmware updates the RAM, the InfoMem and the configuration header with the new information. When the shimmer is undocked, the firmware updates the calibration dump file with these parameters. Please note that if the shimmer is powered off before undocked, the received new calibration information will be lost, and thus, the calibration dump file will not be updated in such case.

The calibration timestamp will be all 0 when default, the calibration parameters are given in Table 7‑11.

Table 7‑11 Default calibration parameters set in firmware if no available calibration information

The following briefly introduces the structure of the calibration dump file. A default calibration dump file is of 538 bytes for LogAndStream, the included ranges of sensors follow the sequence in Table 7‑12. The file structure supports more calibration parameters if necessary. The structure of the calibration dump file is shown as in Table 7‑13. The first two bytes are the Total_Length bytes. This length is the total data length except the “Total_Length” itself. i.e. if the file is 538 bytes long, including the first two Total_Length bytes and 536 data bytes, then there should be Total_Length=536 in this case. Following the Total_Length bytes are 8 version info bytes. The sensor calibration timestamp and parameters per range are stored after the first 10 bytes in sequence.

Table 7‑13 Calibration Dump file structure

8. Reading the SD card Data

Note: The SD card format reference is the current, authoritative description of the directory structure, the file header and the data format for both generations. The tables in this chapter describe LogAndStream v0.11.x for Shimmer3.

Shimmer3R: A Shimmer3R does not log while it is docked or connected to USB, because the card is handed to the PC. Its files can also be read over Bluetooth without docking. In the data, Shimmer3R orders some channels differently and sends pressure before temperature; Consensys and the Shimmer APIs handle this.

TODO(S3R): Compare this chapter with the SD card format reference and, if it holds nothing the reference lacks, replace it with a short summary and the link, as was done for the Bluetooth commands in 7.1.

It is recommended that the Consensys software (available from the Shimmer website1) be used to read the data from the SD card. However, using the following guidelines, the user may use their platform of choice to read and parse the data.

8.1. SD card directory structure

The data directory structure on the SD card is as follows:

• A folder called “data” is created by the firmware in the SD card top-level directory if it does not already exist.

• A subfolder is created within the data folder each time a new “experiment id” is encountered in the sdlog.cfg file (default is “default_exp”).

• A new subfolder is created within the appropriate experiment folder each time logging starts. The naming convention for this folder is the Shimmer name specified in the sdlog.cfg file (default is the abbreviated form of the Shimmer3 id number (e.g. IDbf5e)), followed by a three digit number which is sequentially incremented at new logging session.

For example, the first time a Shimmer3 with name “device1” in experiment “experiment1” logs data to the SD card, it will create the folder data/experiment_*/device1-000/. The next time it stops and subsequently restarts logging, it will create data/experiment_*/device1-001/, etc. The * in the filename represents the time-code at which the Shimmer was last configured through ConsensysBASIC, ConsensysPRO or ShimmerCapture.

• Within this subfolder, data is logged in files which are named sequentially with a three digit number indicating the order of logging, starting with 000. The file is closed after 1 hour of continuous data logging and a new file is opened in the same subfolder, such that the second file is called 001 and so on. The last file is closed when logging stops and a new file (in a different folder) is created the next time logging begins.

8.2. Parsing the raw data file

8.2.1. Raw data file configuration header

Each RAW data file (e.g. data/experiment1/device1-000/000), begins with a header which contains the Shimmer3 configuration information. The structure of this header is outlined in the tables below, where each row of each table represents one byte and the individual bits are separated, where appropriate, depending on whether each bit represents an independent binary value or the entire byte contains a single value. Empty cells and cells with constant values represent bits that are reserved for future use. For Shimmer3, there are a total 256 bytes in the header.

Byte # 0 – 9: Enabled Sensors

Byte # 10 – 19: Trial Configuration

Byte # 20 – 29: Not applicable to LogAndStream

Byte # 30 – 39: Firmware and Shimmer Parameters

The Shimmer Version indicates the hardware version for which the code was compiled. Shimmer3 is denoted by 3. The FW Version bytes define the firmware version. The Type field will always be 3 for LogAndStream firmware images. Major and Minor versions are indicated by a two- and one-byte value, respectively (e.g., for LogAndStream v1.0, Major = 1 and Minor = 0). The Release field can be ignored by users; it will have a value of 0.

Byte # 40: Derived Sensors

Used to enable derived sensors from the native Shimmer sensors in Shimmer data acquisition software applications, specifically Consensys.

Byte # 41 – 43: Reserved for future use

Byte # 44 – 51: Real Time Clock Difference

This value is the difference between the local PC time (using Unix epoch) and the time on the Shimmer internal clock. It is used primarily to enable real world timestamps in Shimmer data acquisition software applications, like Consensys. See Section 6.2 for more information.

Byte # 52 – 55: Configuration Time

This value comes from the Configuration ID parameter from Section 7.2.3.

Byte # 56-76: ExG Configuration

For Shimmer3, these parameters refer to the ExG module which is made up of two ADS1292R chips.

Byte # 76-181: Calibration parameters

Bytes 76 to 181 store the calibration data for individual Shimmer3 sensors. These parameters are used to calibrate the RAW data during post-experimentation analysis. The layout and description of these bytes can be found in the Appendix section of this document (i.e., Section 9.4).

Byte # 182-213: Calibration Timestamps

Calibration Timestamps was a new feature introduced in v0.7.0.

Bytes 182 to 213 store the calibration timestamps for individual Shimmer3 sensors. Such timestamps are used to calibrate the RAW data during post-experimentation analysis. The layout and description of these bytes can be found in the Appendix section of this document (i.e., Section 9.4).

Byte # 214-216: Daughter Card ID bytes

Daughter Card ID bytes were a new feature introduced in v0.7.0.

Bytes 214 to 216 store the Daughter Card ID bytes. These three bytes are read from the daughter card, representing the version numbers of the card. If no expansion cards are appended to the Shimmer3 unit, the values of these three bytes are 0xFF.

Byte # 217 – 250: Reserved for future use

8.2.2. Initial Timestamp

Immediately after the configuration header, there is a 5-byte timestamp, whose value is the time on the local Shimmer3 clock (in units of ticks of the 32 kHz clock) when the first sample in that file was recorded.

8.2.3. Raw data file format

Following these 256 bytes, the logged sensor data begins (i.e. at the 257^th byte in the file). The data is written in blocks of up to 512 bytes. These blocks are continuously written until data has been logged to a given file for one hour, at which time the current file is closed and a new file is opened, with the same format as the current one (i.e. the configuration header is repeated in every file).

The exact number of bytes per block and the interpretation of those bytes of data depend on the enabled sensors. The total number of bytes written per block, Bp, is given by:

Bp = N \times (Bs + 3)

where:

Bs = Nc3 * 3 + Nc2 * 2 + Nc1, is the number of bytes per sample, which depends on the number of 3byte channels (Nc3), number of 2byte channels (Nc2) and the number of 1-byte channels (Nc1) required by the enabled sensors;

N = floor(512/(Bs + 3)), is the integer number of samples per block, with a buffer of 512 bytes and the extra 3 bytes per sample are the 24-bit (16-bit in LogAndStream v0.5.0 or earlier) timestamps from the Shimmer3‘s local 32 kHz clock.

All of the bytes in the block are interleaved local 24-bit (16-bit in LogAndStream v0.5.0 or earlier) timestamps and the associated sensor samples. For example, with the accelerometer and the gyroscope enabled, there are three 2-byte channels for the accelerometer and three 2-byte channels for the gyroscope in each sample and the data would be as shown in the following table until a total of Bp = 510 bytes is reached (Bs = 12; N = floor((512)/(12 + 3)) = 34).

In LogAndStream v0.5.0 or earlier the Shimmer timestamp channel was only two bytes (16-bit)

Order of raw data channels

The order in which the sensors appear in the data channels depends on which sensors are enabled. The firmware will assign the channels in the order outlined in Table 8‑2 (top to bottom). Note that “Accel (LN)” refers to the low noise accelerometer, whilst “Accel (WR)” refers to the wide range accelerometer. The number of channels per sensor is listed beside the channel name. The endianess of the bytes for each channel is also specified.

9. Appendices

9.1. InfoMem contents

Note: The configuration memory reference is the current, authoritative byte map for both generations, with each field’s options and defaults.

TODO(S3R): Compare this appendix, and the calibration bytes in 9.4, with the configuration and calibration references, and replace them with links if nothing here is missing there.

The information in this section applies to LogAndStream Firmware v0.4.0 (or later) only. Note that some of the parameters described (e.g. those related to synchronisation) do not apply to LogAndStream Firmware but are included for completeness.

The configuration and calibration parameter values are stored by the Shimmer3 in the InfoMem, which is the part of the Shimmer3 memory that survives a reset or power cycle but is overwritten when the Shimmer3 is reprogrammed. The format of the configuration data stored in InfoMem is as follows:

Table 9‑1 InfoMem layout overview.

Selected Sensors – InfoMem Bytes 3 to 5

The Selected sensors bytes have a single bit assigned to each sensor as follows:

Table 9‑2 Selected Sensor Bytes

Sensor Config Bytes – InfoMem Bytes 6 to 9

The Sensor Config bytes contain the following parameters:

Table 9‑3 Sensor Config bytes

ExG Configuration Bytes – InfoMem Bytes 10 to 29

These bytes store the configuration bytes which are sent to the ECG/EMG Expansion Board if one is connected and enabled. For detailed information on these bytes please refer to either the ECG User Guide or the EMG User Guide – both of which are available for download from the members section of the Shimmer website6.

BT Communication Baud Rate – InfoMem Byte 30

This byte stores the baud rate at which the Shimmer’s microcontroller communicates with the on‑board Bluetooth module and consequently, back to a base device. There are 11 allowable options, as listed in Table 9‑4:

Table 9‑4 BT Communication Baud Rate byte options

Derived Channels – InfoMem Bytes 31 to 33

These bytes contain flags to indicate the type of peripheral that is attached to the analog channels. These bytes have no explicit function in firmware and are included to allow software applications, specifically Consensys, to correctly label the data. In custom applications, they may be used as the developer sees fit.

Calibration Parameters – InfoMem Bytes 34 to 117

The calibration parameters for the inertial measurement units (accelerometer, gyroscope and magnetometer) consist of a three-element offset bias vector, a three-element sensitivity vector and a 3×3-element alignment matrix⁹. The structure of these values when they are sent to/from the Shimmer3 and stored in InfoMem is as follows:

• Each of the 3 offset bias vector values are stored as 16-bit signed integers (big endian) and are contained in bytes 0-5.

• Each of the 3 sensitivity vector values are stored as 16-bit signed integers (big endian) and are contained in bytes 6-11.

• Each of the 9 alignment matrix values are stored as 8-bit signed integers and are contained in bytes 12-20.

MPL Parameters – InfoMem Bytes 118 to 186

These bytes are not relevant for LogAndStream firmware.

SD Logging – Experiment parameters – InfoMem Bytes 187 to 229

Table 9‑5 Experiment Settings

InfoMem contents changed flags – InfoMem Byte 230

This byte contains the ‘InfoMem changed’ (bit 0) and ‘calibration changed’ (bit 1) flags, which indicate to the firmware that the InfoMem contents have been updated and that the new parameters should be written to the configuration file and/or the calibration files on the SD card. See Sections 7.2.1 and 7.3.1 for more details.

Slave Node IDs – InfoMem Bytes 256 – 381

These bytes contain the mac addresses of the slave nodes, with six (6) bytes used per node.

9.2. Shimmer3 sensor conflicts

The Shimmer3 sensors conflicts are shown below in Table 9‑6. Note that all Accelerometer, Magnetometer, Gyroscope, Pressure and Vbattery sensor channels are compatible with any other sensors.

Table 9‑6: Shimmer3 sensor conflicts.

9.3. Example of the Shimmer3 configuration file (sdlog.cfg)

accel=1

gyro=1

mag=1

exg1_24bit=0

exg2_24bit=0

gsr=0

extch7=0

extch6=0

br_amp=0

vbat=1

accel_d=0

extch15=0

intch1=0

intch12=0

intch13=0

intch14=0

accel_mpu=0

mag_mpu=0

exg1_16bit=0

exg2_16bit=0

pres_bmp180=0

sample_rate=51.20

mg_internal_rate=6

mg_range=1

acc_internal_rate=0

accel_mpu_range=0

pres_bmp180_prec=0

gsr_range=4

exp_power=0

gyro_range=1

gyro_samplingrate=155

acc_range=0

acc_lpm=0

acc_hrm=1

user_button_enable=1

rtc_error_enable=1

sd_error_enable=1

iammaster=0

sync=0

low_battery_autostop=0

interval=0

myid=0

Nshimmer=0

shimmername=Shimmer_9507

experimentid=DefaultTrial

configtime=1495542804

baud_rate=9

derived_channels=0

max_exp_len=0

EXG_ADS1292R_1_CONFIG1=0

EXG_ADS1292R_1_CONFIG2=128

EXG_ADS1292R_1_LOFF=16

EXG_ADS1292R_1_CH1SET=0

EXG_ADS1292R_1_CH2SET=0

EXG_ADS1292R_1_RLD_SENS=0

EXG_ADS1292R_1_LOFF_SENS=0

EXG_ADS1292R_1_LOFF_STAT=0

EXG_ADS1292R_1_RESP1=2

EXG_ADS1292R_1_RESP2=1

EXG_ADS1292R_2_CONFIG1=0

EXG_ADS1292R_2_CONFIG2=128

EXG_ADS1292R_2_LOFF=16

EXG_ADS1292R_2_CH1SET=0

EXG_ADS1292R_2_CH2SET=0

EXG_ADS1292R_2_RLD_SENS=0

EXG_ADS1292R_2_LOFF_SENS=0

EXG_ADS1292R_2_LOFF_STAT=0

EXG_ADS1292R_2_RESP1=2

EXG_ADS1292R_2_RESP2=1

9.4. SD data file configuration header – sensor calibration bytes

Byte # 76-96: Wide Range Accelerometer Calibration

For Shimmer3, these parameters refer to the LSM303DLHC accelerometer. There are other accelerometers also populated on the Shimmer3 circuit board; the associated calibration parameters for the other devices are stored in a later section of the configuration header.

Byte # 97 – 117: Gyroscope Calibration

Byte # 118 – 138: Magnetometer calibration

Byte # 139 – 159: Analog Accelerometer Calibration

Byte # 160 – 181: Temperature (BMP180) and Pressure Calibration

Byte # 160 – 181 and Byte # 222-223: Temperature (BMP280) and Pressure Calibration

Byte # 182 – 213: Calibration Timestamps

Calibration Timestamps was a new feature introduced in v0.7.0.

9.5. Legacy Firmware: LogAndStream v0.6.0 (or earlier) Writing calibration parameters

Writing calibration parameters to InfoMem via UART

Using the same interface as that which is used for writing configuration parameters to InfoMem (see Section 7.2.1); the calibration parameters can be written directly to InfoMem via UART commands. If the calibration parameters are modified via UART, the ‘calibration changed’ flag should be set to true to notify the firmware that the calibration file on the SD card and the InfoMem do not hold the same calibration parameters. See Section 9.1 for a description of which bit in InfoMem contains this flag.

The ability to write calibration parameters to InfoMem via UART is a new feature in LogAndStream v0.4.0 and was added to support key functionality in Consensys software. To avoid conflicts between InfoMem parameters and calibration files, it is not recommended that this method of writing calibration parameters is used outside of Shimmer software applications.

Writing calibration parameters to file

Users should note that this is not a recommended method of providing calibration parameters for the Shimmer when using LogAndStream v0.4.0 (or later).

To supply calibration parameters by file, the user should create a folder called “Calibration” or “calibration” in the top-level SD card directory and should save calibration files within that folder. The format of the file should match the output given by the Shimmer 9DoF Calibration application’s “Save to file” function. There should be a separate file for each sensor and range (as required) and the filenames should follow the defined convention, in order to identify the sensor and range for which the parameters apply, as listed in Table 9‑7 below.

Table 9‑7 Calibration file name and keyword

The provision of a separate calibration file for each sensor and range is a new feature in LogAndStream v0.4.0 and it was implemented to enable the simultaneous storage of calibration parameters for multiple ranges of a given sensor type, which was previously not possible. Backwards compatibility has been included, such that any existing legacy calibration files, with filename “calibParams.ini” (case sensitive), may be used with LogAndStream v0.4.0.

Calibration parameters are read from the calibration folder or from the InfoMem according to the following:

• If the ‘calibration changed’ flag is set, the firmware will read the calibration parameters from InfoMem and will replace any existing calibration file(s).

• If the ‘calibration changed’ flag is not set, the firmware will check the range settings of the enabled sensors and look for an appropriate calibration file on the SD card. Assuming that the file exists, the calibration will be read and copied to InfoMem.

• If the ‘calibration changed’ flag is not set and no appropriate calibration file exists on the SD card, the firmware will check that there is a valid set of calibration parameters on the InfoMem. Assuming that there is a valid set, (a) new calibration file(s) will be written with the calibration parameters from InfoMem.

• If the ‘calibration changed’ flag is not set, no appropriate calibration file exists on the SD card, and there is not a valid set of calibration parameters in the InfoMem, the firmware will assume default calibration parameters, as given in Table 7‑11. No file will be created in this case.

Table 9‑8 Default calibration parameters set in firmware if no calibration file present on microSD card.

9.6. Troubleshoot

Red and yellow LEDs flash when I undock or reboot my Shimmer.

The red and yellow LEDs flashing on the Shimmer3 indicate that the firmware either cannot create the required directories or cannot write to the required files on the SD card. Try the following steps to rectify the problem:

1. Check that the SD card is correctly inserted.

2. Ensure that the SD card has a capacity of 2 GB or less and is compatible with Shimmer3 (refer to the Shimmer User Manual, available for download from www.shimmersensing.com).

3. Ensure that the SD card memory is not full.

4. Ensure that the experimentid and shimmername parameters, specified in the sdlog.cfg file contain only alphanumeric characters (a,…, z, A,…, Z, 0,…, 9), dash (‘-‘) and underscore (‘_’).

Note: The above SD error LED sequence is present in LogAndStream_v0.9.0 and later. Older versions of LogAndStream couple both SD and RTC errors into a 0.1s Blue/0.1s Green LED flashing sequence.

When I undock or power on my Shimmer, the green LED continuously flashes at a rate of 5Hz

Try power-cycling the Shimmer3. The problem may be due to an error in Bluetooth initialisation.
If the problem persists after power-cycling, try changing the SD card for a newer one – if the SD card is corrupt, the firmware may fail to correctly read the configuration file.

The data file is empty after logging

Ensure that you log for a minimum of one minute in order for data to be written to the SD card.

The configuration does not match the parameters in the sdlog.cfg file.

The sdlog.cfg file is read once each time the Shimmer3 is rebooted or undocked so the configuration will always match the most recent configuration file at the time of logging. If the parameters in the configuration header of your data files do not match those in the sdlog.cfg file, it is likely that you have changed the sdlog.cfg file contents since logging the data in question.

The calibration parameters in the configuration header do not match the Calibration/calibParams.ini file.

Calibration parameters will only be loaded for sensors that are enabled; calibration parameters for disabled sensors will all have zero value.

If a sensor is enabled and the calibration parameters do not match the calibration file, try the following steps to rectify the problem:

1. Ensure that the calibration file is stored in the correct file location from the SD card top level directory, according to one of the following (case-sensitive) options:

   • /Calibration/calibParams.ini

   • /calibration/calibParams.ini

No other file path will be recognised by the firmware.

2. Ensure that you have implemented the correct byte-order and endianness when you read the calibration parameters from the configuration header, according to the information in Section 8.2.