BrainBodyMeasures

From Neuroelectric's Wiki
Jump to: navigation, search

Using | Enobio or | StarStim you can access different Body and Brain Measurements and develop your applications based on them.

These applications rely on NIC for EEG data streaming over TCP/IP (see Interacting with NIC for more information). You can use the software if you would like to access the raw data.

The development of applications however is easier if you access already processed data and features. We have prepared 2 different software tools that gives you access to Enobio features: NE Feature Xtractor and the Brain Polyphony software. We describe them in the following sections.

You can access the features provided by these 2 tools from different programs. The data can be accessed remotely through TCP and OSC. For this you have to program either a TCP or OSC clients. We provide examples and tools implementing this access from Matlab/Simulink, Pure Data, and Processing. Moreover we provide an example of the kind of applications you can develop, for instance sonifying your brain waves.

Contents

NE Feature Xtractor

The Feature Xtractor is a standalone application that streams Enobio/Starstim its raw data and several features through a TCP/IP protocol. The feature xtractor is released as binary executable file for MS-Windows to be found HERE. It has been tested for Windows 7 and 8. If you do not have any computer with this OS, you can still use it. Please ask Starlab staff, we can provide you a solution.

Measures

You can access following Brain and Body Measurements with the NE Feature Xtractor:

  • Brain Features: Up to 5 configurable measures associated with workload, relaxation, drwosiness, etc (ask Starlab staff for concrete configuration).

  • Muscle features: Up to 5 configurable measures of particular muscle activity.

  • Raw data: Up to 20 channels of the device measured data for you to further process.

Instructions

Follow these steps in order to configure your feature xtractor:

1. Write the MAC address of your device in the spaces provided. The MAC address will be 00:07:80:XX:XX:XX. If you want to read data from a previously recorded file, select the checkbox "Read from file" and write the name of the file (with its extension) in the textbox next to it (for Mac users, you need to specify the whole path). The file needs to be placed in the same directory as the Feature Xtractor application.

2. Select and configure the desired features. In case of the brain and muscle features you have two operands to configure. Ask Starlab staff which operands to use for measuring particular states. In case you would like to configure then yourself see the following. For each operand, you can configure the frequency band on which the power will be calculated (Delta, Theta, Beta, Alpha, Gamma, Band1, Band2) and the channel on which this power will be measured. Band1 and Band2 are user-defined bands that you can configure in the spinboxes below. You can also configure the operation that will be applied between these two operands (summation, difference, multiplication, division or no operation). If No operation is selected, the second operand will be ignored. You can also write the name of the specific features (Relaxation, Workload, etc).

3. Click the "Open Enobio" button to connect with your device. Even if you are reading from a file, you should press first this button. This task may take some seconds to finish. If some error message appears, make sure your device is switched on or reset your bluetooth (or make sure it can find the specified file in case you are reading from a file).

4. Click the "Start EEG" button to start streaming your features.

Address and ports

You need to connect to different ports for getting the raw data stream or the feature stream. If you need both simultaneously access the raw data port and the feature port with two separate TCP clients. The IP address to which you need to connect is the IP address of the computer on which the Feature Xtractor is running and the ports are the following:

- Raw data port: 7126

- Feature port: 7125

Data format

For each new time sample of the raw data, you will receive a string with the following fields (note that '\t' corresponds to a tabulator and ChannelX will be a number):

Channel1\tChannel2\tChannel3\tChannel4\t....Channel8\tTimestamp\t

For each new feature sample, you will receive a string will the following fields:

Feature1\tFeature2\tFeature3\tFeature4\tFeature5\t

If you do not have all the features activated, you will only receive those features that are selected, in order or appearance.

File reproduction for development tests

You can develop your sonification / musicification using a recorded EEG data file as a tested, and then try a live data stream when it is ready. For this NE Feature Xtractor provides the functionality to stream the data from a file. In order to use this option you can use following files:

Brain Polyphony

Brain Polyphony is a stand alone software we have developed as a customization of NeuroSurfer. As such it is a complex software providing similar functionality as the NE Feature Xtractor. However there are some differences with it. The first one is the fact that it works for MS-Windows (tested on Windows 7 and 8) and Mac OS X (tested on 10.7.X). There also some differences in its functionality :

- It allows only to access 2 brain features at a time normalized in the range of [-1, 1].

- It provides access to heart activity features such as heart-rate, heart-rate variability and heart beats.

- It allows to access muscular features normalized in the range [-1,1].

- It provides also access to the data of the accelerometer integrated in the Enobio/StarStim device and detection of movements.

Measures

You can access following Brain and Body Measurements with the BrainPolyphony software:

  • Brain Features: 2 fixed measures associated with valence and arousal (ask Starlab staff for concrete configuration). This features are normalized via an initial baseline in between -1 and 1. Normalization limits can be customized after the baseline calculation.

  • Heart features: 3 measures associated to the heart activity, i.e. heart beats (0/1 at each beat), heart rate (range 40-150, ask for help if it gets out of these limits), and heart rate variability (in the range 0-120, which has been associated with arousal level).

  • Muscle features: Measure of muscular activity power at one Enobio/Starstim channel. Muscular features are normalized between -1 and 1 using an initial baseline. Normalization limits can be customized after the baseline calculation.

  • Movement features: Enobio/Starstim includes a three axis accelerometer that is used to measure movement. The module of the three axis is streamed as movement feature. If the measured movement exceeds a threshold customized by the user the system detects that a movement occured and streams a value equal to 1. If the movement doesn't exceed this threshold the streamed value is 0.

Instructions

Follow these steps in order to configure your feature xtractor:

1. Launch BrainPolyphony application and pair your Enobio/Starstim device.

2. Configure your EEG features, from which channels will muscular and heart activity be measured and the normalization baseline duration. Please contact Starlab for support in your initial parameterization of the system.

3. Click the 'On' button and visually check in the BrainPolyphony application the measured activity at every channel. Click 'Baseline' and wait still during the baseline calculation. After the baseline calculation the system will start the features streaming via TCP/IP.

4. In the EEG tab you can modify feature 1 and feature 2 (named as valence and arousal) normalization thresholds as well as the movement threshold. This tab provides visualization of the incoming EEG features and its normalized values.

5. In the ECG-EMG tab you can monitor the heart activity and change the muscular normalization threshold.This tab also provides visualization tools of the calculted heart and muscular features.

Address and ports

You need to connect to the IP address of the computer where BrainPolyphony software is running to the 1234 port. If you experience problems with the connection take into account that firewalls shall be disabled.

Data format

For each new time sample of the feature data, you will receive a string with the following fields (note that '\t' corresponds to a tabulator):

Valence\tArousal\tHR\tHRV\tHeartBeat\tEMG\tAccelerometer\tMovement

where each field corresponds to:

1 - Valence: Valence value measured in the EEG. Normalized [-1, 1]

2 - Arousal: Arousal value measured in the EEG. Normalized [-1, 1]

3 - HR: Heart rate in bpm

4 - HRV: Heart rate variability in bpm

5 - HeartBeat: 1 if a heart beat occurred, 0 otherwise

6 - EMG: Measued EMG value normalized in range [-1 1]

7 - Accelerometer: Acclerometer three axis module sqrt(x^2 + y^2 + z^2)

8 - Movement: 1 if movement was detected, 0 otherwise.

Example:

-0.184935 -0.220831 149.254 8.2436 1 -1 80.56 0

Application Development

For the development of any application the simplest option is to develop a TCP client in the framework of your choice. The TCP client will then access the data streaming from either NE Feature Xtractor, or BrainPolyphony. You can access as well the data streaming in OSC, through the Pure Data module we provide below.

If you would like to develop a BCI application, the best is to choose any of the standard BCI frameworks existent. We provide there a description of the integration with BCI2000, and OpenVibe.

TCP2OSC application

We provide HERE an application that can connect to either NE Feature Xtractor, or BrainPolyphony to translate the feature values they send to OSC messages. The binary to be run in MS-Windows can be downloaded from HERE. The code to be compiled in other platforms can be found HERE.

The syntax to run the tcp2osc is:

tcp2osc --option [IP address] [OSC IP address] [OSC port]

There are three available options:

--raw to parse raw data from Feature Server,

--feature to parse features from Feature Server, and

--bp to parse features from BreainPolyphony application.

[IP address] is the address where the Feature Server/BrainPolyphony runs (default value: "localhost")

[OSC IP address] is the address where the OSC messages are sent (default value: "239.200.200.200")

[OSC port] is the port where the OSC messages are sent (default value: 9977)


The OSC address patter with the --raw option is:

/channelN where N stands for the channel number from 1 to 8/20


The OSC address patter with the --feature option is:

/featureN where N stands for the feature ordinal from 1 to 5


The OSC address patter with the --bp option takes the following values:

/Valence /Arousal /HR /HRV /HeartBeat /EMG /Accel /Mov

Pure Data

Enobio/Starstim has been integrated with PureData, a real-time graphical programming environment for audio, video, and graphical processing.

Pure Data (aka Pd) is an open source visual programming language. Pd enables musicians, visual artists, performers, researchers, and developers to create software graphically, without writing lines of code. Pd is used to process and generate sound, video, 2D/3D graphics, and interface sensors, input devices, and MIDI.

PD has more powerful sound synthesis and computer music capabilities than Processing, but is not as strong on Interactivity and visualisation. We provide 2 different Pd patches for translating TCP to OSC, and for reading a OSC stream.

TCP to OSC conversion

The first Pd patch we provide is a translator from TCP to OSC. It is intended to receive the data streamed by the BrainPolyphony software, parse the TCP message, and translate it into an OSC one. </p>

TCP2OSC is a PD patch that provides an interface to the Enobio features streamed by the BrainPolyphony . The patch reads data streaming live from a TCP socket.

This patch provides:

  • capability to read a live streaming EEG features from a TCP socket connected to BrainPolyphony.
  • capability to stream the features in OSC format.
  • a visualisation of the data
  • Note that there is no sonification - thats your job.

Setup:

  • Download and install PD Extended - http://puredata.info
  • Downlad and unzip 'Enobio_Sonar2015.zip' from [1]
  • Open BP_Enobio_Server.pd
  • Connect to BrainPolyphony feature streaming by selecting 'Conectar_a_Enobio'.
  • Edit the IP address in patch enobio_TCP_client and write down the IP address of the computer running BrainPolyphony, leave the defaault port 1234 (ensure you are in the same LAN as the broadcasting BrainPolyphony)
  • In pd_monitor you should see now the numerical value of the received features
  • Select OSC_Multicast to start streaming the features in OSC with the following labelling: /Valence (EEG Feature 1), /Arousal (EEG Feature 2), /HR (Heart Rate), /HRV (Heart Rate Variability), /HeartBeat (Heart Beat), /EMG (Muscular Feature), /Accel (Accelerometer Module), /Mov (If movement was detected).

OSC client

The secod Pd patch we provide is a parser of the OSC messages sent by the former patch TCP2OSC. Together they are intended to receive the data streamed by the BrainPolyphony software, parse the OSC message streamed by TCP2OSC, and easily interface with your sonification. </p>

OSCclient is a PD patch that provides an interface to the Enobio features streamed in OSC. The patch reads data streaming live from a OSC socket via the TCP to OSC translation patch. One good option is to use the functionality of the NE Feature Xtractor to stream EEG data recorded in a file to make a first version of your sonification. You can thence work the life sonification using the direct data from an Enobio.

This patch provides:

  • capability to read a live streaming EEG features from an OSC socket connected to TCP2OSC, which should be on its turn connected to NE Feature Xtractor or BrainPolyphony.
  • Note that there is no sonification - thats your job.

Setup:

  • Download and install PD Extended - http://puredata.info
  • Downlad the PD OSC Client from [2]
  • Launch OSC_BP_Client and start receiving BrainPolyphony features with previously defined labels.

BrainPolyphony Sonification Patch

You can download a working example of Pure Data sonification in: File:MHD PureData 2015 2.zip developed by Jordi Sala from Mobility Lab during Sonar MHD 2015 and start listening to the music coming out of your body.

Processing

Enobio/Starstim has been integrated with Processing. Processing is an open source programming language and environment for people who want to create images, animations, and interactions. The integration can be easily done with aany TCP client that can be connected to either the NE Feature Xtractor or the BrainPolyphony.

Processing is particularly good for visualization and interactive interfaces. Minim is a simple synth that comes with Processing and brings some powerful signal processing functionality. You will need to upgrade to Minim 2.1.0 BETA (see below) in order to use the more sophisticated capabilities of the Ugens. You can find an example of Processing integration in SonifyNoiseBandsFromEnobioFile20Ch.

Matlab / Simulink Feature Xtractor

An example of MATLAB/Simulink receiving the TCP streaming can be found HERE. It receives data from the TCP from NIC and streams on another TCP port the calculated features through ASCII strings. The output features have been computed within Matlab. We offer as well output of MIDI sequences.

Sonification of EEG

General Description

Data sonifications have often been criticised as unpleasant and unmusical to listen to. As part of Music Hack Day 2013 in Barcelona it was explored the musical side of data sonification using an EEG dataset. It was explored the potential that data sonifications can be composed to create an emotional, aesthetic and cultural experience. Can these sonifications also provide insights into the EEG dataset used to compose them? Could sonification become a popular mass medium for audiences who tune in for both the listening experience and the information content?
"Sonification of EEG"

References:

  • "Barrass, Stephen; ",Sonifications for concert and live performance,AI & society,27,2,281-283,2012,Springer London
  • "Barrass, Stephen; ",The aesthetic turn in sonification towards a social and cultural medium,AI & society,27,2,177-181,2012,Springer-Verlag
  • "Barrass, Stephen; Vickers, Paul; ",Sonification design and aesthetics,,,,,2011,Logos Verlag - http://sonification.de/handbook/index.php/chapters/chapter7/
  • "Barrass, Stephen; Whitelaw, Mitchell; Bailes, Freya; ","Listening to the mind listening: an analysis of sonification reviews, designs and correspondences",Leonardo Music Journal,,,13-19,2006,MIT Press

Example: Play Your Mood

As a result of our technology provision in MHD 2013 Oscar ter Hofstede and Marcel Farrés developed a hack for translating electrophysiological features into commands for multimodal representation of emotional state with a graph and an audio track playing. This software was later refined by Marcel Farrés into the Play Your Mood.

Play Your Mood v2.0 is a music player that, after analyzing all your music library (using Essentia+Gaia), creates mood-related play-list and then reproduces it depending on your mood. The mood is approximated using real time Enobio/Emotiv EEG data or also can be choose by the user (switch between master/MoodPlayer).

You can find the code in GitHub.

Personal tools
Namespaces

Variants
Actions
BASIC USE
EEG APPLICATIONS
STIMULATION (tCS)
DATA ANALYSIS
ADVANCED
Toolbox