  The purpose of this work is to propose a framework for the benchmarking of
EEG amplifiers, headsets, and electrodes providing objective recommendation for
a given application. The framework covers: data collection paradigm, data
analysis, and statistical framework. To illustrate, data was collected from 12
different devices totaling up to 6 subjects per device. Two data acquisition
protocols were implemented: a resting-state protocol eyes-open (EO) and
eyes-closed (EC), and an Auditory Evoked Potential (AEP) protocol.
Signal-to-noise ratio (SNR) on alpha band (EO/EC) and Event Related Potential
(ERP) were extracted as objective quantification of physiologically meaningful
information. Then, visual representation, univariate statistical analysis, and
multivariate model were performed to increase results interpretability.
Objective criteria show that the spectral SNR in alpha does not provide much
discrimination between systems, suggesting that the acquisition quality might
not be of primary importance for spectral and specifically alpha-based
applications. On the contrary, AEP SNR proved much more variable stressing the
importance of the acquisition setting for ERP experiments. The multivariate
analysis identified some individuals and some systems as independent
statistically significant contributors to the SNR. It highlights the importance
of inter-individual differences in neurophysiological experiments (sample size)
and suggests some device might objectively be superior to others when it comes
to ERP recordings. However, the illustration of the proposed benchmarking
framework suffers from severe limitations including small sample size and sound
card jitter in the auditory stimulations. While these limitations hinders a
definite ranking of the evaluated hardware, we believe the proposed
benchmarking framework to be a modest yet valuable contribution to the field.
