  We studied how lagged linear regression can be used to detect the physiologic
effects of drugs from data in the electronic health record (EHR). We
systematically examined the effect of methodological variations ((i) time
series construction, (ii) temporal parameterization, (iii) intra-subject
normalization, (iv) differencing (lagged rates of change achieved by taking
differences between consecutive measurements), (v) explanatory variables, and
(vi) regression models) on performance of lagged linear methods in this
context. We generated two gold standards (one knowledge-base derived, one
expert-curated) for expected pairwise relationships between 7 drugs and 4 labs,
and evaluated how the 64 unique combinations of methodological perturbations
reproduce gold standards. Our 28 cohorts included patients in Columbia
University Medical Center/NewYork-Presbyterian Hospital clinical database. The
most accurate methods achieved AUROC of 0.794 for knowledge-base derived gold
standard (95%CI [0.741, 0.847]) and 0.705 for expert-curated gold standard (95%
CI [0.629, 0.781]). We observed a 0.633 mean AUROC (95%CI [0.610, 0.657],
expert-curated gold standard) across all methods that re-parameterize time
according to sequence and use either a joint autoregressive model with
differencing or an independent lag model without differencing. The complement
of this set of methods achieved a mean AUROC close to 0.5, indicating the
importance of these choices. We conclude that time- series analysis of EHR data
will likely rely on some of the beneficial pre-processing and modeling
methodologies identified, and will certainly benefit from continued careful
analysis of methodological perturbations. This study found that methodological
variations, such as pre-processing and representations, significantly affect
results, exposing the importance of evaluating these components when comparing
machine-learning methods.
