EEG Models Inflated by Subject Overlap Between Training and Test Sets

When a deep learning model trains and tests on scalp EEG recordings from the same individuals, its reported accuracy says nothing about how it will perform on a new patient walking into the lab. That is the core of a new analysis published on Towards Data Science, which examines how subject overlap between training and test sets — a textbook case of data leakage — continues to distort performance claims across EEG machine learning studies.

The problem is structural, not incidental. EEG datasets are small by deep learning standards, and many pipelines split recordings into segments before partitioning those segments into train and test sets. The split looks random at the segment level. It is not random at the person level. Segments from the same subject, recorded in the same session on the same hardware with the same electrode montage, land on both sides of the divide. The model learns to recognize the person. It then gets credit for recognizing the condition.

For readers in clinical neurophysiology, this is a familiar failure mode in another costume. It parallels the situation in which an epileptologist reviewing a scalp EEG has already seen the same patient's previous recording: pattern recognition gets easier for reasons that have nothing to do with the pathology at hand.

What does subject leakage actually change?

Everything measurable. A model that has memorized subject-specific features — baseline rhythms, artifact signatures, electrode impedances, anatomically driven spatial patterns picked up by the montage — can achieve striking validation numbers while learning essentially nothing generalizable. Reported accuracy becomes an upper bound on memorization, not an estimate of clinical utility.

The stakes are concrete for practitioners evaluating published claims:

• A model trained and tested with subject overlap can look deployment-ready while failing on the first unseen patient.
• Cross-session and cross-site generalization, the properties that matter for a monitoring unit or reading room, remain untested in such setups.
• Headlines built on leaky splits cannot be benchmarked against studies using proper subject-wise cross-validation.

The correct methodology is subject-wise (or patient-wise) splitting: every recording from one individual stays on one side of the divide, and evaluation proceeds across people. This is the design any lab should demand before believing an accuracy figure, and the design any group submitting to a clinical journal should build in from the first experiment.

Why EEG makes this trap easy to fall into

EEG data magnifies the leakage problem relative to other modalities. Recordings are long, so one session yields thousands of segments, giving an illusion of dataset size that dissolves the moment you count distinct subjects. Between-subject variability in EEG is large — anatomy, age, electrode placement, vigilance state all shift the signal — which means subject-identifying information is rich and easy for a network to exploit.

The result is a systematic inflation of reported performance in the literature. Two studies of the same task, one with subject overlap and one without, can report numbers that differ dramatically while using near-identical architectures. Comparisons across papers become unreliable. Meta-analyses inherit the distortion.

What should labs and reviewers do now?

The practical fixes are cheap and immediate:

• Report the number of subjects, not just the number of segments or trials, in every methods section.
• Use leave-N-subjects-out cross-validation as the default evaluation scheme.
• State explicitly, in datasets and code releases, which subjects appear in which split.
• Treat previously published within-subject results as optimistic bounds, not benchmarks.

For journals and reviewers in clinical neurophysiology, the checklist is shorter still: no subject-wise split, no acceptance. A single line in the methods section — "data were split at the subject level" — should be a gating requirement for any EEG decoding claim.

Where this leaves the field

The analysis joins a growing push toward leakage-aware reporting in neural signal machine learning, at a moment when deep learning papers on EEG classification, emotion recognition, and brain–computer interfaces are appearing at high volume. None of these models is FDA-cleared on the strength of within-subject numbers, and none should be judged on them either. As multi-subject, multi-site EEG datasets become the standard testbed, the field's next step is clear: re-baseline existing tasks under strict subject-wise evaluation and find out which methods survive contact with truly unseen brains.

By Marcus Bennett