The effect of criterion shifts on neural measures of information processing under post hoc sorting. (A) When the criterion shifts to the right, the respondent becomes more conservative, whereas a leftward criterion shift reflects a more liberal response criterion. The average signal strength of both unseen and seen trials is greater under a conservative response criterion than under a liberal response criterion, both for seen and for unseen trials (diagonal line pointing leftward). (B) A simulation in which it is assumed that internal signal strength is a reflection of neural processing. When simulating signal and noise trials (histograms of signal and noise trials, left panel), a selection based on a conservative or liberal criterion would result in different average signal strengths, which would affect the sensitivity of a decoding analysis on neural data of signal vs noise for both seen and unseen trials, depicted as Area Under the Curve (AUC) in the right panel.

Two experiments with a criterion manipulation. (A) In Experiment 1, participants had to execute a detection task in 10-minute blocks by pressing a button whenever they perceived a square target (top stimulus in a continuous RSVP). In half of the blocks, misses were signaled by a tone and a small monetary deduction (liberal criterion condition) whereas in the other half of the blocks false alarms were signaled by a tone and a small monetary deduction (conservative criterion condition). (B) Same as in panel A, but now for Experiment 2, in which participants responded according to the PAS. (C and D) Behavior associated with Experiment 1 and 2. Both hits and false alarms increased under the liberal condition compared to the conservative condition, whereas sensitivity remained approximately the same. Note that in panel D, to be able to compute hit and false alarm rates the PAS was conceptualized as a type I response scale, with PAS0 indexing absent responses and PAS1-3 indexing present responses. Minor differences between experiments are detailed in the Methods.

Classification performance with and without post hoc sorting. (A) Classification performance of target present vs target absent time-locked to stimulus onset, expressed in Area Under the Curve (AUC), separately for Experiment 1 and Experiment 2, as well as average classification performance, shaded areas are standard error of the mean (left panel). The average performance across both experiments clearly shows three local classification performance peaks at 137 ms, 266 ms and 430 ms. The distribution of cortical activity at these peaks was highly similar for Experiment 1 and Experiment 2, as shown in topographic Current Source Density maps that were obtained from the forward transformed classification weights, obtained from training a classifier using all electrodes for visualization purposes (right panel). (B) Classification performance of target present vs target absent after post hoc sorting on ‘seen’ and ‘unseen’ responses (here collapsed across ‘seen’ and ‘unseen’, see Figure 4 for the uncollapsed data), separately for Experiment 1 and Experiment 2, and separately for the liberal and the conservative condition. (C) Same as in B, but this time performing classification analysis on all trials without first post hoc sorting into ‘seen’ and ‘unseen’ trials (using the same classifiers as used in B). See supplementary Figure 2 for the complete time series.

Differential effects on classification performance of post hoc sorting on either ‘seen’ or ‘unseen’ conditions. (A) Classification performance in Experiment 1 for the liberal and conservative condition, separately for ‘unseen’ trials (left panel) and for ‘seen’ trials (right panel). (B) Same as in A, but now for Experiment 2. (C-D). Simulations showing the effect of criterion shifts under low sensitivity or high sensitivity when the threshold is the same in C and D. When overall sensitivity is low (C), the effect of criterion shifts is more likely to appear in the ‘seen’ condition, whereas when sensitivity is high (D), the effect of criterion shifts is more likely to appear in the ‘unseen’ condition. (E-F) The effect of criterion shifts under an overall conservative or an overall liberal criterion when sensitivity is the same. When the overall criterion is conservative (E), the effect of criterion shifts is more likely to appear in the ‘seen’ condition, whereas when the overall criterion is liberal (F), the effect of criterion shifts is more likely to appear in the ‘unseen’ condition.

Repeated Measures ANOVA after post hoc sorting).

In cases where Mauchly’s test of sphericity was violated, the Greenhouse-Geisser corrected values are provided below it on the second row.

Repeated Measures ANOVA control (no post hoc sorting).

In cases where Mauchly’s test of sphericity was violated, the Greenhouse-Geisser corrected values are provided below it on the second row.

Repeated Measures ANOVA for experiment 1 (detection), after post hoc sorting.

There were no violations of Mauchly’s test of sphericity.

Repeated Measures ANOVA for experiment 2 (PAS), after post hoc sorting.

In cases where Mauchly’s test of sphericity was violated, the Greenhouse-Geisser corrected values are provided below it on the second row.

Decoding timelines of liberal and conservative with post hoc sorting (A) and without post hoc sorting, control analysis (B).

Graphical depiction of leave-one-person-out cross validation scheme. A classifier is always trained on all target versus no-target trials of a given participant and tested in a different participant on the various cells of any given experimental design. See methods for details.