A generalized cortical activity pattern at internally generated mental context boundaries during unguided narrative recall
- Department of Psychological and Brain Sciences, Johns Hopkins University, United States
Figures
Figure 1 with 3 supplements
Experimental procedures and univariate responses.
(A) In the encoding phase, subjects watched 10 short movies approximately 2–8 min long. Each movie started with a 6 s title scene. In the free spoken recall phase, subjects verbally recounted each movie plot in as much detail as possible regardless of the order of presentation. After recalling one movie, subjects spontaneously proceeded to the next movie, and the transitions between movies were considered as internally driven boundaries. Red arrows indicate the boundaries (onsets and offsets) between watched or recalled movies. Black arrows indicate the non-boundary moments (middle) of each watched or recalled movie. (B) Whole-brain maps of unthresholded mean activation (blood oxygen level-dependent [BOLD] signals z-scored across all volumes within a scanning run) following between-movie boundaries during recall (4.5–19.5 s from the offset of each movie). Blue areas indicate regions with lower-than-average activation, where the average activation of a scanning run was z = 0. Likewise, red areas indicate regions with higher-than-average activation. White outlines indicate areas that showed significantly lower or higher activation following between-movie boundaries compared to non-boundary periods (false discovery rate-corrected q < 0.05; minimum surface area = 16 mm2). The non-boundary periods were defined as the middle 15 s of each recalled movie, shifted forward by 4.5 s. Changes in whole-brain univariate responses across time around the boundaries are shown in Figure 1—video 1 (recall phase) and Figure 1—video 2 (encoding phase).
Figure 1—figure supplement 1
Mean activation time courses around between-movie boundaries.
For each subject and region of interest (top row = posterior medial cortex [PMC]; middle row = angular gyrus [ANG]; bottom row = auditory cortex [AUD]), blood oxygen level-dependent [BOLD] signals measured during the encoding phase (left column) or recall phase (right column) were locked to the offset of each watched or recalled movie, and then averaged across movies. Thin gray lines show individual subjects’ time courses. Thick black lines show the mean time courses averaged across all subjects. Red bars on the x-axis indicate the 15 s boundary period time window (4.5–19.5 s from the offset of each movie) used for subsequent analyses comparing the boundary and non-boundary periods.
-
Figure 1—figure supplement 1—source data 1
Source data for Figure 1—figure supplement 1.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig1-figsupp1-data1-v2.xlsx
Figure 1—video 1
Changes in univariate activation at between-movie boundaries during recall (video).
The animation shows the time series of whole-brain activation maps (blood oxygen level-dependent [BOLD] signals z-scored across all volumes within a scanning run) locked to the offset of the recall of each movie, from 30 s before to 45 s after the offset. Within each of the 7.5 s time windows shown as a red bar on the time axis, BOLD signals in each vertex were averaged across time points, movies, and subjects. Blue-cyan areas indicate regions with lower-than-average activation. Red-yellow areas indicate regions with higher-than-average activation.
Figure 1—video 2
Changes in univariate activation at between-movie boundaries during encoding (video).
The animation shows the time series of whole-brain activation maps (blood oxygen level-dependent [BOLD] signals z-scored across all volumes within a scanning run) locked to the offset of each movie clip during the encoding phase, from 30 s before to 45 s after the offset. Within each of the 7.5 s time windows shown as a red bar on the time axis, BOLD signals in each vertex were averaged across time points, movies, and subjects. Blue-cyan areas indicate regions with lower-than-average activation. Red-yellow areas indicate regions with higher-than-average activation.
Figure 2 with 2 supplements
Consistent activation patterns associated with between-movie boundaries.
(A) Schematic of the pattern similarity analysis. Boundary patterns were defined as the mean pattern averaged across 15 s following the offset of each watched or recalled movie. Non-boundary patterns were defined as the mean pattern averaged across 15 s in the middle of each watched or recalled movie. For each subject and cortical parcel (Schaefer et al., 2018; 200 parcels per hemisphere), we computed pairwise between-movie pattern similarity (Pearson correlation), separately for boundary patterns and non-boundary patterns measured during recall (a and b, blue arrows). We also computed between-movie and between-phase (encoding-recall) pattern similarity, again separately for boundary and non-boundary patterns (c and d, red arrows). The time windows for both boundary and non-boundary periods were shifted forward by 4.5 s to account for the hemodynamic response delay. (B) Whole-brain t statistic map of cortical parcels that showed consistent between-movie boundary patterns during recall. These parcels displayed significantly greater between-movie pattern similarity in the boundary condition compared to the non-boundary condition during recall. The map was masked by parcels that showed significantly positive between-movie pattern similarity in the boundary condition during recall. Both effects were Bonferroni corrected across parcels (p<0.05). (C) Whole-brain t statistic map of cortical parcels that showed consistent between-movie boundary patterns across encoding and recall. These parcels displayed significantly greater between-movie and between-phase pattern similarity in the boundary condition compared to the non-boundary condition. The map was masked by parcels that showed significantly positive between-movie and between-phase pattern similarity in the boundary condition. Both effects were Bonferroni corrected across parcels (p<0.05).
Figure 2—figure supplement 1
Consistent activation patterns during shorter (4.5 s) time windows following between-movie boundaries.
(A) Whole-brain t statistic map of cortical parcels that showed consistent between-movie boundary patterns during recall. These parcels displayed significantly greater between-movie pattern similarity in the boundary condition compared to the non-boundary condition during recall. The map was masked by parcels that showed significantly positive between-movie pattern similarity in the boundary condition during recall. Both effects were Bonferroni corrected across parcels (p<0.05). (B) Whole-brain t statistic map of cortical parcels that showed consistent between-movie boundary patterns across encoding and recall. These parcels displayed significantly greater between-movie and between-phase pattern similarity in the boundary condition compared to the non-boundary condition. The map was masked by parcels that showed significantly positive between-movie and between-phase pattern similarity in the boundary condition. Both effects were Bonferroni corrected across parcels (p<0.05). For both (A) and (B), boundary periods were defined as 4.5–9 s from the offset of each movie. Non-boundary periods were defined as the middle 4.5 s of each movie, shifted forward by 4.5 s to account for hemodynamic response delay.
Figure 2—figure supplement 2
Similar visual input cannot explain between-movie boundary patterns consistent across experimental phases.
(A) To test whether shared visual features (i.e., mostly blank black screen) produced boundary (offset) patterns consistent across encoding and recall, we performed a whole-brain pattern similarity analysis. For each subject and cortical parcel, we computed the mean correlation between boundary patterns across different movies and experimental phases (a, red arrow). We also computed the mean correlation between encoding boundary patterns and recall non-boundary (middle) patterns across different movies (c, blue arrow). Note that visual input (a fixation dot) was identical across boundary and non-boundary periods during recall. The duration of boundary and non-boundary periods was 15 s. (B) Whole-brain t statistic map of cortical parcels that showed greater pattern correlations between encoding and recall boundary patterns (a, red arrow) compared to correlations between encoding boundary patterns and recall non-boundary patterns (c, blue arrow). Bonferroni correction was applied across parcels to correct for multiple comparisons (p<0.05). Several parcels in higher associative cortices showed greater correlations between encoding and recall boundary patterns, suggesting that low-level visual features contributed little to the consistent boundary patterns in those areas.
Figure 3 with 4 supplements
Boundary pattern in the posterior medial cortex (PMC).
(A) PMC activation pattern similarity (Pearson correlation) between the 10 movie stimuli (M1–10), conditions (offset = boundary, middle = non-boundary), and experimental phases (encoding, recall), averaged across all subjects. The boundary pattern of a movie was defined as the mean pattern averaged across the 15 s window following the offset of the movie. The non-boundary pattern was defined as the mean pattern averaged across the 15 s window in the middle of a movie. The time windows for both boundary and non-boundary patterns were shifted forward by 4.5 s to account for the hemodynamic response delay. PMC regions of interest (ROIs) are shown as white areas on the inflated surface of a template brain. (B) Subject-specific mean activation patterns associated with between-movie boundaries during encoding (left) and recall (right). The boundary patterns were averaged across all movies and then z-scored across vertices within the PMC ROI mask, separately for each experimental phase. PMC (demarcated by black outlines) of four example subjects (S1–4) are shown on the medial surface of the right hemisphere of the fsaverage6 template brain. (C) Within-phase (recall-recall) and between-phase (encoding-recall) pattern similarity across different movies, computed separately for the boundary (offset) and non-boundary (middle) patterns in PMC. Bar graphs show the mean across subjects. Circles represent individual subjects. Error bars show SEM across subjects. ***p<.001. (D) Time-point-by-time-point PMC pattern similarity across the encoding phase and recall phase activation patterns around between-movie boundaries, averaged across all subjects. The time series of activation patterns were locked to either the onset (left) or the offset (right) of each movie. During encoding, the onset of a movie and the offset of the preceding movie were separated by a 6 s title scene. During recall, onsets and offsets of recalled movies were separated by, on average, a 9.3 s pause (boundaries concatenated across subjects, SD = 16.8 s). Dotted lines on the left and right panels indicate the mean offset times of the preceding movies and the mean onset times of the following movies, respectively. Note that in this figure zero corresponds to the true stimulus/behavior time, with no shifting for hemodynamic response delay. Areas outlined by black lines indicate correlations significantly different from zero after multiple comparisons correction (Bonferroni corrected p<0.05). Time–time correlations within each experimental phase can be found in Figure 3—figure supplement 4.
-
Figure 3—source data 1
Source data for Figure 3.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig3-data1-v2.xlsx
Figure 3—figure supplement 1
Subject-specific boundary patterns in the posterior medial cortex (PMC).
The between-movie boundary patterns were averaged across all movies and then z-scored across vertices within the PMC region of interest (ROI) mask, separately for the encoding phase and the recall phase. PMC of 11 subjects (S5–15) are shown on the medial surface of the right hemisphere of the fsaverage6 template brain.
Figure 3—figure supplement 2
Boundary pattern in the angular gyrus (ANG).
(A) ANG activation pattern similarity (Pearson correlation) between the 10 movie stimuli (M1–10), conditions (offset = boundary, middle = non-boundary), and experimental phases (encoding, recall), averaged across all subjects. The boundary pattern of a movie was defined as the mean pattern averaged across the 15 s window following the offset of the movie. The non-boundary pattern was defined as the mean pattern averaged across the 15 s window in the middle of a movie. The time windows for both boundary and non-boundary patterns were shifted forward by 4.5 s to account for the hemodynamic response delay. ANG regions of interest (ROIs) are shown as white areas on the inflated surface of a template brain. (B) Subject-specific mean activation patterns associated with between-movie boundaries during encoding (left) and recall (right). The boundary patterns were averaged across all movies and then z-scored across vertices within the ANG ROI mask, separately for each experimental phase. ANG (demarcated by black outlines) of four example subjects (S1–4) are shown on the lateral surface of the left hemisphere of the fsaverage6 template brain. (C) Within-phase (recall-recall) and between-phase (encoding-recall) pattern similarity across different movies, computed separately for the boundary (offset) and non-boundary (middle) patterns in ANG. Bar graphs show the mean across subjects. Circles represent individual subjects. Error bars show SEM across subjects. ***p<0.001. (D) Time-point-by-time-point ANG pattern similarity across the encoding phase and recall phase activation patterns around between-movie boundaries, averaged across all subjects. The time series of activation patterns were locked to either the onset (left) or the offset (right) of each movie. Dotted lines on the left and right panels indicate the mean offset times of the preceding movies and the mean onset times of the following movies, respectively. Note that in this figure zero corresponds to the true stimulus/behavior time, with no shifting for hemodynamic response delay. Areas outlined by black lines indicate correlations significantly different from zero after multiple comparisons correction (Bonferroni corrected p<0.05).
-
Figure 3—figure supplement 2—source data 1
Source data for Figure 3—figure supplement 2.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig3-figsupp2-data1-v2.xlsx
Figure 3—figure supplement 3
Boundary patterns in regions of interest measured during shorter (4.5 s) time windows.
Within-phase (recall-recall) and between-phase (encoding-recall) pattern similarity across different movies, computed separately for the boundary (offset) and non-boundary (middle) patterns in the posterior medial cortex (PMC; left panel) and the angular gyrus (ANG; right panel). Boundary periods were defined as 4.5–9 s from the offset of each movie. Non-boundary periods were defined as the middle 4.5 s of each movie, shifted forward by 4.5 s to account for hemodynamic response delay. Bar graphs show the mean across subjects. Circles represent individual subjects. Error bars show SEM across subjects. ***p<0.001.
-
Figure 3—figure supplement 3—source data 1
Source data for Figure 3—figure supplement 3.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig3-figsupp3-data1-v2.xlsx
Figure 3—figure supplement 4
Time–time pattern similarity in the posterior medial cortex (PMC).
The similarity matrices show Pearson correlations between PMC patterns across time points around between-movie boundaries during encoding (A, B) and recall (C, D), calculated within each subject and then averaged across all subjects. The time series of activation patterns were locked to either the onset (A, C) or the offset (B, D) of each movie. Dotted lines in (A) and (C) indicate the mean offset times of the preceding movies. Dotted lines in (B) and (D) indicate the mean onset times of the following movies. Note that in this figure zero corresponds to the true stimulus/behavior time, with no shifting for hemodynamic response delay. Areas outlined by black lines indicate correlations that significantly deviate from zero after multiple comparisons correction (Bonferroni corrected p<0.05). The boundary pattern emerged following the offsets but preceded the onsets of watched or recalled movies. In addition, the boundary pattern was stronger and lasted longer following encoding offsets compared to recall offsets; this may be because boundaries between movies were more salient during initial movie watching as they accompanied both external and internal mental context changes, whereas recall boundaries accompanied internal context changes only. Encoding boundaries were also more unpredictable and may require a more gradual build-up of the upcoming mental context, compared to self-generated boundaries between already stored memories during recall.
-
Figure 3—figure supplement 4—source data 1
Source data for Figure 3—figure supplement 4.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig3-figsupp4-data1-v2.xlsx
Figure 4 with 2 supplements
Comparing between-movie and within-movie boundary patterns in the posterior medial cortex (PMC).
(A) Schematic of the analysis. For each subject, we created the template PMC activation pattern associated with between-movie boundaries by averaging activation patterns following the offset of each between-movie boundary (orange bars), separately for encoding and recall phases. Likewise, the template within-movie event boundary pattern was created by averaging the activation patterns following the offset of each within-movie boundary during encoding (green bars). We then measured the similarity (Pearson correlation) between the mean between-movie boundary patterns during encoding and recall (a, orange arrow). We also measured the similarity between the mean within-movie boundary pattern during encoding and the mean between-movie boundary pattern during recall (b, green arrow). For both between- and within-movie boundaries, boundary periods were 15 s long, shifted forward by 4.5 s. (B) Pattern similarity between template boundary patterns. The orange bar shows the mean correlation across the between-movie boundary patterns during encoding and recall. The green bar shows the mean correlation across the between-movie boundary pattern during recall and the within-movie boundary pattern during encoding. Circles represent individual subjects. Error bars show SEM across subjects. ***p<0.001 against zero.
-
Figure 4—source data 1
Source data for Figure 4.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig4-data1-v2.xlsx
Figure 4—figure supplement 1
Comparing within-movie boundary patterns and non-boundary (middle) patterns in the posterior medial cortex (PMC) during encoding.
The mean correlation between within-movie boundary patterns across different movies (within-within) was greater than zero (t(14) = 5.23, p<0.001, Cohen’s dz = 1.35, 95% CI = [0.03,0.07]). The mean correlation between within-movie boundary patterns and non-boundary patterns across different movies (within-middle) was also greater than zero (t(14) = 3.73, p=0.002, Cohen’s dz = .96, 95% CI = [0.01,0.02]). Critically, within-movie boundary patterns were more similar to each other than to non-boundary patterns (within-within vs. within-middle; t(14) = 3.29, p=0.005, Cohen’s dz = .85, 95% CI of the difference = [0.01,0.06]). The duration of within-movie boundary and non-boundary periods was 15 s. Two non-boundary patterns that partially overlapped with the within-movie boundary patterns were excluded from analysis. Circles represent individual subjects. Error bars show SEM across subjects. **p<0.01, ***p<0.001.
-
Figure 4—figure supplement 1—source data 1
Source data for Figure 4—figure supplement 1.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig4-figsupp1-data1-v2.xlsx
Figure 4—figure supplement 2
Examining the effects of boundary period time windows on the between- and within-movie boundary pattern similarity in the posterior medial cortex (PMC).
(A) Pattern similarity between the template boundary patterns in PMC measured during a shorter (4.5 s) boundary period time window following the offset of each boundary. The orange bar shows the average correlation across the mean between-movie boundary patterns during encoding and recall. The green bar shows the average correlation across the mean between-movie boundary pattern during recall and the mean within-movie event boundary pattern during encoding. There was a strong positive correlation across the encoding and recall between-movie boundary patterns (t(14) = 15.08, p<0.001, Cohen’s dz = 3.89, 95% CI = [0.52,0.69]), whereas no such correlation was observed across the within- and between-movie boundary patterns (t(14) = 1.26, p=0.23, Cohen’s dz = .32, 95% CI = [–0.26,0.07]). Circles represent individual subjects. Error bars show SEM across subjects. ***p<0.001 against zero. (B) PMC pattern correlations across time points around between-movie boundaries during recall and within-movie event boundaries during encoding. The time series of activation patterns were locked to the offset of a movie or a prominent within-movie event. The correlations were first calculated within each subject and then averaged across all subjects. Time zero corresponds to the true stimulus/behavior time, with no shifting for hemodynamic response delay. Areas outlined by black lines indicate correlations that significantly deviate from zero after multiple comparisons correction (Bonferroni corrected p<0.05). No significant positive correlations were observed across encoding and recall immediately following the within- and between-movie boundaries.
-
Figure 4—figure supplement 2—source data 1
Source data for Figure 4—figure supplement 2.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig4-figsupp2-data1-v2.xlsx
Figure 5 with 2 supplements
Examining the effects of silence on the generalized boundary pattern.
For each subject, we computed a Pearson correlation between the mean activation pattern of the moments of silence during encoding (blue bars) and the mean activation pattern of between-movie boundaries during recall (orange bar) in the posterior medial cortex (PMC) and the auditory cortex (AUD). The moments of silence near between-movie boundaries (i.e., within the first 45 s of each movie) during encoding were excluded from the analysis. PMC and AUD regions of interest are shown as white areas on the inflated surface of template brains. Gray bars on the right panel indicate the mean pattern similarity across subjects. Circles represent individual subjects. Error bars show SEM across subjects. ***p<0.001 against zero.
-
Figure 5—source data 1
Source data for Figure 5.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig5-data1-v2.xlsx
Figure 5—figure supplement 1
Time series of audio amplitudes during encoding and the similarity to the recall boundary pattern.
(A) Audio amplitudes of the movie stimuli. The audio amplitudes were convolved with a hemodynamic response function and z-scored across time points. (B) Posterior medial cortex (PMC) pattern similarity (Pearson correlation) between each volume of encoding data and the template between-movie boundary pattern measured during recall (average of 4.5–19.5 s from the offset of each recalled movie). (C) Auditory cortex (AUD) pattern similarity between each volume of encoding data and the template between-movie boundary pattern measured during recall. In (A–C), the vertical dotted line in the middle indicates the boundary between the two encoding scanning runs. Vertical red lines indicate the offsets of each movie clip. In (B) and (C), black lines show the mean across subjects. Shaded areas indicate the standard deviation across subjects. Red dots mark time points that showed significantly positive pattern correlations after multiple comparisons correction across time points (Bonferroni corrected p<0.05).
-
Figure 5—figure supplement 1—source data 1
Source data for Figure 5—figure supplement 1.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig5-figsupp1-data1-v2.xlsx
Figure 5—figure supplement 2
Relationship between audio amplitudes during encoding and the similarity to the recall boundary pattern.
We computed correlations between the time series of audio amplitudes and the time series of similarity between the recall boundary pattern and each volume of encoding data in the posterior medial cortex (PMC) and the auditory cortex (AUD). Time points within each of the 10 movies after excluding the first 45 s were included in the analysis. PMC correlations were not significantly different from zero (t(14) = 1.7, p=0.11, Cohen’s dz = 0.44, 95% CI = [–0.04,0.004]), whereas AUD showed significantly negative correlations (t(14) = 18.66, p<0.001, Cohen’s dz = 4.82, 95% CI = [–0.26, –0.21]). Bar graphs show the mean across subjects. Circles represent individual subjects. Error bars show SEM across subjects. ***p<0.001 against zero.
-
Figure 5—figure supplement 2—source data 1
Source data for Figure 5—figure supplement 2.
- https://cdn.elifesciences.org/articles/73693/elife-73693-fig5-figsupp2-data1-v2.xlsx
Author response image 1
Author response image 2
Author response image 3
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
A generalized cortical activity pattern at internally generated mental context boundaries during unguided narrative recall
eLife 11:e73693.
https://doi.org/10.7554/eLife.73693
