Flexible utilization of spatial- and motor-based codes for the storage of visuo-spatial information
- Neurosciences Graduate Program, University of California, San Diego, United States
- Department of Machine Learning, Carnegie Mellon University, United States
- Neuroscience Institute, Carnegie Mellon University, United States
- Department of Psychology, University of California, San Diego, United States
- Ernst Strüngmann Institute (ESI) for Neuroscience in Cooperation with Max Planck Society, Germany
- Kavli Foundation for the Brain and Mind, University of California, San Diego, United States
Figures
Figure 1 with 2 supplements
Task design, behavioral results, and univariate results.
(A) During each trial of the main working memory task, participants remembered the spatial position of a target dot presented at a random angular position (7° eccentricity from fixation). After a 16 s delay, participants made a binary response to indicate which half of a ‘response’ disk the target dot position had been presented on. At the beginning of the delay, a ‘preview’ disk was shown that either exactly matched the response disk (top row; ‘informative’) or had a random orientation relative to the response disk (bottom row; ‘uninformative’). In the example trial depicted, a participant in the informative condition would have pressed the button corresponding to the lighter gray side of the disk. By contrast, in the uninformative condition a participant would have pressed the button corresponding to the dark gray side, as only the final response disk was relevant to their response (see Methods, Task: Main working memory for more details). (B) Average behavioral accuracy (left) and response time (right) in the informative and uninformative conditions (individual participants are shown with gray lines). Error bars represent ±1 SEM across participants. Significance of condition differences was computed using paired t-tests (a single asterisk indicates p<0.05, and three asterisks indicate p<0.001). (C) In a separate spatial working memory mapping task, participants remembered a target dot position for 12 s and responded by moving a probe dot around an invisible circle to match the remembered position (see Methods, Task: Spatial working memory mapping). This mapping task was used to generate an independent dataset to train decoding models (see Methods, Analysis: Spatial position decoding). (D) Univariate hemodynamic response functions in three representative regions of interest from early visual cortex (V1), parietal cortex (IPS0), and motor cortex (M1) during the informative (dark blue) and uninformative (light blue) conditions of the main working memory task. Timepoint zero indicates the onset of the memory target. Shaded gray rectangles indicate the time periods when the ‘preview’ disk was onscreen (3.5–4.5 s) and when the response disk was onscreen (16.5–18.5 s). Shaded error bars represent ±1 SEM across participants. Gray dots indicate timepoints showing a significant condition difference (evaluated using a Wilcoxon signed-rank test with permutation, all p-values <0.05; see Methods, Analysis: Univariate for details). This plot shows three representative ROIs, see Figure 1—figure supplement 1 for data from all ROIs.
Figure 1—figure supplement 1
Hemodynamic response function in each region of interest (ROI) during the informative (dark blue) and uninformative (light blue) conditions, full set of ROIs.
Timepoint zero indicates the time of target onset; shaded gray rectangles indicate the time periods when the ‘preview’ disk was onscreen (3.5–4.5 s) and when the response disk was onscreen (16.5–18.5 s). Shaded error bars represent ±1 SEM across participants. Gray dots indicate timepoints showing a significant condition difference, evaluated using a Wilcoxon signed-rank test with permutation, see Methods, Analysis: Univariate for details.
Figure 1—figure supplement 2
Univariate responses in sensorimotor regions of interest (ROIs), separated by which finger was used to make a behavioral response.
(A) Informative condition. (B) Uninformative condition. For each voxel, hemodynamic response functions (HRFs) were computed separately for trials in each condition where the response finger was contralateral (blue) or ipsilateral (pink) relative to the voxel’s hemisphere. Contralateral and ipsilateral HRFs were then combined across all voxels in both hemispheres for this figure. Timepoint zero indicates the time of target onset; shaded gray rectangles indicate the time periods when the ‘preview’ disk was onscreen (3.5–4.5 s) and when the response disk was onscreen (16.5–18.5 s). Shaded error bars represent ±1 SEM across participants. Gray dots indicate timepoints showing a significant difference between contralateral and ipsilateral trials, evaluated using a Wilcoxon signed-rank test with permutation, see Methods, Analysis: Univariate for details.
Figure 2 with 3 supplements
Visual and parietal areas represent spatial position more strongly during trials that require sensory-like spatial memory than during trials that allow re-coding into action-oriented memory.
(A) Schematic of the decoding procedure. Continuous values of angular position were divided into eight discrete bins, and four binary decoders were trained to discriminate between patterns corresponding to bins 180° apart. The final decoding accuracy was the average accuracy over these four binary decoders. (B) Decoding accuracy for each region of interest (ROI) and task condition. The spatial decoder was always trained on data from the delay period of an independent spatial working memory mapping task (Figure 1C), and tested on data from the delay period of the main working memory task (averaged within a window 8–12.8 s from start of trial; see Methods, Analysis: Spatial position decoding for more details). Error bars reflect ±1 SEM across participants, and light gray lines indicate individual participants. Dots above bars and pairs of bars indicate the level of statistical significance within each condition, and between conditions, respectively (two-tailed p-values obtained using a Wilcoxon signed-rank test with permutation testing, see Methods, Analysis: Spatial position decoding). Dot sizes reflect significance level. (C) Spatial decoding accuracy over time in three example ROIs. Timepoint zero indicates the target onset time. Shaded gray rectangles indicate the periods of time when the ‘preview’ (3.5–4.5 s) and ‘response’ (16.5–18.5 s) disks were onscreen. Shaded error bars represent ±1 SEM across participants, colored dots indicate significance of decoding within each condition, and gray dots indicate significant condition differences, with dot sizes reflecting significance levels as in B. Gray brackets just above the x-axis in (C) indicate the time range over which data were averaged to produce (B) (i.e. 8–12.8 s). See Figure 2—figure supplement 1 for time-resolved spatial decoding in all visual and motor ROIs.
Figure 2—figure supplement 1
Time-resolved spatial decoding accuracy in every region of interest (ROI).
All decoding was done using the spatial working memory mapping task as a training set (see Methods, Analysis: Spatial position decoding for details). Timepoint zero indicates the time of target onset; shaded gray rectangles indicate the periods of time when the ‘preview’ disk was onscreen (3.5–4.5 s) and when the response disk was onscreen (16.5–18.5 s). Shaded error bars represent ±1 SEM across participants, colored dots indicate significance of decoding within each condition, and gray dots indicate significant condition differences, with dot sizes reflecting significance levels.
Figure 2—figure supplement 2
Spatial decoding performance differs across conditions, even when training and testing a decoder within each task condition separately.
See Methods, Analysis: Spatial position decoding for details on classification procedure. Notably, within-condition spatial decoding showed a highly similar pattern of results to the analysis using the independent training set (Figure 2B), though the condition differences were slightly smaller (two-way repeated measures ANOVA with ROI and task condition as factors: main effect of ROI: F(13,65) = 12.873, p<0.001; main effect of condition: F(1,5) = 11.461, p=0.017; ROI × condition interaction: F(13,65) = 2.581, p=0.011; p-values obtained using permutation test; see Methods). Error bars reflect ±1 SEM across participants, and light gray lines indicate individual participants. Dots above bars and pairs of bars indicate the statistical significance of decoding within each condition, and of condition differences, respectively, both evaluated using non-parametric statistics. Dot sizes reflect significance level.
Figure 2—figure supplement 3
Decoding accuracy for the orientation of the ‘preview’ disk stimulus (see Figure 1A).
Decoding accuracy for this stimulus, which briefly appeared on the screen early in the delay period, was similar across conditions (two-way repeated measures ANOVA with ROI and task condition as factors: main effect of ROI: F(13,65) = 13.4, p<0.001; main effect of task condition: F(1,5) = 2.164, p=0.193; ROI × condition interaction F(13,65) = 0.500, p=0.932; p-values obtained using permutation test; see Methods). Decoding was performed using data averaged over a time window 4.8–9.6 s into the trial, see Methods, Analysis: Disk orientation decoding for details. Error bars reflect ±1 SEM across participants, and light gray lines indicate individual participants. Dots above bars and pairs of bars indicate the statistical significance of decoding within each condition, and of condition differences, respectively, both evaluated using non-parametric statistics. Note that in this analysis, no significant condition differences were detected, and thus there are no dots above pairs of bars. Dot sizes reflect significance level.
Figure 3 with 1 supplement
Action-oriented memory representations can be decoded from sensorimotor regions of interest (ROIs) during the delay period.
(A) A linear decoder was trained to classify the finger (left or right index) associated with the correct motor action on each trial, using data measured during the delay period of trials in each task condition separately (averaged within a window 8–12.8 s from start of trial; see Methods, Analysis: Action decoding for more details). Error bars reflect ±1 SEM across participants, and light gray lines indicate individual participants. Dots above bars and pairs of bars indicate the statistical significance of decoding accuracy within each condition, and of condition differences, respectively (two-tailed p-values obtained using a Wilcoxon signed-rank test with permutation testing, see Methods, Analysis: Action decoding). Dot sizes reflect significance level. (B) Action decoding accuracy over time in three example ROIs. Timepoint zero indicates the target onset time. Shaded gray rectangles indicate the periods during which the ‘preview’ (3.5–4.5 s) and ‘response’ (16.5–18.5 s) disks were onscreen. Shaded error bars represent ±1 SEM across participants. Colored dots indicate significance of decoding accuracy within each condition, and gray dots indicate significant condition differences, with dot sizes reflecting significance levels as in A. Gray brackets just above the x-axis in (B) indicate the time range in which data were averaged to produce (A) (8–12.8 s). For time-resolved decoding in all ROIs, see Figure 3—figure supplement 1.
Figure 3—figure supplement 1
Time-resolved action decoding accuracy in every region of interest (ROI).
All decoding was done using data from the same task condition for training and testing (see Methods, Analysis: Action decoding for details). Timepoint zero indicates the time of target onset; shaded gray rectangles indicate the periods of time when the ‘preview’ disk was onscreen (3.5–4.5 s) and when the response disk was onscreen (16.5–18.5 s). Shaded error bars represent ±1 SEM across participants, colored dots indicate significance of decoding within each condition, and gray dots indicate significant condition differences, with dot sizes reflecting significance levels.
Figure 4
Perceptual and action-oriented memory representations generalize from signals associated with perceptual input and physical button-press responses, respectively.
(A) Decoding accuracy of spatial memory position when training on data from an independent sensory localizer (see Methods, Task: Spatial localizer, and Methods, Analysis: Spatial position decoding for details on this task and on decoding procedure). (B) Decoding accuracy of action-oriented memory codes (i.e. the finger associated with the correct motor action on each trial) when training on data from an independent button pressing task (see Methods, Task: Sensorimotor cortex localizer and Methods, Analysis: Action decoding for details on this task and on decoding procedure). In both panels, error bars reflect ±1 SEM across participants, and light gray lines indicate individual participants. Dots above bars and pairs of bars indicate the statistical significance of decoding accuracy within each condition, and of condition differences, respectively, both evaluated using non-parametric statistics. Dot sizes reflect significance level. Inserts show a cartoon of the localizer tasks used to train the decoder for these analyses.
Additional files
-
Supplementary file 1
Table listing the number of voxels in each ROI for each participant and hemisphere.
Sizes of retinotopic visual ROIs (V1-IPS3) are after thresholding with a spatial localizer (see Methods, Task: Spatial Localizer). S1, M1, and PMc were defined using a button-pressing task (see Methods, Task: Sensorimotor Cortex Localizer). All analyses in this paper were done using bilateral ROIs (i.e. concatenating the left and right hemispheres of each ROI).
- https://cdn.elifesciences.org/articles/75688/elife-75688-supp1-v1.xlsx
-
Transparent reporting form
- https://cdn.elifesciences.org/articles/75688/elife-75688-transrepform1-v1.docx
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Flexible utilization of spatial- and motor-based codes for the storage of visuo-spatial information
eLife 11:e75688.
https://doi.org/10.7554/eLife.75688
