Working memory (WM) is thought to provide a flexible mental workspace which allows organisms to hold information in mind about past experiences and use it to guide future behavior (Atkinson and Shiffrin, 1968; Baddeley and Hitch, 1974; Jonides et al., 2016). This system supports a wide range of cognitive tasks, each with its own specific demands and processing constraints. Due to these varied demands, it is likely that the neural mechanisms of WM are not universal across all tasks, but adaptively adjust to the requirements of the current situation. Among the many possible differences in task requirements, one key factor is the degree to which a task encourages a perceptual versus an action-oriented coding format. For instance, in tasks that require memory for fine visual details, such as searching for a specific kind of bird based on a picture in a guidebook, the best strategy might be to represent information in a format resembling past visual inputs (a perceptual, or ‘sensory-like’ code). Other tasks, such as remembering a series of directions for driving to your friend’s house, permit the use of multiple strategies. This driving task could be achieved using a perceptual coding strategy, such as maintaining a visuo-spatial representation of a street map. However, a better approach might be to re-code the visual information into another format, such as a series of motor plans for upcoming actions (an action-oriented, or ‘motor-like’ code). Such flexible re-coding of information from a more perceptual to a more action-oriented code could serve to reduce the dimensionality of representations when the correspondence between a remembered stimulus and a required action is known in advance. Thus, even tasks that are often thought to share a core component (e.g. memory in a visual format) might be accomplished via very different strategies and neural codes. Critically, past studies arguing that WM is supported by different neural loci often use tasks that differentially rely on perceptual versus action-oriented codes. As a result, evaluating claims about neural mechanisms is challenging without directly accounting for the potential impact of task demands in shaping how information is stored and used in WM.

The paradigms used in human neuroimaging experiments typically sit at the perceptual end of this continuum. In most fMRI studies of visual WM, participants are required to remember precise values of continuously varying features, and to report these remembered features using responses that cannot be pre-planned during the delay period (Albers et al., 2013; Christophel et al., 2012; Ester et al., 2009; Harrison and Tong, 2009; Lorenc et al., 2018; Rademaker et al., 2019; Serences et al., 2009; Xing et al., 2013). Such tasks may encourage top-down recruitment of the same early sensory areas that support high-precision perceptual representations (Awh and Jonides, 2001; Gazzaley and Nobre, 2012; Pasternak and Greenlee, 2005; Serences, 2016; Sreenivasan et al., 2014). Consistent with this idea of sensory recruitment, most studies find that patterns of voxel activation measured in visual cortex encode information about specific feature values held in memory, supporting the role of visual cortex in maintaining detailed visual representations during WM.

However, different neural loci are often implicated using visual WM tasks that enable action-oriented codes. One example is a memory-guided saccade task employed to study spatial WM using non-human primate (NHP) electrophysiological approaches. In these tasks, the position of a briefly presented cue is remembered, but that position is also predictive of the saccade that must be made at the end of the trial. Thus, the animal could solve the task by re-coding information from a spatial code into a motor code. Single unit recordings made during these tasks suggest an important role for the prefrontal cortex (PFC) in maintaining remembered information across brief delay periods (Funahashi et al., 1989; Fuster and Alexander, 1971; Goldman-Rakic, 1995). Consistent with these findings, others have suggested that action-oriented WM in general may rely more heavily on areas involved in planning and motor production, and less on early sensory cortex (Boettcher et al., 2021; Curtis et al., 2004; Myers et al., 2017). However, not all past studies have explicitly controlled the possibility for response-related remapping of information, making the boundary between perceptual and action-oriented codes unclear.

Based on the work discussed above, visual WM may be implemented quite flexibly, as opposed to having a singular neural locus or mechanism. Consistent with this idea, there are indications that even information related to the same stimulus might be stored at several different loci and in different formats (Iamshchinina et al., 2021; Lee et al., 2013; Rademaker et al., 2019; Serences, 2016). At the same time, few experiments have directly tested whether representations of memorized features in early visual cortex are subject to task modulation. Indeed, an alternative theory of WM storage suggests that memory-related signals in early sensory areas are epiphenomenal, in which case, task goals are not expected to affect the strength of representations in this part of the brain (Leavitt et al., 2017; Xu, 2020). Thus, our understanding of how behavioral requirements influence the memory networks and the codes that route information between early sensory and other brain regions is incomplete. In particular, it is not yet clear the extent to which perceptual and action-oriented codes may be jointly employed within the context of a single WM paradigm.

Here we study visuo-spatial WM to determine if the neural mechanisms supporting information storage are flexible in the face of changing behavioral requirements, or if, alternatively, the same mechanisms are recruited irrespective of the ability to employ different strategies. Participants performed a visuo-spatial WM task in which the required behavioral response on each trial was either not known until the end of the memory delay (‘uninformative’ condition), or known in advance of the memory delay (‘informative’ condition; Figure 1A). This manipulation was intended to promote the use of perceptual (‘sensory-like’, spatial) and action-oriented (‘motor-like’) memory codes, respectively. We then compared representations of perceptual and action-related mnemonic information in early visual, parietal, and sensorimotor cortical regions.

Figure 1 with 2 supplements see all

Download asset Open asset

To preview, we found that information about remembered spatial positions in retinotopic visual cortex decreased when the required response was known in advance, in accordance with a decreased reliance on visual cortex for information storage when action-oriented coding could be utilized. Furthermore, this decrease was accompanied by the emergence of an action-oriented memory representation in somatosensory, motor, and premotor cortex. Cross-generalization of decoding from independent model-training tasks further supports a shift from ‘sensory-like’ to ‘motor-like’ when comparing the two task conditions. These results demonstrate that the neural networks supporting WM – even in the context of a single paradigm often used to study visuo-spatial WM – can be strategically reconfigured depending on task requirements.

While in the magnetic resonance imaging (MRI) scanner, participants performed a spatial WM task in which they were required to remember the spatial position of a briefly presented target dot across a 16-s delay period. The small white target dot could appear anywhere along an invisible circle with radius of 7°. After the delay, participants reported the dot’s position by comparing their memory to a response probe – a disk with two halves (light and dark gray). They used their left or right index finger to indicate on which of the two halves of the disk the target dot had been presented (Figure 1A). We manipulated participants’ ability to pre-plan their motor action by presenting a ‘preview’ disk at the beginning of the delay period. The preview disk was preceded by a cue indicating whether the trial was part of the ‘informative’ or ‘uninformative’ condition. On informative trials, the preview disk matched the response disk, allowing participants to anticipate their button press at the end of the delay. On uninformative trials, the preview disk orientation was random with respect to that of the response disk, requiring that participants maintain a precise representation of the target dot position. Informative and uninformative trials were randomly intermixed throughout each run.

Task performance was overall better on trials where participants could plan their motor action in advance compared with trials when they could not (Figure 1B). Participants were significantly faster in the informative condition (informative mean ± SEM: 0.57 ± 0.03 s; uninformative: 1.08 ± 0.06 s; t₍₅₎ = –9.618; p<0.001) and also more accurate in the informative condition (informative mean ± SEM: 93.92 ± 2.12%; uninformative: 89.83 ± 1.12%; t₍₅₎=3.463; p=0.018). These behavioral benefits suggest that participants used the preview disk to pre-plan their motor action in the informative condition.

Next, we examined the average fMRI responses in both visual and sensorimotor cortical areas, which were each independently localized (see Methods, Identifying regions of interest). Our visual ROIs were retinotopically defined areas in occipital and parietal cortex, and our sensorimotor ROIs were action-related regions of primary somatosensory cortex (S1), primary motor cortex (M1), and premotor cortex (PMc). We used linear deconvolution to calculate the average hemodynamic response function (HRF) for voxels in each ROI during each task condition (see Methods, Analysis: Univariate; Dale, 1999; Dale and Buckner, 1997).

As expected, the BOLD signal in all retinotopic ROIs increased following visual stimulation (Figure 1D, left two panels; Figure 1—figure supplement 1). We also replicated the typical finding that occipital retinotopic areas V1–hV4 do not show sustained BOLD activation during the memory delay period (Offen et al., 2009; Riggall and Postle, 2012; Serences et al., 2009; Harrison and Tong, 2009) in either of our two task conditions. Also as expected, mean BOLD responses in parietal areas IPS0-3 showed elevated late delay period activation in the uninformative condition relative to the informative condition. This pattern is consistent with the use of a spatial code in the uninformative condition and a motor code in the informative condition (Curtis and D’Esposito, 2003; D’Esposito, 2007; Ester et al., 2015; Riggall and Postle, 2012). Sensorimotor areas S1, M1, and PMc showed a condition difference in the opposite direction, with higher mean BOLD responses in the informative compared with the uninformative condition at several timepoints early in the delay period, consistent with an increased reliance on an action-oriented memory code (Calderon et al., 2018; Donner et al., 2009). Note that these univariate results include voxels from both hemispheres. As expected, reliable interhemispheric differences can also be observed in sensorimotor areas, with higher activation in the hemisphere contralateral to the planned button press in the informative condition (Figure 1—figure supplement 2).

The univariate results described above are consistent with the idea of an information ‘handoff’ between cortical regions as a function of perceptual and action-oriented task strategies. This brings up the important question of how such a task-related shift is reflected in population-level representations across the brain, and whether it coincides with a shift from a more sensory-like spatial memory code to a more motor-like memory code. To evaluate this question, we used a multivariate linear classifier to decode the angular spatial position of the remembered dot, based on multivoxel activation patterns measured in each ROI during the delay period (Figure 2). By assessing spatial decoding accuracy in each ROI, we could look at the extent to which the underlying neural code reflected a sensory-like visual representation of memorized spatial position.

Figure 2 with 3 supplements see all

Download asset Open asset

To facilitate an independent comparison of decoding accuracy between the informative and the uninformative task conditions in the main WM task, we used data from an independent spatial WM task to train the classifier (Sprague et al., 2019; Figure 1C; see Methods, Task: Spatial working memory mapping). Before performing the decoding analysis, we subtracted from each single-trial activation pattern the mean across voxels on the same trial. This was done to ensure that any condition-specific changes in the mean BOLD responses did not contribute to differences in classification accuracy (see Methods, Analysis: Spatial position decoding). We then sorted the continuous angular positions into eight non-overlapping bins and used a decoding scheme with four binary classifiers, where each binary classifier was independently trained to discriminate between spatial positions that fell into bins separated by 180° (see Figure 2A and Rademaker et al., 2019). The final decoding accuracy for each task condition reflects the average decoding accuracy across all four of these binary classifiers, where chance is 50% (see Methods, Analysis: Spatial position decoding).

The results of this multivariate analysis demonstrate that across the uninformative and the informative conditions, spatial decoding accuracy was strongest in early visual areas V1, V2, V3, and V3AB, and became progressively weaker at more anterior regions of the visual hierarchy (Figure 2B). Spatial decoding accuracy was at chance in the three sensorimotor areas S1, M1, and PMc. Importantly, there was also a pronounced effect of task condition (Figure 2B and C). Decoding accuracy in most retinotopic areas was significantly higher for the uninformative condition, in which participants were forced to rely on a spatial memory code, compared to the informative condition, in which participants could convert to an action-oriented memory for the button press required at the end of the delay (two-way repeated measures ANOVA with ROI and task condition as factors: main effect of ROI: F_(13,65) = 24.548, p<0.001; main effect of task condition: F_(1,5) = 35.537, p=0.001; ROI × condition interaction F_(13,65) = 5.757, p<0.001; p-values obtained using permutation test; see Methods, Analysis: Spatial position decoding). Pairwise comparisons showed that spatial decoding accuracy was significantly higher in the uninformative compared to the informative condition in V1–hV4, LO2, and IPS0. In later IPS subregions (IPS1-3), spatial decoding accuracy was above chance in the uninformative memory condition, but at chance in the informative condition, without a significant difference between conditions. Finally, time-resolved analyses revealed that this difference between the conditions was not present during and immediately after encoding of the target dot, when the trial condition was not yet known. Instead, condition differences emerged approximately 5–6 s after the presentation of the preview disk and persisted until the final response disk appeared (Figure 2C, Figure 2—figure supplement 1). Importantly, the difference in spatial decoding accuracy between task conditions replicated when we trained and tested a decoder using data from within each condition of the main WM task as opposed to training on the independent spatial WM task. This indicates that the difference between task conditions was robust to a range of analysis choices (Figure 2—figure supplement 2; see Methods, Analysis: Spatial position decoding).

In addition to demonstrating the robustness of our findings, the two separate analysis approaches described above allow us to distinguish between two potential explanations for the observed condition difference. While one possibility is that visual cortex activation patterns contain less overall information about spatial position in the informative relative to the uninformative condition, another possibility is that the format of the representations differs between the two task conditions (e.g. Lorenc et al., 2020; Vaziri-Pashkam and Xu, 2017). Specifically, compared to the informative condition, the sensory-like spatial format used in the uninformative condition may have been more similar to the format of the independent spatial WM task (which also required a sensory-like strategy because the starting position of the response dot was unknown until the end of the trial; see Figure 1C). This similarity could have led to higher cross-generalization of the decoder to the uninformative condition, even if the overall spatial information content was similar between the two conditions in the main WM task. The fact that the condition difference persists when training and testing within condition rules this latter possibility out. Instead, our analyses support the interpretation that spatial WM representations in visual cortex are adaptively enhanced in the uninformative condition (when a sensory-like code is required) relative to the informative condition (where an action-oriented code can be used).

Is the quality of mnemonic spatial representations selectively modulated between task conditions, or might non-mnemonic factors – such as global arousal level – contribute to the observed differences in decoding accuracy? If non-mnemonic factors play a role, they should interact with the processing of all stimuli shown after the condition cue, including the ‘preview disk’ stimulus (see Figure 1A). A supplementary analysis revealed that the orientation of the preview disk (i.e. the boundary between its light- and dark-gray sides) could be decoded with above chance accuracy in several visual cortex ROIs in both conditions, but without a significant difference between conditions (Figure 2—figure supplement 3; see Methods, Analysis: Preview disk orientation decoding for details). This indicates that neural processing of the preview disk, a visual stimulus physically presented on the screen, was not modulated by task condition. Therefore, the effect of task condition on information content was specific to the spatial memory representation itself, and not due to a more global modulatory effect or difference in the signal-to-noise ratio of representations in early visual cortex.

In addition to spatial position decoding, we investigated the possibility of an action-oriented format of the mnemonic code by decoding upcoming actions. Since motor responses were always made with the left or the right index finger, we trained a binary linear classifier to predict which finger was associated with the required button-press on each trial based on delay period activation patterns in each ROI (Figure 3A, see Methods, Analysis: Action decoding for detailed classification procedure). This analysis revealed above chance decoding of upcoming actions in each of the three sensorimotor ROIs (S1, M1, and PMc) in the informative condition but not the uninformative condition. In contrast, all retinotopic visual ROIs showed chance level action decoding for both conditions (two-way repeated measures ANOVA with ROI and task condition as factors: main effect of ROI: F_(13,65) = 4.003, p<0.001; main effect of condition F_(1,5) = 3.802, p=0.106; ROI × condition interaction: F_(13,65) = 2.937, p=0.001; p-values obtained using permutation test; see Methods, Analysis: Action decoding). A time-resolved decoding analysis revealed that, in the informative condition, information about a participants’ upcoming action began to emerge approximately 4 s after the onset of the preview disk stimulus, decreased slightly toward the end of the delay period, then rose steeply after the response disk onset when the participant actually executed a motor action (Figure 3B, Figure 3—figure supplement 1). The increase in action decoding accuracy during the last part of the trial appeared sooner for the informative condition than the uninformative condition, in agreement with the speeding of behavioral response times in the informative condition (Figure 1B).

Figure 3 with 1 supplement see all

Download asset Open asset

Together, the above results support the hypothesis that the informative and uninformative conditions differentially engaged regions of cortex and neural coding formats for information storage during WM. To characterize this difference in coding format in a more concrete way, we next leveraged two additional independent model training tasks. First, we used a spatial localizer during which participants viewed high-contrast flickering checkerboards at various spatial positions (see Methods, Task: Spatial localizer). With this dataset, we trained a classifier on sensory-driven visuo-spatial responses (using the same spatial position decoding method as described previously, see Figure 2A). We found that spatial position information generalized from this sensory-driven perceptual training data to both conditions of our main WM task (Figure 4A). This result supports the idea that the coding format used during spatial WM is sensory-like in nature, closely resembling the responses in visual cortex to spatially localized perceptual input. Importantly, decoding performance in early visual and parietal areas was again higher for the uninformative than the informative condition. This implies that the uninformative condition of our task resulted in a stronger and more ‘sensory-like’ memory code. The second independent model training task was a sensorimotor cortex localizer where participants physically pressed buttons with their left and right index fingers (see Methods, Task: Sensorimotor cortex localizer). We trained a decoder on this task, labeling trials according to which finger was used to physically press the buttons on each trial. We then tested this decoder on delay period activation in our main WM task. This analysis revealed above-chance decoding accuracy of the participant’s upcoming action in S1, M1, and PMc, in the informative condition only (Figure 4B). This result supports the idea that the memory representations in the informative condition were stored in a more motor-like format which closely resembled signals measured during actual button press responses.

Figure 4

Download asset Open asset

Past studies of visual WM have led to competing theories of how information is stored in the brain. Some work has emphasized the recruitment of early sensory areas, while other work has emphasized parietal and frontal association areas. Nevertheless, a general consensus exists that WM is a flexible resource (Baddeley and Hitch, 1974; Gazzaley and Nobre, 2012; Iamshchinina et al., 2021; Serences, 2016). Here we demonstrate, within a single spatial WM paradigm, that task requirements are a critical determinant of how and where WM is implemented in the brain. These data provide a partial unifying explanation for divergent prior findings that implicate different regions in visual WM (Bettencourt and Xu, 2015; Ester et al., 2016; Iamshchinina et al., 2021; Rademaker et al., 2019; Xu, 2018; Xu, 2020). More importantly, however, our data show that WM flexibly engages different cortical areas and coding formats, even in the context of a task that is commonly used to study a single underlying construct (i.e. visuo-spatial WM). These findings highlight the goal-oriented nature of WM: the brain’s mechanisms for storage are not fully specified by the type of information being remembered, but instead depend on how the encoded information will be used to guide future behavior.

We used a spatial WM task wherein participants could anticipate their behavioral response ahead of the delay interval on half of the trials (Figure 1A), encouraging re-coding of visuo-spatial memory information to a more action-oriented representational format. On the other half of trials, participants were unable to anticipate their behavioral response, and had to rely on a sensory-like spatial memory representation. The decoding accuracy of a remembered position from activation patterns in visual cortex was lower when participants had the opportunity to anticipate their eventual motor action, compared to when they had to rely exclusively on sensory-like visual memory. This effect was consistent across a range of early visual and parietal ROIs (Figure 2), supporting the idea that the recruitment of visual cortex for WM storage is task dependent. Conversely, during the same task condition where visuo-spatial representations became weaker in visual cortex, sensorimotor ROIs showed above-chance decoding of the participant’s planned action during the delay period (Figure 3). We additionally show that the format of these two memory representations was markedly different: the spatial memory code generalized from an independent task in which stimuli at various spatial positions were actively viewed, whereas the action-related memory code generalized from an independent task in which participants made actual alternating button-press responses (Figure 4). Together, these results demonstrate that rather than having one fixed locus or mechanism, visual WM can be supported by different loci and coding schemes that are adapted to current task requirements.

While we found widespread decoding of the contents of WM, we generally did not find sustained increases in univariate BOLD responses during the delay (Figure 1D; Figure 1—figure supplement 1). Specifically, visual areas V1–hV4 and LO2 showed no sustained BOLD responses or differences in the mean BOLD signal between conditions during the delay period, despite these areas showing condition differences in spatial decoding accuracy. This adds to an existing body of research showing a dissociation between univariate and multivariate indices of WM storage (Emrich et al., 2013; Ester et al., 2015; Harrison and Tong, 2009; Riggall and Postle, 2012; Serences et al., 2009). It additionally suggests that the observed differences in decoding performance cannot be explained by global changes in signal due to arousal or task difficulty, but are instead attributable to differences in the amount of information represented within population-level patterns of activation in visual cortex. Furthermore, we found that information about the preview disk (see Figure 1A), a physical stimulus that appeared briefly on the screen early in the delay period, was not significantly different between task conditions (Figure 2—figure supplement 3). This finding further highlights that the observed modulation of spatial decoding accuracy in visual cortex was specific to the memory representation, and not due to a non-specific change in signal-to-noise ratio within early visual areas.

At the same time, in multiple subregions of IPS, we did observe higher univariate delay period activation for the uninformative condition relative to the informative condition. In contrast, the opposite was true in sensorimotor areas (Figure 1D; Figure 1—figure supplement 1). This finding parallels previous work showing higher univariate delay period activation in IPS during a task where oculomotor responses were decoupled from spatial memory items (i.e. encouraging a more perceptual strategy), whereas oculomotor areas showed higher activation when responses and memoranda were yoked (i.e. encouraging a more action-oriented strategy; Curtis et al., 2004). This past work suggested specialized networks for perceptual ‘sensory-like’ versus action-oriented ‘motor-like’ memories. Our current demonstration of a shift from visual to sensorimotor codes further builds on this by showing how the neural systems underlying WM may be flexibly reconfigured, and memory representations flexibly reformatted, based on behavioral requirements.

While information about the remembered spatial position was substantially lower in the informative condition than the uninformative condition, decoding accuracy did not fall entirely to chance in early visual cortex during informative cue trials (Figure 2B). With the possible exception of area V1, this was true even when looking at decoding at the very end of the WM delay period (Figure 2C, Figure 2—figure supplement 1). Several factors may have contributed to this above chance decoding accuracy. First, it is possible that on some fraction of informative trials, participants failed to properly encode or process the condition cue, and as a result maintained a high-precision spatial representation throughout the delay period. Such lapses of attention are possible, given that the conditions in our task were randomly interleaved on a trial-by-trial basis. Second, it is possible that participants faithfully followed the informative cue, and made use of an action-oriented strategy, but some amount of visuo-spatial memory information was nonetheless maintained in visual cortex. This would suggest that the recruitment of sensory-like codes in visual cortex is at least partially obligatory when performing a visuo-spatial WM task. This account is consistent with past findings showing that items which are not immediately relevant for a task, but will be relevant later (i.e. ‘unattended memory items’), can still be decoded from early visual cortex (Iamshchinina et al., 2021; re-analysis of data from Christophel et al., 2018; but see also Lewis-Peacock et al., 2012; LaRocque et al., 2017). However, this account is inconsistent with past findings showing that when a once-maintained item is no longer needed for behavior, it is no longer decodable from early visual areas (Harrison and Tong, 2009; Lewis-Peacock et al., 2012; Sprague et al., 2014; Sprague et al., 2016; Lorenc et al., 2020). In our paradigm, we cannot rule out that – even on trials where participants could completely discard spatial memory information in favor of an action-oriented code – some amount of visuo-spatial information was still maintained in visual cortex, reflecting a partially distributed code. Some support for the idea that sensory- and action-like codes can be maintained and accessed simultaneously during a memory task comes from a study by van Ede et al., 2019a, who showed such simultaneity using EEG during a task where both action-related and sensory-like codes were relevant for performance. Further experiments will be needed to determine how such distributed codes may be engaged during memory tasks in general, and how the extent to which a neural code is more distributed versus more punctate may change as a function of task requirements.

We observed above-chance delay period decoding of participants’ planned actions in S1, M1, and PMc in the informative condition. Each of these areas has previously been shown to exhibit preparatory motor activity in the context of tasks involving delayed reaching or finger pressing (Ariani et al., 2022; Calderon et al., 2018; Cisek and Kalaska, 2005; Donner et al., 2009). In our data, information regarding the upcoming action became detectable around 4 s after the onset of the preview disk – roughly the same time that spatial decoding accuracy in visual cortex dropped in the informative compared to the uninformative condition (Figure 2C and Figure 3B). While suggestive, the temporal resolution of fMRI does not allow us to draw firm conclusions about the relative timing of each process. Nevertheless, these results are broadly consistent with a theory in which action selection unfolds in parallel to task-related processing of visual input (Cisek and Kalaska, 2010; Donner et al., 2009; Klein-Flügge and Bestmann, 2012; van Ede et al., 2019a). Additionally, we found that information about upcoming actions declined toward the end of the delay period, dropping to chance just before the response disk appeared (Figure 3B). This time course is consistent with a transient signal related to the initial formation of an action-oriented memory representation, and aligns with recent EEG findings in human motor cortex following a retro-cue in a visual WM task (Boettcher et al., 2021).

Past work has investigated the influence of action-oriented coding on WM representations. For instance, it has been shown that high priority items, meaning those that are relevant for upcoming actions, tend to be represented more robustly than items that are not immediately relevant to behavior (Christophel et al., 2012; Lewis-Peacock et al., 2012; Lorenc et al., 2020; Rose et al., 2016; Sprague et al., 2016; but see also Barbosa et al., 2021; Iamshchinina et al., 2021). Prioritized WM representations can be further reconfigured so that they are optimized for future behavior, which may include the activation of circuits related to motor output (Myers et al., 2017; Nobre and Stokes, 2019; Schneider et al., 2017; Souza and Oberauer, 2016; van Ede et al., 2019a). These past studies of action preparation have typically used relatively coarse measures of visual information content such as decoding which of two items was currently selected. In contrast, here we measured the information about a continuous remembered feature value that was reflected in patterns of activation in visual cortex. We simultaneously showed a decrease in spatial information in visual cortex and an increase in action-related information in sensorimotor cortex when an action-oriented mnemonic format was encouraged. This finding solidifies the evidence showing that information may shift between qualitatively different kinds of codes as a function of behavioral requirements, as opposed to just waxing and waning over time within the same format/brain areas.

Past work on saccadic eye movements further supports the link between the neural circuits that support WM and those that support action preparation and execution. For instance, prior work has demonstrated that when retrieving or selecting an item within WM, eye movements exhibit a systematic bias toward the spatial position at which the item was originally presented, even without a physical stimulus to guide eye movements (Spivey and Geng, 2001; Ferreira et al., 2008; van Ede et al., 2019b; van Ede et al., 2020). These eye movements may play a functional role in retrieving items from WM (Ferreira et al., 2008; van Ede et al., 2019b) and can index processes such as attentional selection within WM (van Ede et al., 2019b; van Ede et al., 2020). In other work, the interaction between memory and action has been shown in the opposite direction, where the prioritization of a feature or location for motor actions can lead to prioritization in WM (Heuer et al., 2020). For example, when participants make a saccade during the delay period of a visual WM task, memory performance is enhanced for items originally presented at the saccade target location (Ohl and Rolfs, 2020). This effect occurs even when saccade target locations conflict with top-down attentional goals, suggesting it reflects an automatic mechanism for selection based on action planning (Ohl and Rolfs, 2020; Heuer et al., 2020). Finally, the planning of both memory-guided saccades and anti-saccades is associated with topographically specific activation in early visual cortex measured with fMRI, suggesting that neural representations of upcoming actions are coded similarly to representations used for WM storage (Saber et al., 2015; Rahmati et al., 2018). Together, these findings suggest an important functional association between WM and motor planning within the domain of eye movement control. Our work expands upon this idea by demonstrating the engagement of motor circuits for the maintenance of action-oriented WM representations using a paradigm outside the realm of eye movement planning. Additionally, while prior work suggested a close correspondence between perceptual and action-oriented WM codes, our work demonstrates that these two types of representations can also be strategically dissociated depending on task demands and the nature of the motor response being planned (i.e. a manual response versus a saccadic response).

Finally, beyond the impact of more perceptual and action-oriented WM strategies, other aspects of task requirements have been shown to influence the neural correlates of WM. For example, one experiment found that information about visual objects could be decoded from extrastriate visual areas on blocks of trials that required memory for visual details, but from PFC on blocks of trials where only the object’s category had to be maintained (Lee et al., 2013). Similarly, another experiment showed that instructing participants to maintain items in a visual, verbal, or semantic code resulted in format-specific patterns of neural activation measured with fMRI (Lewis-Peacock et al., 2015). These findings, along with our results, further support a framework in which behavioral requirements strongly influence the format of WM representations.

In our action-decoding analyses (Figure 3 and Figure 4B), we interpret the decoded representations as primarily reflecting the planning of upcoming actions and maintenance of an action plan in WM, rather than reflecting overt motor actions. This interpretation is justified because participants were explicitly instructed to withhold any physical finger movements or button presses until the end of the delay period, when the probe disk appeared. We observed participants throughout their behavioral training and scanning sessions to ensure that no noticeable finger movements were performed. Nevertheless, we cannot rule out the possibility that some amount of low-grade motor activity (such as a subtle increase in pressure on the button box, or increased muscle tone in the finger) occurred during the delay period in the informative condition and contributed to our decoding results. Further experiments incorporating a method such as electromyography recordings from the finger muscles would be necessary to resolve this. At the same time, this possibility of low-grade motor activity does not substantially change the interpretation of our results, namely that the representation of upcoming actions in motor, premotor, and supplementary motor cortex reflects the maintenance of a prospective action plan within WM. As detailed in the preceding paragraphs, a large body of work has drawn a link between action preparation and WM in the brain, with motor circuits often actively engaged by the selection and manipulation of items within WM, and motor actions in turn influencing the quality of memory representations. Based on this framework, the action-oriented representations in our task, independent of whether they may reflect a contribution from physical motor activity, are within the range of representations that can be defined as WM.

In contrast to human neuroimaging experiments, which often find evidence for sensory recruitment (i.e. WM decoding from early sensory cortices), research with NHPs has generally found less evidence for maintenance of feature-specific codes in early sensory regions. The difference in results between human and primate studies may be partially accounted for by differences in measurement modality. For example, the BOLD response is driven in part by modulations of local field potentials (Boynton, 2011; Goense and Logothetis, 2008; Logothetis et al., 2001; Logothetis and Wandell, 2004), which may contain information about remembered features, such as motion direction, even when single unit spike rates do not (Mendoza-Halliday et al., 2014). Part of the discrepancy may also be related to differences in tasks used in human studies versus NHP studies. For example, some NHP studies use tasks that allow for motor coding strategies, such as the traditional memory-guided saccade paradigm (Funahashi et al., 1989; Fuster and Alexander, 1971; Goldman-Rakic, 1995). At the same time, other NHP experiments have dissociated sensory and action-related information (Funahashi et al., 1993; Mendoza-Halliday et al., 2014; Miller et al., 1996; Panichello et al., 2019), suggesting that the difference between perceptual and action-oriented coding alone cannot account for differences between studies. Another aspect of task design that may contribute to these differences is that NHP tasks often require animals to remember one of a small, discrete set of stimuli (e.g. Funahashi et al., 1993; Mendoza-Halliday et al., 2014; Miller et al., 1996). This may encourage a more discretized categorical representation rather than a highly detailed representation, which may rely less strongly on the finely tuned neural populations of early sensory areas (Lee et al., 2013). For example, some NHP electrophysiology studies have shown spiking activity in V1 that reflects the contents of WM using tasks that require precise spatial representations that may not be easily re-coded into a non-sensory format (e.g. the curve-tracing task used by Van Kerkoerle and colleagues; Supèr et al., 2001; van Kerkoerle et al., 2017).

Together, our findings suggest that the neural mechanisms that underlie WM are dynamic and can be flexibly adjusted, even in the context of a single paradigm typically used to study visuo-spatial WM. When our participants were given the possibility to switch from a purely visual to a motor-based WM code, the amount of information regarding precise spatial position dropped in early visual and parietal cortex, while information about an upcoming action became decodable from sensorimotor cortex. This experiment highlights how in a single paradigm, we can measure multiple dissociable mechanisms supporting WM. More broadly, these results open the door for future experiments to explore other task factors that may dynamically alter how WM is represented in the brain, and further push the boundaries of our knowledge about WM and cognitive flexibility.

The dataset from this manuscript is publicly available on Open Science Framework at https://osf.io/te5g2/. All code associated with the manuscript is publicly available on GitHub at https://github.com/mmhenderson/wm_flex, (copy archived at swh:1:rev:a25485d9983bd13b99f99f98d24b0faeaff50005).

1. 1. Henderson MM
   2. Rademaker RL
   3. Serences JT

   (2022) Open Science Framework

   ID 10.17605/OSF.IO/TE5G2. Open Data 2022: Flexible utilization of spatial- and motor-based codes for the storage of visuo-spatial information.

   https://osf.io/te5g2/

1. 1. Albers AM
   2. Kok P
   3. Toni I
   4. Dijkerman HC
   5. de Lange FP

   (2013) Shared representations for working memory and mental imagery in early visual cortex

   Current Biology 23:1427–1431.

   https://doi.org/10.1016/j.cub.2013.05.065

   • PubMed
   • Google Scholar
2. 1. Andersson JLR
   2. Skare S
   3. Ashburner J

   (2003) How to correct susceptibility distortions in spin-echo echo-planar images: application to diffusion tensor imaging

   NeuroImage 20:870–888.

   https://doi.org/10.1016/S1053-8119(03)00336-7

   • PubMed
   • Google Scholar
3. 1. Ariani G
   2. Pruszynski JA
   3. Diedrichsen J

   (2022) Motor planning brings human primary somatosensory cortex into action-specific preparatory states

   eLife 11:e69517.

   https://doi.org/10.7554/eLife.69517

   • Google Scholar
4. 1. Atkinson RC
   2. Shiffrin RM

   (1968)

   Human Memory: A Proposed System and its Control Processes

   Psychology of Learning and Motivation 2:89–195.

   • Google Scholar
5. 1. Awh E
   2. Jonides J

   (2001) Overlapping mechanisms of attention and spatial working memory

   Trends in Cognitive Sciences 5:119–126.

   https://doi.org/10.1016/s1364-6613(00)01593-x

   • PubMed
   • Google Scholar
6. 1. Baddeley AD
   2. Hitch G

   (1974) Working memory

   Psychology of Learning and Motivation 8:47–89.

   https://doi.org/10.1016/S0079-7421(08)60452-1

   • Google Scholar
7. Preprint

   1. Barbosa J
   2. Soldevilla DL
   3. Compte A

   (2021) Unattended Short-Term Memories Are Maintained in Active Neural Representations

   PsyArXiv.

   https://psyarxiv.com/qv6fu/

   • Google Scholar
8. 1. Bettencourt KC
   2. Xu Y

   (2015) Decoding the content of visual short-term memory under distraction in occipital and parietal areas

   Nature Neuroscience 19:150–157.

   https://doi.org/10.1038/nn.4174

   • Google Scholar
9. 1. Boettcher SEP
   2. Gresch D
   3. Nobre AC
   4. van Ede F

   (2021) Output planning at the input stage in visual working memory

   Science Advances 7:13.

   https://doi.org/10.1126/sciadv.abe8212

   • Google Scholar
10. 1. Boynton GM

    (2011) Spikes, BOLD, attention, and awareness: A comparison of electrophysiological and fMRI signals in V1

    Journal of Vision 11:12.

    https://doi.org/10.1167/11.5.12

    • PubMed
    • Google Scholar
11. 1. Brainard DH

    (1997)

    The Psychophysics Toolbox

    Spatial Vision 10:433–436.

    • PubMed
    • Google Scholar
12. 1. Brodmann K

    (1909) Physiology of the cortex as an organ

    Brodmann’s Localisation in the Cerebral Cortex 1:239–262.

    https://doi.org/10.1007/b138298

    • Google Scholar
13. 1. Calderon CB
    2. Van Opstal F
    3. Peigneux P
    4. Verguts T
    5. Gevers W

    (2018) Task-Relevant Information Modulates Primary Motor Cortex Activity Before Movement Onset

    Frontiers in Human Neuroscience 12:93.

    https://doi.org/10.3389/fnhum.2018.00093

    • PubMed
    • Google Scholar
14. 1. Christophel TB
    2. Hebart MN
    3. Haynes JD

    (2012) Decoding the contents of visual short-term memory from human visual and parietal cortex

    The Journal of Neuroscience 32:12983–12989.

    https://doi.org/10.1523/JNEUROSCI.0184-12.2012

    • PubMed
    • Google Scholar
15. 1. Christophel TB
    2. Iamshchinina P
    3. Yan C
    4. Allefeld C
    5. Haynes JD

    (2018) Cortical specialization for attended versus unattended working memory

    Nature Neuroscience 21:494–496.

    https://doi.org/10.1038/s41593-018-0094-4

    • PubMed
    • Google Scholar
16. 1. Cisek P
    2. Kalaska JF

    (2005) Neural Correlates of Reaching Decisions in Dorsal Premotor Cortex: Specification of Multiple Direction Choices and Final Selection of Action

    Neuron 45:801–814.

    https://doi.org/10.1016/j.neuron.2005.01.027

    • PubMed
    • Google Scholar
17. 1. Cisek P
    2. Kalaska JF

    (2010) Neural Mechanisms for Interacting with a World Full of Action Choices

    Annual Review of Neuroscience 33:269–298.

    https://doi.org/10.1146/annurev.neuro.051508.135409

    • PubMed
    • Google Scholar
18. 1. Curtis CE
    2. D’Esposito M

    (2003) Persistent activity in the prefrontal cortex during working memory

    Trends in Cognitive Sciences 7:415–423.

    https://doi.org/10.1016/s1364-6613(03)00197-9

    • PubMed
    • Google Scholar
19. 1. Curtis CE
    2. Rao VY
    3. D’Esposito M

    (2004) Maintenance of spatial and motor codes during oculomotor delayed response tasks

    The Journal of Neuroscience 24:3944–3952.

    https://doi.org/10.1523/JNEUROSCI.5640-03.2004

    • PubMed
    • Google Scholar
20. 1. Dale AM
    2. Buckner RL

    (1997) Selective averaging of rapidly presented individual trials using fMRI

    Human Brain Mapping 5:329–340.

    https://doi.org/10.1002/(SICI)1097-0193(1997)5:5<329::AID-HBM1>3.0.CO;2-5

    • PubMed
    • Google Scholar
21. 1. Dale AM

    (1999)

    Optimal experimental design for event-related fMRI

    Human Brain Mapping 8:109–114.

    • PubMed
    • Google Scholar
22. 1. Dale AM
    2. Fischl B
    3. Sereno MI

    (1999) Cortical Surface-Based Analysis

    NeuroImage 9:179–194.

    https://doi.org/10.1006/nimg.1998.0395

    • PubMed
    • Google Scholar
23. 1. Donner TH
    2. Siegel M
    3. Fries P
    4. Engel AK

    (2009) Buildup of choice-predictive activity in human motor cortex during perceptual decision making

    Current Biology 19:1581–1585.

    https://doi.org/10.1016/j.cub.2009.07.066

    • PubMed
    • Google Scholar
24. 1. D’Esposito M

    (2007) From cognitive to neural models of working memory

    Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 362:761–772.

    https://doi.org/10.1098/rstb.2007.2086

    • PubMed
    • Google Scholar
25. 1. Emrich SM
    2. Riggall AC
    3. Larocque JJ
    4. Postle BR

    (2013) Distributed patterns of activity in sensory cortex reflect the precision of multiple items maintained in visual short-term memory

    The Journal of Neuroscience 33:6516–6523.

    https://doi.org/10.1523/JNEUROSCI.5732-12.2013

    • PubMed
    • Google Scholar
26. 1. Engel SA
    2. Glover GH
    3. Wandell BA

    (1997) Retinotopic organization in human visual cortex and the spatial precision of functional MRI

    Cerebral Cortex (New York, N.Y) 7:181–192.

    https://doi.org/10.1093/cercor/7.2.181

    • PubMed
    • Google Scholar
27. 1. Ester EF
    2. Serences JT
    3. Awh E

    (2009) Spatially global representations in human primary visual cortex during working memory maintenance

    The Journal of Neuroscience 29:15258–15265.

    https://doi.org/10.1523/JNEUROSCI.4388-09.2009

    • PubMed
    • Google Scholar
28. 1. Ester EF
    2. Sprague TC
    3. Serences JT

    (2015) Parietal and Frontal Cortex Encode Stimulus-Specific Mnemonic Representations during Visual Working Memory

    Neuron 87:893–905.

    https://doi.org/10.1016/j.neuron.2015.07.013

    • PubMed
    • Google Scholar
29. 1. Ester EF
    2. Rademaker RL
    3. Sprague TC

    (2016) How Do Visual and Parietal Cortex Contribute to Visual Short-Term Memory?

    ENeuro 3:150–157.

    https://doi.org/10.1523/ENEURO.0041-16.2016

    • PubMed
    • Google Scholar
30. 1. Ferreira F
    2. Apel J
    3. Henderson JM

    (2008) Taking a new look at looking at nothing

    Trends in Cognitive Sciences 12:405–410.

    https://doi.org/10.1016/j.tics.2008.07.007

    • PubMed
    • Google Scholar
31. 1. Fischl B
    2. Rajendran N
    3. Busa E
    4. Augustinack J
    5. Hinds O
    6. Yeo BTT
    7. Mohlberg H
    8. Amunts K
    9. Zilles K

    (2008) Cortical folding patterns and predicting cytoarchitecture

    Cerebral Cortex (New York, N.Y) 18:1973–1980.

    https://doi.org/10.1093/cercor/bhm225

    • PubMed
    • Google Scholar
32. 1. Fulton JF

    (1935) A note on the definition of the “motor” and “premotor” areas

    Brain 58:311–316.

    https://doi.org/10.1093/brain/58.2.311

    • Google Scholar
33. 1. Funahashi S
    2. Bruce CJ
    3. Goldman-Rakic PS

    (1989) Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex

    Journal of Neurophysiology 61:331–349.

    https://doi.org/10.1152/jn.1989.61.2.331

    • PubMed
    • Google Scholar
34. 1. Funahashi S
    2. Chafee MV
    3. Goldman-Rakic PS

    (1993) Prefrontal neuronal activity in rhesus monkeys performing a delayed anti-saccade task

    Nature 365:753–756.

    https://doi.org/10.1038/365753a0

    • PubMed
    • Google Scholar
35. 1. Fuster JM
    2. Alexander GE

    (1971) Neuron activity related to short-term memory

    Science (New York, N.Y.) 173:652–654.

    https://doi.org/10.1126/science.173.3997.652

    • PubMed
    • Google Scholar
36. 1. Gazzaley A
    2. Nobre AC

    (2012) Top-down modulation: bridging selective attention and working memory

    Trends in Cognitive Sciences 16:129–135.

    https://doi.org/10.1016/j.tics.2011.11.014

    • PubMed
    • Google Scholar
37. 1. Goense JBM
    2. Logothetis NK

    (2008) Neurophysiology of the BOLD fMRI signal in awake monkeys

    Current Biology 18:631–640.

    https://doi.org/10.1016/j.cub.2008.03.054

    • PubMed
    • Google Scholar
38. 1. Goldman-Rakic PS

    (1995) Cellular basis of working memory

    Neuron 14:477–485.

    https://doi.org/10.1016/0896-6273(95)90304-6

    • PubMed
    • Google Scholar
39. 1. Greve DN
    2. Fischl B

    (2009) Accurate and robust brain image alignment using boundary-based registration

    NeuroImage 48:63–72.

    https://doi.org/10.1016/j.neuroimage.2009.06.060

    • PubMed
    • Google Scholar
40. 1. Harrison SA
    2. Tong F

    (2009) Decoding reveals the contents of visual working memory in early visual areas

    Nature 458:632–635.

    https://doi.org/10.1038/nature07832

    • PubMed
    • Google Scholar
41. 1. Henderson M
    2. Serences JT

    (2019) Human frontoparietal cortex represents behaviorally relevant target status based on abstract object features

    Journal of Neurophysiology 121:1410–1427.

    https://doi.org/10.1152/jn.00015.2019

    • PubMed
    • Google Scholar
42. Software

    1. Henderson M

    (2022) wm_flex, version a25485d

    GitHub.

    https://github.com/mmhenderson/wm_flex
43. 1. Heuer A
    2. Ohl S
    3. Rolfs M

    (2020) Memory for action: a functional view of selection in visual working memory

    Visual Cognition 28:388–400.

    https://doi.org/10.1080/13506285.2020.1764156

    • Google Scholar
44. 1. Iamshchinina P
    2. Christophel TB
    3. Gayet S
    4. Rademaker RL

    (2021) Essential considerations for exploring visual working memory storage in the human brain

    Visual Cognition 29:425–436.

    https://doi.org/10.1080/13506285.2021.1915902

    • Google Scholar
45. 1. Jenkinson M
    2. Smith S

    (2001) A global optimisation method for robust affine registration of brain images

    Medical Image Analysis 5:143–156.

    https://doi.org/10.1016/s1361-8415(01)00036-6

    • PubMed
    • Google Scholar
46. 1. Jenkinson M
    2. Bannister P
    3. Brady M
    4. Smith S

    (2002) Improved optimization for the robust and accurate linear registration and motion correction of brain images

    NeuroImage 17:825–841.

    https://doi.org/10.1016/s1053-8119(02)91132-8

    • PubMed
    • Google Scholar
47. 1. Jenkinson M
    2. Beckmann CF
    3. Behrens TEJ
    4. Woolrich MW
    5. Smith SM

    (2012) FSL

    NeuroImage 62:782–790.

    https://doi.org/10.1016/j.neuroimage.2011.09.015

    • PubMed
    • Google Scholar
48. 1. Jerde TA
    2. Curtis CE

    (2013) Maps of space in human frontoparietal cortex

    Journal of Physiology, Paris 107:510–516.

    https://doi.org/10.1016/j.jphysparis.2013.04.002

    • PubMed
    • Google Scholar
49. 1. Jonides J
    2. Lacey SC
    3. Nee DE

    (2016) Processes of Working Memory in Mind and Brain

    Current Directions in Psychological Science 14:2–5.

    https://doi.org/10.1111/j.0963-7214.2005.00323.x

    • Google Scholar
50. 1. Klein-Flügge MC
    2. Bestmann S

    (2012) Time-dependent changes in human corticospinal excitability reveal value-based competition for action during decision processing

    The Journal of Neuroscience 32:8373–8382.

    https://doi.org/10.1523/JNEUROSCI.0270-12.2012

    • PubMed
    • Google Scholar
51. 1. Kleiner M
    2. Brainard DH
    3. Pelli DG
    4. Broussard C
    5. Wolf T
    6. Niehorster D

    (2007) What’s new in Psychtoolbox-3? A free cross-platform toolkit for psychophysiscs with Matlab and GNU/Octave

    In Cognitive and Computational Psychophysics 36:v070821.

    https://doi.org/10.1068/v070821

    • Google Scholar
52. 1. LaRocque JJ
    2. Riggall AC
    3. Emrich SM
    4. Postle BR

    (2017) Within-Category Decoding of Information in Different Attentional States in Short-Term Memory

    Cerebral Cortex (New York, N.Y) 27:4881–4890.

    https://doi.org/10.1093/cercor/bhw283

    • PubMed
    • Google Scholar
53. 1. Leavitt ML
    2. Mendoza-Halliday D
    3. Martinez-Trujillo JC

    (2017) Sustained Activity Encoding Working Memories: Not Fully Distributed

    Trends in Neurosciences 40:328–346.

    https://doi.org/10.1016/j.tins.2017.04.004

    • PubMed
    • Google Scholar
54. 1. Lee SH
    2. Kravitz DJ
    3. Baker CI

    (2013) Goal-dependent dissociation of visual and prefrontal cortices during working memory

    Nature Neuroscience 16:997–999.

    https://doi.org/10.1038/nn.3452

    • PubMed
    • Google Scholar
55. 1. Lewis-Peacock JA
    2. Drysdale AT
    3. Oberauer K
    4. Postle BR

    (2012) Neural evidence for a distinction between short-term memory and the focus of attention

    Journal of Cognitive Neuroscience 24:61–79.

    https://doi.org/10.1162/jocn_a_00140

    • PubMed
    • Google Scholar
56. 1. Lewis-Peacock JA
    2. Drysdale AT
    3. Postle BR

    (2015) Neural Evidence for the Flexible Control of Mental Representations

    Cerebral Cortex (New York, N.Y) 25:3303–3313.

    https://doi.org/10.1093/cercor/bhu130

    • PubMed
    • Google Scholar
57. 1. Logothetis NK
    2. Pauls J
    3. Augath M
    4. Trinath T
    5. Oeltermann A

    (2001) Neurophysiological investigation of the basis of the fMRI signal

    Nature 412:150–157.

    https://doi.org/10.1038/35084005

    • PubMed
    • Google Scholar
58. 1. Logothetis NK
    2. Wandell BA

    (2004) Interpreting THE BOLD signal

    Annual Review of Physiology 66:735–769.

    https://doi.org/10.1146/annurev.physiol.66.082602.092845

    • PubMed
    • Google Scholar
59. 1. Lorenc ES
    2. Sreenivasan KK
    3. Nee DE
    4. Vandenbroucke ARE
    5. D’Esposito M

    (2018) Flexible Coding of Visual Working Memory Representations during Distraction

    The Journal of Neuroscience 38:5267–5276.

    https://doi.org/10.1523/JNEUROSCI.3061-17.2018

    • PubMed
    • Google Scholar
60. 1. Lorenc ES
    2. Vandenbroucke ARE
    3. Nee DE
    4. de Lange FP
    5. D’Esposito M

    (2020) Dissociable neural mechanisms underlie currently-relevant, future-relevant, and discarded working memory representations

    Scientific Reports 10:11195.

    https://doi.org/10.1038/s41598-020-67634-x

    • PubMed
    • Google Scholar
61. 1. Mendoza-Halliday D
    2. Torres S
    3. Martinez-Trujillo JC

    (2014) Sharp emergence of feature-selective sustained activity along the dorsal visual pathway

    Nature Neuroscience 17:1255–1262.

    https://doi.org/10.1038/nn.3785

    • PubMed
    • Google Scholar
62. 1. Miller EK
    2. Erickson CA
    3. Desimone R

    (1996)

    Neural mechanisms of visual working memory in prefrontal cortex of the macaque

    The Journal of Neuroscience 16:5154–5167.

    • PubMed
    • Google Scholar
63. 1. Myers NE
    2. Stokes MG
    3. Nobre AC

    (2017) Prioritizing Information during Working Memory: Beyond Sustained Internal Attention

    Trends in Cognitive Sciences 21:449–461.

    https://doi.org/10.1016/j.tics.2017.03.010

    • PubMed
    • Google Scholar
64. 1. Nobre AC
    2. Stokes MG

    (2019) Premembering Experience: A Hierarchy of Time-Scales for Proactive Attention

    Neuron 104:132–146.

    https://doi.org/10.1016/j.neuron.2019.08.030

    • PubMed
    • Google Scholar
65. 1. Offen S
    2. Schluppeck D
    3. Heeger DJ

    (2009) The role of early visual cortex in visual short-term memory and visual attention

    Vision Research 49:1352–1362.

    https://doi.org/10.1016/j.visres.2007.12.022

    • PubMed
    • Google Scholar
66. 1. Ohl S
    2. Rolfs M

    (2020) Bold moves: Inevitable saccadic selection in visual short-term memory

    Journal of Vision 20:11.

    https://doi.org/10.1167/jov.20.2.11

    • PubMed
    • Google Scholar
67. 1. Panichello MF
    2. DePasquale B
    3. Pillow JW
    4. Buschman TJ

    (2019) Error-correcting dynamics in visual working memory

    Nature Communications 10:1–11.

    https://doi.org/10.1038/s41467-019-11298-3

    • PubMed
    • Google Scholar
68. 1. Pasternak T
    2. Greenlee MW

    (2005) Working memory in primate sensory systems

    Nature Reviews. Neuroscience 6:97–107.

    https://doi.org/10.1038/nrn1603

    • PubMed
    • Google Scholar
69. 1. Penfield W
    2. Boldrey E

    (1937) Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation

    Brain 60:389–443.

    https://doi.org/10.1093/brain/60.4.389

    • Google Scholar
70. 1. Rademaker RL
    2. Chunharas C
    3. Serences JT

    (2019) Coexisting representations of sensory and mnemonic information in human visual cortex

    Nature Neuroscience 22:1336–1344.

    https://doi.org/10.1038/s41593-019-0428-x

    • PubMed
    • Google Scholar
71. 1. Rahmati M
    2. Saber GT
    3. Curtis CE

    (2018) Population Dynamics of Early Visual Cortex during Working Memory

    Journal of Cognitive Neuroscience 30:219–233.

    https://doi.org/10.1162/jocn_a_01196

    • PubMed
    • Google Scholar
72. 1. Riggall AC
    2. Postle BR

    (2012) The relationship between working memory storage and elevated activity as measured with functional magnetic resonance imaging

    The Journal of Neuroscience 32:12990–12998.

    https://doi.org/10.1523/JNEUROSCI.1892-12.2012

    • PubMed
    • Google Scholar
73. 1. Rose NS
    2. LaRocque JJ
    3. Riggall AC
    4. Gosseries O
    5. Starrett MJ
    6. Meyering EE
    7. Postle BR

    (2016) Reactivation of latent working memories with transcranial magnetic stimulation

    Science (New York, N.Y.) 354:1136–1139.

    https://doi.org/10.1126/science.aah7011

    • PubMed
    • Google Scholar
74. 1. Saber GT
    2. Pestilli F
    3. Curtis CE

    (2015) Saccade Planning Evokes Topographically Specific Activity in the Dorsal and Ventral Streams

    Journal of Neuroscience 35:245–252.

    https://doi.org/10.1523/JNEUROSCI.1687-14.2015

    • PubMed
    • Google Scholar
75. 1. Schneider D
    2. Barth A
    3. Wascher E

    (2017) On the contribution of motor planning to the retroactive cuing benefit in working memory: Evidence by mu and beta oscillatory activity in the EEG

    NeuroImage 162:73–85.

    https://doi.org/10.1016/j.neuroimage.2017.08.057

    • PubMed
    • Google Scholar
76. 1. Serences JT
    2. Ester EF
    3. Vogel EK
    4. Awh E

    (2009) Stimulus-Specific Delay Activity in Human Primary Visual Cortex

    Psychological Science 20:207–214.

    https://doi.org/10.1111/j.1467-9280.2009.02276.x

    • PubMed
    • Google Scholar
77. 1. Serences JT

    (2016) Neural mechanisms of information storage in visual short-term memory

    Vision Research 128:53–67.

    https://doi.org/10.1016/j.visres.2016.09.010

    • PubMed
    • Google Scholar
78. 1. Sereno MI
    2. Dale AM
    3. Reppas JB
    4. Kwong KK
    5. Belliveau JW
    6. Brady TJ
    7. Rosen BR
    8. Tootell RBH

    (1995) Borders of Multiple Visual Areas in Humans Revealed by Functional Magnetic Resonance Imaging

    Science 268:889–893.

    https://doi.org/10.1126/science.7754376

    • PubMed
    • Google Scholar
79. 1. Smith SM

    (2002) Fast robust automated brain extraction

    Human Brain Mapping 17:143–155.

    https://doi.org/10.1002/hbm.10062

    • PubMed
    • Google Scholar
80. 1. Souza AS
    2. Oberauer K

    (2016) In search of the focus of attention in working memory: 13 years of the retro-cue effect

    Attention, Perception & Psychophysics 78:1839–1860.

    https://doi.org/10.3758/s13414-016-1108-5

    • PubMed
    • Google Scholar
81. 1. Spivey MJ
    2. Geng JJ

    (2001) Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects

    Psychological Research 65:235–241.

    https://doi.org/10.1007/s004260100059

    • PubMed
    • Google Scholar
82. 1. Sprague TC
    2. Ester EF
    3. Serences JT

    (2014) Reconstructions of information in visual spatial working memory degrade with memory load

    Current Biology 24:2174–2180.

    https://doi.org/10.1016/j.cub.2014.07.066

    • PubMed
    • Google Scholar
83. 1. Sprague TC
    2. Ester EF
    3. Serences JT

    (2016) Restoring Latent Visual Working Memory Representations in Human Cortex

    Neuron 91:694–707.

    https://doi.org/10.1016/j.neuron.2016.07.006

    • PubMed
    • Google Scholar
84. 1. Sprague TC
    2. Boynton GM
    3. Serences JT

    (2019) The Importance of Considering Model Choices When Interpreting Results in Computational Neuroimaging

    ENeuro 6:ENEURO.0196-19.2019.

    https://doi.org/10.1523/ENEURO.0196-19.2019

    • PubMed
    • Google Scholar
85. 1. Sreenivasan KK
    2. Curtis CE
    3. D’Esposito M

    (2014) Revisiting the role of persistent neural activity during working memory

    Trends in Cognitive Sciences 18:82–89.

    https://doi.org/10.1016/j.tics.2013.12.001

    • PubMed
    • Google Scholar
86. 1. Supèr H
    2. Spekreijse H
    3. Lamme VAF

    (2001) A neural correlate of working memory in the monkey primary visual cortex

    Science (New York, N.Y.) 293:120–124.

    https://doi.org/10.1126/science.1060496

    • PubMed
    • Google Scholar
87. 1. Swisher JD
    2. Halko MA
    3. Merabet LB
    4. McMains SA
    5. Somers DC

    (2007) Visual topography of human intraparietal sulcus

    The Journal of Neuroscience 27:5326–5337.

    https://doi.org/10.1523/JNEUROSCI.0991-07.2007

    • PubMed
    • Google Scholar
88. 1. van Ede F
    2. Chekroud SR
    3. Nobre AC

    (2019a) Human gaze tracks attentional focusing in memorized visual space

    Nature Human Behaviour 3:462–470.

    https://doi.org/10.1038/s41562-019-0549-y

    • PubMed
    • Google Scholar
89. 1. van Ede F
    2. Chekroud SR
    3. Stokes MG
    4. Nobre AC

    (2019b) Concurrent visual and motor selection during visual working memory guided action

    Nature Neuroscience 22:477–483.

    https://doi.org/10.1038/s41593-018-0335-6

    • PubMed
    • Google Scholar
90. 1. van Ede F
    2. Board AG
    3. Nobre AC

    (2020) Goal-directed and stimulus-driven selection of internal representations

    PNAS 117:24590–24598.

    https://doi.org/10.1073/pnas.2013432117

    • PubMed
    • Google Scholar
91. 1. van Kerkoerle T
    2. Self MW
    3. Roelfsema PR

    (2017) Layer-specificity in the effects of attention and working memory on activity in primary visual cortex

    Nature Communications 8:1–14.

    https://doi.org/10.1038/ncomms13804

    • PubMed
    • Google Scholar
92. 1. Vaziri-Pashkam M
    2. Xu Y

    (2017) Goal-Directed Visual Processing Differentially Impacts Human Ventral and Dorsal Visual Representations

    The Journal of Neuroscience 37:8767–8782.

    https://doi.org/10.1523/JNEUROSCI.3392-16.2017

    • PubMed
    • Google Scholar
93. 1. Wandell BA
    2. Dumoulin SO
    3. Brewer AA

    (2007) Visual field maps in human cortex

    Neuron 56:366–383.

    https://doi.org/10.1016/j.neuron.2007.10.012

    • PubMed
    • Google Scholar
94. 1. Winawer J
    2. Witthoft N

    (2015) Human V4 and ventral occipital retinotopic maps

    Visual Neuroscience 32:E020.

    https://doi.org/10.1017/S0952523815000176

    • PubMed
    • Google Scholar
95. 1. Woolrich MW
    2. Ripley BD
    3. Brady M
    4. Smith SM

    (2001) Temporal autocorrelation in univariate linear modeling of FMRI data

    NeuroImage 14:1370–1386.

    https://doi.org/10.1006/nimg.2001.0931

    • PubMed
    • Google Scholar
96. 1. Xing Y
    2. Ledgeway T
    3. McGraw PV
    4. Schluppeck D

    (2013) Decoding working memory of stimulus contrast in early visual cortex

    The Journal of Neuroscience 33:10301–10311.

    https://doi.org/10.1523/JNEUROSCI.3754-12.2013

    • PubMed
    • Google Scholar
97. 1. Xu Y

    (2018) Sensory Cortex Is Nonessential in Working Memory Storage

    Trends in Cognitive Sciences 22:192–193.

    https://doi.org/10.1016/j.tics.2017.12.008

    • PubMed
    • Google Scholar
98. 1. Xu Y

    (2020) Revisit once more the sensory storage account of visual working memory

    Visual Cognition 28:433–446.

    https://doi.org/10.1080/13506285.2020.1818659

    • PubMed
    • Google Scholar

Author details

1. Margaret M Henderson

   1. Neurosciences Graduate Program, University of California, San Diego, San Diego, United States
   2. Department of Machine Learning, Carnegie Mellon University, Pittsburgh, United States
   3. Neuroscience Institute, Carnegie Mellon University, Pittsburgh, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft, Writing – review and editing

   For correspondence

   mmhender@cmu.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-9375-6680

2. Rosanne L Rademaker

   1. Department of Psychology, University of California, San Diego, San Diego, United States
   2. Ernst Strüngmann Institute (ESI) for Neuroscience in Cooperation with Max Planck Society, Frankfurt, Germany

   Contribution

   Conceptualization, Investigation, Methodology, Software, Writing – review and editing

   Competing interests

   No competing interests declared

3. John T Serences

   1. Neurosciences Graduate Program, University of California, San Diego, San Diego, United States
   2. Department of Psychology, University of California, San Diego, San Diego, United States
   3. Kavli Foundation for the Brain and Mind, University of California, San Diego, San Diego, United States

   Contribution

   Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Software, Supervision, Writing – review and editing

   Competing interests

   Reviewing editor, eLife

• 1,169

  views

• 189

  downloads

• 36

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.