In the fMRI session of Experiment 1, human participants (n=24) completed two tasks, maintenance and categorization, inside an MRI scanner. The maintenance task was a delayed match-to-sample WM task of orientations. Participants only needed to maintain the cued orientation throughout a memory delay. In the categorization task, participants also started with maintaining an orientation. After the task cue, they needed to categorize the remembered orientation into one of two categories using the cued categorization rule. Within an experimental block of nine trials, participants randomly switched between the two tasks. Across blocks, participants randomly switched between two categorization rules acquired during a preceding learning session. We randomized response mapping across trials to avoid potential influence by motor-planning signals (see Figure 1A for probe design). Prior to the main session, participants first completed a behavioral learning session to learn two categorization rules (Rule A and Rule B, see Figure 1B) that were orthogonal to each other. Participants acquired the two rules with equal familiarity (t(23) = 0.24, p=0.813; for Rule A, M=0.85, SD = 0.050; for Rule B, M=0.85, SD = 0.05; Figure 1C) and comparable precision (averaged error in reported boundaries for Rule A were 8.80°±6.29° and that for Rule B was 9.02°±5.69°; Figure 1D).

Figure 1 with 1 supplement see all

Download asset Open asset

Overall, participants performed equally well on both tasks in the fMRI session. Accuracy for the maintenance task (M ± SD: 0.81±0.07) and that for the categorization task (0.82±0.05) did not significantly differ (t(23) = 1.51, p=0.144; Figure 1E), suggesting that the two tasks were matched in terms of task difficulty. In line with previous categorization studies demonstrating a boundary effect (Ester et al., 2020; Freedman and Assad, 2006), only in the categorization task but not in the maintenance task did participants perform better for trials distant from category boundaries in terms of both accuracy and reaction time (see Figure 1G). These results demonstrated the effect of categorization and confirmed that participants faithfully followed task instructions.

The primary goal of this study was to determine the role of stimulus representation in various cortices in WM. Using conventional multivariate encoding and decoding methods, we tracked stimulus (i.e. orientation) representation in three brain regions of interest (ROIs) that have been implicated in representing WM information, including EVC, intraparietal sulcus (IPS), and superior precentral sulcus (sPCS) (Christophel et al., 2018; Ester et al., 2015; Hallenbeck et al., 2021; Yu and Shim, 2017), see Figure 2—figure supplement 1 for a visualization of the anatomical locations of the ROIs.

First, we used multivariate inverted encoding models (IEMs) (Brouwer and Heeger, 2009; Brouwer and Heeger, 2011; Ester et al., 2015; Rademaker et al., 2019; Yu and Shim, 2017) to reconstruct orientation representation at the population level in each ROI. Figure 2A shows example orientation reconstructions from representative time points. To quantify the strength of orientation reconstructions, we calculated the reconstruction fidelity by first projecting the channel response at each orientation onto a vector at the cued orientation and then averaging the projected vectors to obtain the representational fidelity (Rademaker et al., 2019). A larger fidelity value indicates a stronger positive representation of orientation (See Figure 2B for illustration). Figure 2C demonstrates the time course of orientation reconstruction as quantified by representational fidelity. Critically, by comparing the representational fidelity values within the same ROI across conditions (maintenance vs. categorization), we minimized the impact of effect size on our comparisons between ROIs, as all the comparisons were primarily performed within the same ROI with a comparable effect size. In EVC, we found significant orientation representation in both maintenance and categorization tasks starting from the sample period (Figure 2C left panel; see Figure 2—source data 1 for full statistics), even when the categorization task did not require explicit memory of stimulus information. Additionally, the strength of orientation representation in the maintenance task became significantly higher than that in the categorization task after the task cue during the delay, suggesting the strength of orientation representations in EVC reflected the degrees of task demands for maintaining visual details. In IPS, orientation representation was significant in both tasks, but did not differ from each other at most time points (Figure 2C middle panel). In sPCS, a reversed pattern was observed. In the maintenance task, orientation information was maintained during early delay period and then dropped to baseline level during late delay period. By contrast, in the categorization task, orientation representation was persistent throughout the delay and response periods. The strength of orientation representation in the categorization task became statistically higher than that in the maintenance task in late delay period (Figure 2C right panel), suggesting that this differential representations of visual stimulus in the frontal cortex reflected the demand for active control of memory contents. To facilitate comparison of the differential stimulus representation across ROIs, we averaged the difference in representational strength across a late task epoch (11–16 s), and the difference in stimulus representation between ROIs remained (Figure 2D).

Figure 2 with 3 supplements see all

Download asset Open asset

Figure 2—source data 1

p-values for the time course of inverted encoding model (IEM) results in Figure 2.

Underline denotes significant results (p<0.05).

https://cdn.elifesciences.org/articles/100287/elife-100287-fig2-data1-v1.xlsx

Download elife-100287-fig2-data1-v1.xlsx

We validated the difference in stimulus representations through a series of control analyses. First, we demonstrated that these results cannot be explained by the specific model used to train the data (Liu et al., 2018; Sprague et al., 2018) nor the specific analytical approach used, because similar patterns were observed when we trained the IEM separately for each condition (Figure 2—figure supplement 2A) or adopted a Support Vector Machine (SVM) decoding approach (Figure 2—figure supplement 2B; Henderson et al., 2022; Rademaker et al., 2019). Mean activation differences between tasks cannot account for the results either, because when we removed the mean differences in BOLD activity between tasks, the difference in representational strength remained (Figure 2—figure supplement 2C). Furthermore, to remove the potential impact of voxel number on IEM, we selected the top 500 of most sample- or delay-selective voxels from each ROI and trained IEM using the selected voxels. Again, this analysis yielded similar findings (Figure 2—figure supplement 2D). Lastly, when repeating the IEM analyses in primary motor cortex (M1), no similar patterns were observed, suggesting that our results cannot be explained by motor- or response-related activity (Figure 2—figure supplement 3).

Together, these results demonstrated enhanced stimulus representation in the frontal cortex with increased demand for active control, as well as those in the visual cortex with increased demand for precise WM maintenance.

Previous WM studies have shown that the strength of stimulus representation in EVC positively correlated with memory performance (Emrich et al., 2013; Ester et al., 2013; Gosseries et al., 2018), suggesting that EVC plays an important role in precise WM maintenance. However, stimulus representation in the frontal cortex rarely predicted behavioral performance in maintenance task (Hallenbeck et al., 2021), although univariate activation in the same brain region can predict memory-guided saccade performance (Curtis et al., 2004). Nevertheless, if frontal stimulus representation is involved in WM control, its behavioral relevance should be subject to observation with increased control demands. Therefore, we assessed the behavioral predictability of stimulus representation during the delay period in EVC, IPS, and sPCS. Figure 3A, B and C illustrate the time course of correlation results, results from representative time points, and results collapsed across the late epoch, respectively. Consistent with previous findings, we found the strength of stimulus representation in EVC during the early delay period predicted behavioral accuracies in both maintenance and categorization tasks (see Figure 3—source data 1 for full statistics). Similar predictability was found in IPS, with stimulus representation predicted behavior in the maintenance task during early delay and in the categorization task throughout the entire memory delay. Interestingly, we found that, throughout the entire memory delay, the strength of stimulus representation in sPCS predicted behavioral accuracies only in the categorization task but not in the maintenance task. These results highlighted the functional significance of stimulus representation in sPCS exclusively for the categorization task.

Figure 3 with 1 supplement see all

Download asset Open asset

Figure 3—source data 1

p-values for the time course of correlation results in Figure 3.

Underline denotes significant results (p<0.05).

https://cdn.elifesciences.org/articles/100287/elife-100287-fig3-data1-v1.xlsx

Download elife-100287-fig3-data1-v1.xlsx

In Experiment 1, participants flexibly switched between two categorization rules to prompt the manipulation of WM content on a trial-by-trial basis. The rule switching increased control demand but also complicated the interpretation of our results. To exclude potential impact of rule switching, we conducted Experiment 2 (n = 24), in which participants performed the maintenance and categorization tasks with only one fixed rule. Behavioral results of Experiment 2 again demonstrated a classic boundary effect and were comparable to Experiment 1, with no significant difference between experiments in terms of either accuracy or reaction time (Fs <1.28, ps >0.26; Figure 1—figure supplement 1). When using IEMs to reconstruct stimulus representation, we found EVC and IPS both showed patterns similar to those in Experiment 1 (Figure 4A left and middle panels), with stimulus representation decreased in EVC in the categorization task and remained at the same level in IPS between the two tasks (see Figure 4—source data 1 for full statistics). The frontal region, sPCS, also showed a differential enhancement of stimulus representation in the categorization task as in Experiment 1, but in an earlier period (Figure 4A right panel). To validate such a temporal difference, we defined an additional early task epoch (5–10 s), and confirmed a significant difference in stimulus representation in sPCS during the early (p=0.015; Figure 4B) but not during the late epoch (p=0.372; Figure 4C). In addition, we performed a mixed ANOVA on experiments (Experiment 1 vs 2) and epochs (early vs. late epoch) and observed a significant interaction effect between the two, F(1, 46)=7.43, p=0.009, suggesting that the two experiments differed in terms of the temporal emergence of the differential stimulus representation in the frontal cortex. Taken together, these results are consistent with our expectation that, with reduced control demand, the differential enhancement of stimulus representation in frontal cortex was still present but decreased during late memory delay. Nevertheless, stimulus representation in Experiment 2 still predicted behavioral performance as in Experiment 1, although the difference between tasks was reduced (Figure 3—figure supplement 1).

Figure 4

Download asset Open asset

Figure 4—source data 1

p-values for the time course of inverted encoding model (IEM) results in Figure 4.

Underline denotes significant results (p<0.05).

https://cdn.elifesciences.org/articles/100287/elife-100287-fig4-data1-v1.xlsx

Download elife-100287-fig4-data1-v1.xlsx

Having observed a differential representation of stimulus in the frontal cortex, we next asked how newly generated information in WM during the categorization task emerged and sustained in the distributed WM network and how representations of the original stimulus and the new information interacted. The categorization task could demand additional generation of category information in WM. We, therefore, trained SVMs to decode category information during the categorization task. For each rule, the SVM decoder was trained to discriminate between the two categories. In both experiments, we found that during the late epoch, category information could be well decoded across ROIs (ps <0.044, Figure 5A; also see Figure 5—figure supplement 1 for full decoding time course), with a marginal difference between experiments in sPCS (p=0.055).

Figure 5 with 3 supplements see all

Download asset Open asset

One might argue that the category decoding results could at least be partially attributed to stimulus similarity. To minimize the impact of stimulus similarity on category decoding, we additionally trained another decoder using the opposite rule (i.e. using category labels from the orthogonal rule; see Figure 5B for results). We then calculated an abstract category index by subtracting decoding accuracy under the opposite rule from that under the true rule (Mok and Love, 2020; Figure 5C). The rationale was that the amount of stimulus similarity would be comparable for the opposite rule, but additional category information, if existed, should result in higher decoding accuracy for the true rule. In other words, positive abstract category index indicates evidence for stimulus-independent category representation. After removing stimulus-related signals, average decoding performance of abstract category was only evident in Experiment 2 (ps <0.017) but not in Experiment 1 (ps >0.14) for all ROIs. Moreover, decoding performance of abstract category was significantly higher in Experiment 2 than Experiment 1 in sPCS (P=0.034; Figure 5D). These results together suggest a potential tradeoff between stimulus difference and category representation in the frontal cortex. This tradeoff was further depicted using linear mixed-effects modeling (LMEM) on representational dissimilarity matrices (RDMs) of neural activities to decouple the contributions of stimulus and category to neural representation (Figure 5—figure supplement 2A). Overall, the LMEM results (Figure 5—figure supplement 2B–D) replicated the above findings, with significant stimulus but not category representation in sPCS in Experiment 1, and a decreased contribution of stimulus but emerging category representation in the same brain region in Experiment 2.

Lastly, for completeness, we repeated the IEM and category decoding analyses on additional ROIs in the frontal cortex, to investigate whether the observed results were specific to sPCS. Specifically, we defined three additional ROIs based on the HCP atlas (Glasser et al., 2016), including the inferior precentral sulcus (iPCS), inferior frontal sulcus (IFS), and middle frontal gyrus (MFG) (Ester et al., 2015; Mackey et al., 2017; Yu and Shim, 2017). Overall, results in the three ROIs were comparably weaker than sPCS (Figure 5—figure supplement 3). There was some indication that the MFG might share some results for orientation representation and category decoding, although this pattern was weaker and was only observed in some analyses in Experiment 2.

Finally, we tested how stimulus representation could emerge in frontal cortex at the mechanistic level using RNN models. Our hypothesis is that precise stimulus representation during WM might emerge in frontal cortex in response to complex task demands such as rule-based categorization. In other words, instead of relying (solely) on category representations, the cortical network might have adopted a different strategy to accommodate flexible task requirements in the current study, for instance, by preserving stimulus information until a later stage of information processing. This different strategy can be implemented by altering the RNN’s output structure. Therefore, the logic of this modeling analysis was to examine whether explicitly placing a demand for the model to preserve stimulus representation would recapitulate our fMRI findings in frontal cortex, in comparison to a model that did not specify such a demand.

Two types of modular RNNs were trained on the maintenance and categorization tasks simultaneously (Masse et al., 2019; Zhou et al., 2021). The networks shared common input and hidden layer structures (i.e. orientation-tuned and retro/task cue-related units as the input layer, recurrent units with short-term synaptic plasticity in the hidden layer [80% excitatory +20% inhibitory units, equally distributed in three separate modules]). The only difference was in the structure of the output layer. The first type of RNN (RNN1; n=20) had only two units in the output layer to indicate networks’ choice (Figure 6A), whereas the second type of RNN (RNN2; n=20) had additional units in the output layer corresponding to the original stimulus information. In other words, the second RNN was designed to maintain stimulus information throughout the network modules. For the common hidden layer, we included three hierarchically organized (posterior, middle, and anterior) modules of recurrent units generated according to neurobiological principles of neuronal connections (e.g. denser connectivity within than between modules) to simulate the interconnected brain areas in our ROI-based fMRI analyses above: the posterior module (Module 1, simulating EVC) was directly connected with the input layer, the middle module (Module 2, simulating IPS) received projections from the posterior module and relaying information to the anterior module, and the anterior module (Module 3, simulating sPCS) projected to the output layer. Task events were simulated as numerical inputs to the model, matching the procedures of Experiment 1 (see Methods for details).

Figure 6 with 1 supplement see all

Download asset Open asset

After successful training, defined as reaching at least 90% accuracy in all tasks in the same training batch, we applied an SVM decoding approach to investigate population-level stimulus representations in neuronal spiking activities of the RNNs. We found that in RNN1, both the middle and anterior modules showed stronger stimulus representation in the maintenance task than the categorization task during the delay period (p_posterior = 0.09,p_middle = 0.011, p_anterior = 0.007; Figure 6B upper panel), opposite to our fMRI observation in IPS and sPCS (Figure 6B inset). In comparison, decoding performance in RNN2, which was explicitly required to maintain stimulus information for the output, yielded results consistent with our human findings, with increased stimulus decoding performance during categorization only in the anterior module (p_posterior = 0.436, p_middle = 0.212, p_anterior = 0.026; Figure 6B lower panel).

Besides the difference in stimulus representation, we further tested whether RNN2 could also replicate the human results on category representation. For this analysis, we focused on abstract category representation to fully remove the impact of stimulus on category decoding. To examine the influence of control demand on category decoding, following our fMRI experiment, we trained 20 additional RNNs with the same output structure as RNN2 (preserving stimulus information) to perform the tasks with a fixed categorization rule, mimicking the task structure of Experiment 2. Consistent with our human findings, we observed increased abstract category decoding performance in the fixed-rule models compared to the flexible-rule RNNs, throughout the modules (p_posterior = 0.045, p_middle = 0.003, p_anterior <0.001; Figure 6C lower panel). Similar differences between fixed-rule and flexible-rule models were also observed in RNN1 (all ps <0.001; Figure 6C upper panel).

To quantify the similarity between human and RNN in terms of stimulus and category representations, we performed representational similarity analysis (RSA) comparing human and RNN data across tasks. Specifically, we aligned the RDMs by either stimulus (Figure 7A) or category (Figure 7C) to prioritize the comparison of stimulus or category representations, respectively. Overall, across experiments, RNN2 consistently demonstrated higher similarity to human data than RNN1, particularly in Module 3 that corresponds to the frontal cortex (Figure 7B and D). In terms of the RDM patterns, consistent with previous results, RNN2 exhibited stronger stimulus representation than RNN1. Moreover, fixed-rule RNNs were found to be more categorical than flexible-rule RNNs regardless of the output structure. Altogether, these findings demonstrated that our fMRI results could be simulated by RNN models when stimulus information for readout was preserved, suggesting that the requirement for flexible control of WM content could demand high-fidelity stimulus representation at the output stage of the model. Notably, we found that RNN2 generally took fewer iterations for training and had fewer failures in learning the task (with a defined maximal number of iterations).

Figure 7

Download asset Open asset

The visual cortex has been considered a critical site for maintaining visual WM in the context of sensorimotor recruitment hypothesis (D’Esposito and Postle, 2015; Harrison and Tong, 2009). This idea, however, has been challenged in recent years due to some seemingly contradictory findings from the human neuroimaging studies. For example, compared to the frontoparietal cortex, mnemonic representations in EVC were found to be more vulnerable to distractors (Bettencourt and Xu, 2016; Hallenbeck et al., 2021; Lorenc et al., 2018). The decodability of memory contents in visual cortex also depends on the specific task type. A previous study showed that in nonvisual tasks that required judgments on object category instead of visual details, memory contents were no longer decodable in the visual cortex (Lee et al., 2013). In this study, we found that, although the strength of stimulus representation in EVC differed between WM maintenance and categorization tasks, a copy of stimulus representation remained in EVC during the categorization task. Moreover, stimulus representations in both tasks were equally predictive of subsequent memory performance, suggesting the functional significance of EVC representations in WM.

The discrepancy between our results and that of the previous work (Lee et al., 2013) could be attributed to the fact that our categorization task required participants to manipulate remembered information according to arbitrary yet flexible categorization rules, rather than simply paying selective attention to different aspects (visual details vs. category membership) of everyday objects. In our case, maintaining visual details of the memoranda was critical for accurate behavioral responses. Our finding is consistent with the prediction of sensorimotor recruitment hypothesis that representation of memory contents in the visual cortex is necessary for the precise maintenance of visual information. The observation of robust category representation in EVC during the response period further indicated the recruitment of EVC in categorization, possibly for boundary comparison and rule implementation. In fact, our results are consistent with a recent study demonstrating significant stimulus representation in EVC even when memoranda had been transformed into a motor format (Henderson et al., 2022). In addition, electrophysiological research in non-human primates has also shown robust feature selectivity in the visual cortex during a categorization task (Brincat et al., 2018), and recent computational modeling work has suggested intact maintenance of sensory information during categorical judgments (Luu and Stocker, 2021).

Compared to the prominent role of EVC in memory maintenance, sPCS in the frontal cortex played a dominant role in WM tasks that require active control of memory contents such as categorization. Although stimulus representations in sPCS have been observed during WM in previous studies, the nature of these representations remained debatable. In WM tasks that required mere maintenance of memoranda, stimulus was not always decodable in the frontal cortex (Emrich et al., 2013; Gosseries et al., 2018; Riggall and Postle, 2012), raising the issue of functional significance of stimulus representation in the frontal cortex. On the other hand, stimulus representation in the frontal cortex could become robust in the face of tasks that require attentional prioritization and extensive training (Bettencourt and Xu, 2016; Christophel et al., 2018; Hallenbeck et al., 2021; Lorenc et al., 2018; Miller et al., 2022). Our current study contributes to the resolution of this issue by demonstrating that stimulus representation in sPCS increased with increasing demands for WM control. This finding is in line with recent computational studies proposing that active WM functions may involve neuronal mechanisms different from that for passive maintenance. For example, passive maintenance could rely mainly on synaptic plasticity mechanisms, whereas active control functions such as distractor resistance and information manipulation involve more neuronal spiking activity (Masse et al., 2019; Wang, 2021). In this study, we provided the first empirical evidence that the frontal cortex exhibits enhanced stimulus representation in categorization task requiring active WM control and this representation is predictive of WM performance. In contrast, stimulus representation of WM maintenance failed to predict WM performance at high control demand. Moreover, by examining additional ROIs in the frontal cortex (i.e. iPCS, IFS, and MFG), we found that results in these brain regions were overall weaker, with the MFG demonstrating the most comparable, albeit much weaker, results to sPCS. The specific involvement of sPCS in our experiment could be due to the type of stimulus (i.e. orientation) we used, as previous work has highlighted a prominent role of sPCS in encoding stimulus-specific information in spatial (Hallenbeck et al., 2021; Sprague and Serences, 2013) and orientation WM (Ester et al., 2015; Yu and Shim, 2017; Yu and Shim, 2019), but to a lesser extent in other WM tasks such as color (Yu and Shim, 2017; Yu and Shim, 2019). It would be of interest to further investigate whether this active control in the sPCS could be generalized to other frontal regions and tasks that require other types of WM control such as mental rotation (Shi and Yu, 2024), and how other types of control task may adapt to changing flexible control demands.

Our results in the frontal cortex are also in line with recent theoretical proposals in the field of cognitive flexibility. To behave flexibly in complex environments with limited cognitive resources, two mechanisms have been proposed: low-dimensional abstraction of stimulus representation for generalization and efficient learning, and high-dimensional stimulus representation for separability and flexible readout (Badre et al., 2021; Flesch et al., 2022; Fusi et al., 2016). Within this framework, high-dimensional stimulus representation during WM might emerge in the frontal cortex in response to complex control demands such as rule-based categorization. The results of the two fMRI experiments in the current study jointly demonstrate a dynamic tradeoff between high-dimensional stimulus and low-dimensional category representations depending on the control demand. Specifically, when control demand was reduced with a single categorization rule in Experiment 2 compared to Experiment 1, the differential stimulus representation in the frontal cortex was also reduced during the late delay period, accompanied by an increase in category decoding performance especially in the frontal cortex. This result is consistent with neurophysiological findings in non-human primates: while robust category selectivity was observed in frontoparietal cortex during the delay period of categorization tasks when the animal was trained on the categorization task only (Brincat et al., 2018; Freedman and Assad, 2006; Freedman et al., 2001; McKee et al., 2014), category selectivity in the parietal cortex was significantly reduced when the animal had been exposed to a maintenance task prior to categorization training (Latimer and Freedman, 2023). Our RNN simulation further confirmed that this dynamic reconfiguration in information coding at the network level can be well explained by a change in the coding strategy for the network readout. In other words, in flexible environments, and with rich prior experience, the brain might adopt an entirely different strategy for processing information in WM. High-dimensional stimulus information might be preserved in its original identity in the higher-order cortex, potentially reducing processing demands in dealing with each task and thereby facilitating efficiency and flexibility (Badre et al., 2021; Flesch et al., 2022; Fusi et al., 2016). One important future direction would be to further address the meta-control mechanisms that determine the flexible selection of coding strategies for WM (Eppinger et al., 2021).

While many previous WM studies have focused on the functional distinction between sensory and frontoparietal cortex, it has remained less clear how frontal and parietal cortices might differ in terms of WM functions. Some studies have reported stimulus representations with similar functionality in frontal and parietal cortex (Christophel et al., 2018; Yu and Shim, 2019), while others have observed differential patterns (Hu and Yu, 2023; Lee et al., 2013; Li et al., 2023). We interpret the differential patterns as reflecting a difference in the potential origin of the corresponding cognitive functions. For example, in our study, sPCS demonstrated the most prominent effect for enhanced stimulus representation during categorization as well as the tradeoff between stimulus difference and category representation, suggesting that sPCS might serve as the source region for such effects. On the other hand, IPS did show visually similar patterns to sPCS in some analyses. For instance, stimulus representation in IPS was visually but not statistically higher in the categorization task. Additionally, stimulus representation in IPS also predicted behavioral performance in the categorization task. These results together support the view that our findings in sPCS do not occur in isolation, but rather reflect a dynamic reconfiguration of functional gradients along the cortical hierarchy from early visual to parietal and then to frontal cortex.

Although the current RNNs effectively captured our key fMRI findings, including increased stimulus representation in the frontal cortex as well as the tradeoff in category representation with varying control demands, we acknowledge that differences remain between the two modalities. For instance, all three RNN modules demonstrated significant abstract category decoding as well as differences in category decoding between experiments, which differed from our fMRI results. This discrepancy could partially be due to the higher signal sensitivity in RNNs. In addition, RNN2 did not show decreased stimulus decoding for categorization in the EVC module. However, we found that applying IEMs to the RNN data revealed a similar negative trend in the EVC module (Figure 6—figure supplement 1), though it was not statistically significant. This result suggests that any negative difference between categorization and maintenance in EVC module was weaker compared to fMRI, if existed. We speculate that enhancing the negative difference in the EVC module might require additional modules or inputs to strengthen fine-grained stimulus representation in EVC, a mechanism that could be of interest for future research.

In conclusion, we observed a distributed cortical network, including early visual, parietal, and frontal cortex, in representing stimulus-specific information in WM. These stimulus representations in visual and frontal cortex played distinct functional roles, with those in EVC contributing primarily to precise maintenance and those in frontal cortex contributing primarily to active control in WM. RNN simulations indicated that the stimulus representation in the frontal cortex might have emerged as a result of output selection to facilitate cognitive flexibility. Collectively, these results help to reconcile current debates on the functional roles of different cortical regions in WM, and provide new insights into how a unified WM framework could support varied control demands.

A total of 54 participants were recruited at the Chinese Academy of Sciences, Shanghai Branch. Twenty-six healthy participants (21 female, all right-handed, mean age = 24.0 ± 1.4 y) were recruited for Experiment 1. Two were excluded due to failure in completing the experiment or low conformity to task instructions, remaining 24 participants who completed the main experiment (19 female, mean age = 23.92 ± 1.41 y). Twenty-eight (22 female, mean age = 24.14 ± 1.51 y) participants were recruited for Experiment 2. Two quitted after behavioral training and two did not finish scanning due to technical problems with the scanner, resulting in 24 participants (20 female, mean age = 24.13 ± 1.60 y) in the final analyses. All participants were neurologically healthy and eligible for MRI, had normal or corrected-to-normal vision, provided written informed consent approved by the Ethics Committee of the Institute of Neuroscience, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences (CEBSIT-2020028), and were monetarily compensated for their participation. Sample sizes were not estimated a priori but were comparable and even superior to those in previous studies.

All stimuli were generated and presented using MATLAB (The MathWorks) and Psychtoolbox 3 extensions (Brainard, 1997; Pelli, 1997). During behavioral training, stimuli were presented on a ThinkVision monitor at a viewing distance of 45 cm. Behavioral responses were acquired with a keyboard. During scanning, stimuli were projected onto a SinoRad monitor (1280×1024 pixels, refreshing at 60 Hz) viewed through a coil-mounted mirror in the scanner at a viewing distance of 90.5 cm. Participants’ behavioral responses were acquired with a Sinorad MRI-compatible button box.

MRI data of Experiment 1 were collected using a 3 Tesla Siemens MRI scanner (Tim Trio; Siemens Healthineers) with a 32-channel head coil at the Functional Brain Imaging Platform at the Institute of Neuroscience, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences. Functional scanning was performed using a gradient-echo echo-planar sequence with the following parameters: repetition time (TR)=1000 ms; echo time (TE)=30 ms; flip angle (FA)=40°; voxel size = 3 × 3 × 3 mm; multi-band accelerate factor = 4; matrix size = 74 × 74; slice number = 60. A high-resolution anatomical T1 image was collected before functional scanning (TR = 2300 ms; TE = 2.98 ms; FOV = 256 × 240 × 192 mm; voxel size = 1 × 1 × 1 mm). During scanning, participants’ head positions were restricted with surrounding paddings to prevent head movements. MRI data of Experiment 2 were collected using identical procedures and settings except that the last eleven participants were scanned using a newly installed 3 Tesla Siemens MRI scanner (Prisma; Siemens Healthineers) at the Functional Brain Imaging Platform.

All preprocessing of individual MRI data was performed using AFNI (https://afni.nimh.nih.gov/) (Cox, 1996; Cox and Hyde, 1997). Functional data of all runs were registered to the last volume of the final run with the first eight volumes of each run removed. Then, individuals’ aligned functional data were registered to their corresponding T1 volume. Alignment of registration was manually checked for each subject to ensure quality. The registered data were further motion corrected and detrended.

Our primary ROI-based analyses focused on three most commonly-studied, WM-related brain areas: early visual cortex (EVC), intraparietal sulcus (IPS) in parietal cortex, and superior precentral sulcus (sPCS) in frontal cortex (Ester et al., 2015; Hallenbeck et al., 2021; Yu and Shim, 2017). We created anatomical ROI masks based on the probabilistic atlas by Wang and colleagues (Wang et al., 2015). EVC (merging bilateral V1, V2, and V3), IPS (merging bilateral IPS0-5), and sPCS (merging bilateral FEF) masks were generated by warping masks from the probabilistic atlas to individuals’ anatomical image in their native space. In order to generate functional ROI masks, we then performed general linear models (GLMs) to quantify task-related univariate activity changes in each voxel. Task events were modeled using boxcar functions convolved with a canonical hemodynamic response function (durations of event epochs for sample, post retro-cue delay, memory delay, and response were 2.5 s, 2 s, 8.5 s, and 2 s, respectively). Six nuisance regressors were also included to account for head motion artifacts in the six dimensions of rigid body motion. Functional EVC mask was defined by the 500 most active voxels during sample display. Functional IPS and sPCS masks were defined by the 500 most active voxels during memory delay. Additional ROIs in the frontal cortex were defined using the HCP atlas (Glasser et al., 2016), including the inferior precentral sulcus (iPCS, generated by merging 6v, 6r, and PEF), inferior frontal sulcus (IFS, generated by merging IFJp, IFJa, IFSp, IFSa, and p47r), middle frontal gyrus (MFG, generated by merging 9-46d, 46, a9-46v, and p9-46v), and primary motor cortex (M1).

Participants’ behavioral performance for the main task was assessed using accuracy and reaction time. Paired t-test was conducted for the two task types (maintenance & categorization) to evaluate differences between conditions.

Statistical significance was evaluated via a sign-flip permutation procedure for all other fMRI analyses. For example, to characterize the significance of IEM fidelity, we computed the p-value by comparing the true mean fidelity of our sample with a null distribution reflecting no IEM fidelity. The null distribution was created by randomly assigning 1 or –1 to fidelity scores of our sample and then averaging the sign-flipped samples for 10,000 times, resulting in a null distribution of 10,000 fidelity scores. To characterize the difference of IEM fidelity between tasks, we sign-flipped the fidelity sample for each condition and then averaged the difference for 10,000 times. The p-value was calculated by comparing the true mean difference with the generated null distribution of difference. p-values were corrected using FDR across ROIs, time , and tasks for all analyses unless specified. An early (5–10 s; 6^th-11^th TR) and late (11–16 s; 12^th-17^th TR) task epoch was also defined to facilitate comparisons between ROIs and experiments when needed.

For RNN decoding results, we pooled decoding accuracies across a critical task period (50–75 time points during delay) to produce summary statistics aligning with what was reported in the fMRI results. Average decoding results were corrected using the FDR method.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

We would like to thank Dr. Mu-ming Poo for valuable comments on an earlier version of the manuscript, and Dr. Tianming Yang for helpful discussions. This work was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences (Grant No. XDB1010202), the Ministry of Science and Technology of China (STI2030-Major Projects 2021ZD0203701, 2021ZD0204202), the National Natural Science Foundation of China (32271089), CAS Project for Young Scientists in Basic Research (YSBR-071), and Shanghai Pujiang Program (22PJ1414400) to QY.

All participants provided written informed consent approved by the Ethics Committee of Institute of Neuroscience, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences (CEBSIT-2020028).

1. Sent for peer review: June 6, 2024
2. Preprint posted: June 7, 2024
3. Reviewed Preprint version 1: September 3, 2024
4. Reviewed Preprint version 2: January 7, 2025
5. Reviewed Preprint version 3: March 14, 2025
6. Version of Record published: April 24, 2025

You can cite all versions using the DOI https://doi.org/10.7554/eLife.100287. This DOI represents all versions, and will always resolve to the latest one.

© 2024, Shao, Zhang et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.