Trial-by-trial inter-areal interactions in visual cortex in the presence or absence of visual stimulation

Reviewer #1 (Public review):

Summary:

In this study, the authors propose a "unifying method to evaluate inter-areal interactions in different types of neuronal recordings, timescales, and species". The method consists of computing the variance explained by a linear decoder that attempts to predict individual neural responses (firing rates) in one area based on neural responses in another area.

The authors apply the method to previously published calcium imaging data from layer 4 and layers 2/3 of 4 mice over 7 days, and simultaneously recorded Utah array spiking data from areas V1 and V4 of 1 monkey over 5 days of recording. They report distributions over "variance explained" numbers for several combinations: from mouse V1 L4 to mouse V1 L2/3, from L2/3 to L4, from monkey V1 to monkey V4, and from V4 to V1. For their monkey data, they also report the corresponding results for different temporal shifts. Overall, they find the expected results: responses in each of the two neural populations are predictive of responses in the other, more so when the stimulus is not controlled than when it is, and with sometimes different results for different stimulus classes (e.g., gratings vs. natural images).

Strengths:

(1) Use of existing data.

(2) Addresses an interesting question.

Weaknesses:

Unfortunately, the method falls short of the state of the art: both generalized linear models (GLMs), which have been used in similar contexts for at least 20 years (see the many papers, both theoretical and applied to neural population data, by e.g. Simoncelli, Paninsky, Pillow, Schwartz, and many colleagues dating back to 2004), and the extension of Granger causality to point processes (e.g. Kim et al. PLoS CB 2011). Both approaches are substantially superior to what is proposed in the manuscript, since they enforce non-negativity for spike rates (the importance of which can be seen in Figure 2AB), and do not require unnecessary coarse-graining of the data by binning spikes (the 200 ms time bins are very long compared to the time scale on which communication between closely connected neuronal populations within an area, or between related areas, takes place).

In terms of analysis results, the work in the manuscript presents some expected and some less expected results. However, because the monkey data are based on only one monkey (misleadingly, the manuscript consistently uses the plural "monkeys"), none of the results specific to that monkey, nor the comparison of that one monkey to mice, are supported by robust data. One of the main results for mice (bimodality of explained variance values, mentioned in the abstract) does not appear to be quantified or supported by a statistical test and is only present in two out of three mice. Moreover, the two data sets differ in too many aspects to allow for any conclusions about whether the comparisons reflect differences in species (mouse vs. monkey), anatomy (L2/3-L4 vs. V1-V4), or recording technique (calcium imaging vs. extracellular spiking).

Reviewer #2 (Public review):

Summary:

In this work, the authors investigated the extent of shared variability in cortical population activity in the visual cortex in mice and macaques under conditions of spontaneous activity and visual stimulation. They argue that by studying the average response to repeated presentations of sensory stimuli, investigators are discounting the contribution of variable population responses that can have a significant impact at the single trial level. They hypothesized that, because these fluctuations are to some degree shared across cortical populations depending on the sources of these fluctuations and the relative connectivity between cortical populations within a network, one should be able to predict the response in one cortical population given the response of another cortical population on a single trial, and the degree of predictability should vary with factors such as retinotopic overlap, visual stimulation, and the directionality of canonical cortical circuits.

To test this, the authors analyzed previously collected and publicly available datasets. These include calcium imaging of the primary visual cortex in mice and electrophysiology recordings in V1 and V4 of macaques under different conditions of visual stimulation. The strength of this data is that it includes simultaneous recordings of hundreds of neurons across cortical layers or areas. However, the weaknesses of calcium dynamics (which has lower temporal resolution and misses some non-linear dynamics in cortical activity) and multi-unit envelope activity (which reflects fluctuations in population activity rather than the variance in individual unit spike trains), underestimate the variability of individual neurons. The authors deploy a regression model that is appropriate for addressing their hypothesis, and their analytic approach appears rigorous and well-controlled.

From their analysis, they found that there was significant predictability of activity between layer II/III and layer IV responses in mice and V1 and V4 activity in macaques, although the specific degree of predictability varied somewhat with the condition of the comparison with some minor differences between the datasets. The authors deployed a variety of analytic controls and explored a variety of comparisons that are both appropriate and convincing that there is a significant degree of predictability in population responses at the single trial level consistent with their hypothesis. This demonstrates that a significant fraction of cortical responses to stimuli is not due solely to the feedforward response to sensory input, and if we are to understand the computations that take place in the cortex, we must also understand how sensory responses interact with other sources of activity in cortical networks. However, the source of these predictive signals and their impact on function is only explored in a limited fashion, largely due to limitations in the datasets. Overall, this work highlights that, beyond the traditionally studied average evoked responses considered in systems neuroscience, there is a significant contribution of shared variability in cortical populations that may contextualize sensory representations depending on a host of factors that may be independent of the sensory signals being studied.

Strengths:

This work considers a variety of conditions that may influence the relative predictability between cortical populations, including receptive field overlap, latency that may reflect feed-forward or feedback delays, and stimulus type and sensory condition. Their analytic approach is well-designed and statistically rigorous. They acknowledge the limitations of the data and do not over-interpret their findings.

Weaknesses:

The different recording modalities and comparisons (within vs. across cortical areas) limit the interpretability of the inter-species comparisons. The mechanistic contribution of known sources or correlates of shared variability (eye movements, pupil fluctuations, locomotion, whisking behaviors) were not considered, and these could be driving or a reflection of much of the predictability observed and explain differences in spontaneous and visual activity predictions. Previous work has explored correlations in activity between areas on various timescales, but this work only considered a narrow scope of timescales. The observation that there is some degree of predictability is not surprising, and it is unclear whether changes in observed predictability with analysis conditions are informative of a particular mechanism or just due to differences in the variance of activity under those conditions. Some of these issues could be addressed with further analysis, but some may be due to limitations in the experimental scope of the datasets and would require new experiments to resolve.

Reviewer #3 (Public review):

Neural activity in the visual cortex has primarily been studied in terms of responses to external visual stimuli. While the noisiness of inputs to a visual area is known to also influence visual responses, the contribution of this noisy component to overall visual responses has not been well characterized.

In this study, the authors reanalyze two previously published datasets - a Ca++ imaging study from mouse V1 and a large-scale electrophysiological study from monkey V1-V4. Using regression models, they examine how neural activity in one layer (in mice) or one cortical area (in monkeys) predicts activity in another layer or area. Their main finding is that significant predictions are possible even in the absence of visual input, highlighting the influence of non-stimulus-related downstream activity on neural responses. These findings can inform future modeling work of neural responses in the visual cortex to account for such non-visual influences.

A major weakness of the study is that the analysis includes data from only a single monkey. This makes it hard to interpret the data as the results could be due to experimental conditions specific to this monkey, such as the relative placement of electrode arrays in V1 and V4. The authors perform a thorough analysis comparing regression-based predictions for a wide variety of combinations of stimulus conditions and directions of influence. However, the comparison of stimulus types (Figure 4) raises a potential concern. It is not clear if the differences reported reflect an actual change in predictive influence across the two conditions or if they stem from fundamental differences in the responses of the predictor population, which could in turn affect the ability to measure predictive relationships. The authors do control for some potential confounds such as the number of neurons and self-consistency of the predictor population. However, the predictability seems to closely track the responsiveness of neurons to a particular stimulus. For instance, in the monkey data, the V1 neuronal population will likely be more responsive to checkerboards than to single bars. Moreover, neurons that don't have the bars in their RFs may remain largely silent. Could the difference in predictability be just due to this? Controlling for overall neuronal responsiveness across the two conditions would make this comparison more interpretable.

Author response:

Reviewer #1:

Summary:

In this study, the authors propose a "unifying method to evaluate inter-areal interactions in different types of neuronal recordings, timescales, and species". The method consists of computing the variance explained by a linear decoder that attempts to predict individual neural responses (firing rates) in one area based on neural responses in another area.

The authors apply the method to previously published calcium imaging data from layer 4 and layers 2/3 of 4 mice over 7 days, and simultaneously recorded Utah array spiking data from areas V1 and V4 of 1 monkey over 5 days of recording. They report distributions over "variance explained" numbers for several combinations: from mouse V1 L4 to mouse V1 L2/3, from L2/3 to L4, from monkey V1 to monkey V4, and from V4 to V1. For their monkey data, they also report the corresponding results for different temporal shifts. Overall, they find the expected results: responses in each of the two neural populations are predictive of responses in the other, more so when the stimulus is not controlled than when it is, and with sometimes different results for different stimulus classes (e.g., gratings vs. natural images).

Strengths:

(1) Use of existing data.

(2) Addresses an interesting question.

Unfortunately, the method falls short of the state of the art: both generalized linear models (GLMs), which have been used in similar contexts for at least 20 years (see the many papers, both theoretical and applied to neural population data, by e.g. Simoncelli, Paninsky, Pillow, Schwartz, and many colleagues dating back to 2004), and the extension of Granger causality to point processes (e.g. Kim et al. PLoS CB 2011). Both approaches are substantially superior to what is proposed in the manuscript, since they enforce non-negativity for spike rates (the importance of which can be seen in Figure 2AB), and do not require unnecessary coarse-graining of the data by binning spikes (the 200 ms time bins are very long compared to the time scale on which communication between closely connected neuronal populations within an area, or between related areas, takes place).

We thank the reviewer for this suggestion. Our goal was to use a simple and unified linear ridge regression framework that can be applied to both calcium imaging (mouse) and MUAe (monkey) data.

We will perform a GLM-based analysis enforcing non-negativity as suggested, including in the GLM any additional available variables that may contribute to the neuronal responses.

We also would like to note that:

● Macaque data: Our MUAe data are binned at 25 ms, not 200 ms. We used the envelope

of multi-unit activity as reported in the original study [1]. We did not perform spike sorting on these data and therefore, strictly speaking, this is not a point process and methods developed for point processes are not directly applicable.

● Mouse data: The Stringer et al. dataset [2,3] uses two-photon calcium imaging sampled at 2.5 or 3 Hz. Additionally, responses were computed by averaging two frames per stimulus (yielding an effective bin size of 666 ms or 800 ms), dictated by acquisition constraints. We will emphasize the low temporal resolution of these signals as a limitation in the discussion section, but we cannot improve the temporal resolution with our analyses. These signals are not point processes either (although there is a correlation between two-photon calcium signals and spike rates).

Regardless of these considerations, the reviewer’s points are well taken, and we will conduct additional analyses as described above.

In terms of analysis results, the work in the manuscript presents some expected and some less expected results. However, because the monkey data are based on only one monkey (misleadingly, the manuscript consistently uses the plural ‘monkeys’), none of the results specific to that monkey, nor the comparison of that one monkey to mice, are supported by robust data.

We will add data from at least two more monkeys, as suggested by the reviewer:

● First, we will include a second monkey from the same dataset [1]. The reason this monkey was not included in the original submission is that the dataset for this second monkey consisted of much less data than the original. For example, for the lights-off condition, the number of V4 channels with signal-to-noise ratio greater than 2 (recommended electrodes to use by dataset authors) is 9-12 in this second monkey, compared to 68-74 in the first monkey [1]. However, we will still add results for this second monkey.

● Additionally, we will include data from a new monkey by collaborating with the Ponce lab who will collect new data for this study.

One of the main results for mice (bimodality of explained variance values, mentioned in the abstract) does not appear to be quantified or supported by a statistical test.

We appreciate this point. We will conduct statistical tests to quantify the degree of bimodality and clarify these findings in the results.

Moreover, the two data sets differ in too many aspects to allow for any conclusions about whether the comparisons reflect differences in species (mouse vs. monkey), anatomy (L2/3-L4 vs. V1-V4), or recording technique (calcium imaging vs. extracellular spiking).

We agree that the methodological and anatomical differences between the mouse and monkey datasets make any direct cross-species comparisons hard to interpret. We explicitly discuss this point in the Discussion section. We will add a section within the Discussion entitled “Limitations of this study”. We will further emphasize that our goal is not to attempt a direct quantitative comparison across species. We will further emphasize that the two experiments differ in terms of: (i) differences in recording modalities (calcium vs. electrophysiology) and associated differences in temporal resolution, neuronal types, and SNR, (ii) cortical targets (layers vs. areas), (iii) sample size, (iv) stimuli, (v) task conditions. In the revised manuscript, we will further highlight that our primary aim is to investigate inter-areal interactions within each species rather than to draw comparisons across species.

Reviewer #2:

Summary:

In this work, the authors investigated the extent of shared variability in cortical population activity in the visual cortex in mice and macaques under conditions of spontaneous activity and visual stimulation. They argue that by studying the average response to repeated presentations of sensory stimuli, investigators are discounting the contribution of variable population responses that can have a significant impact at the single trial level. They hypothesized that, because these fluctuations are to some degree shared across cortical populations depending on the sources of these fluctuations and the relative connectivity between cortical populations within a network, one should be able to predict the response in one cortical population given the response of another cortical population on a single trial, and the degree of predictability should vary with factors such as retinotopic overlap, visual stimulation, and the directionality of canonical cortical circuits.

To test this, the authors analyzed previously collected and publicly available datasets. These include calcium imaging of the primary visual cortex in mice and electrophysiology recordings in V1 and V4 of macaques under different conditions of visual stimulation. The strength of this data is that it includes simultaneous recordings of hundreds of neurons across cortical layers or areas. However, the weaknesses of calcium dynamics (which has lower temporal resolution and misses some non-linear dynamics in cortical activity) and multi-unit envelope activity (which reflects fluctuations in population activity rather than the variance in individual unit spike trains), underestimate the variability of individual neurons. The authors deploy a regression model that is appropriate for addressing their hypothesis, and their analytic approach appears rigorous and well-controlled.

We agree that both calcium imaging and multi-unit envelope recordings have inherent limitations in capturing the variability of individual neuron spiking. Among other factors, the slower temporal resolution of calcium signals can blur fast spiking events, and multi-unit envelopes can mask single-unit heterogeneity. In the Discussion, we will explicitly mention these modality-specific caveats and note that our approach is meant to capture shared variability at the population level rather than the fine temporal structure of individual neurons and individual spikes.

From their analysis, they found that there was significant predictability of activity between layer II/III and layer IV responses in mice and V1 and V4 activity in macaques, although the specific degree of predictability varied somewhat with the condition of the comparison with some minor differences between the datasets. The authors deployed a variety of analytic controls and explored a variety of comparisons that are both appropriate and convincing that there is a significant degree of predictability in population responses at the single trial level consistent with their hypothesis. This demonstrates that a significant fraction of cortical responses to stimuli is not due solely to the feedforward response to sensory input, and if we are to understand the computations that take place in the cortex, we must also understand how sensory responses interact with other sources of activity in cortical networks. However, the source of these predictive signals and their impact on function is only explored in a limited fashion, largely due to limitations in the datasets. Overall, this work highlights that, beyond the traditionally studied average evoked responses considered in systems neuroscience, there is a significant contribution of shared variability in cortical populations that may contextualize sensory representations depending on a host of factors that may be independent of the sensory signals being studied.

We will include a section within the Discussion to emphasize the limitations in the datasets used in this study. We also agree and appreciate the reviewer’s description and will borrow some of the reviewer’s terminology to provide context in the Discussion section.

The different recording modalities and comparisons (within vs. across cortical areas) limit the interpretability of the inter-species comparisons.

We agree that the methodological and anatomical differences between the mouse and monkey datasets make any direct cross-species comparisons hard to interpret. We explicitly discuss this point in the Discussion section. We will add a section within the Discussion entitled “Limitations of this study”. We will further emphasize that our goal is not to attempt a direct quantitative comparison across species. We will further emphasize that the two experiments differ in terms of: (i) differences in recording modalities (calcium vs. electrophysiology) and associated differences in temporal resolution, neuronal types, and SNR, (ii) cortical targets (layers vs. areas), (iii) sample size, (iv) stimuli, (v) task conditions. In the revised manuscript, we will further highlight that our primary aim is to investigate inter-areal interactions within each species rather than to draw comparisons across species.

Strengths:

This work considers a variety of conditions that may influence the relative predictability between cortical populations, including receptive field overlap, latency that may reflect feed-forward or feedback delays, and stimulus type and sensory condition. Their analytic approach is well-designed and statistically rigorous. They acknowledge the limitations of the data and do not over-interpret their findings.

Weaknesses:

The different recording modalities and comparisons (within vs. across cortical areas) limit the interpretability of the inter-species comparisons.The mechanistic contribution of known sources or correlates of shared variability (eye movements, pupil fluctuations, locomotion, whisking behaviors) were not considered, and these could be driving or a reflection of much of the predictability observed and explain differences in spontaneous and visual activity predictions.

We also appreciate this important point. We agree that multiple behavioral factors may significantly contribute to shared variability. In our analyses of the mouse data, we addressed non-visual influences by projecting out “non-visual ongoing neuronal activity” (as shown in Figure 6C, following the approach in Stringer et al. 2019). Additionally, we will further evaluate the contribution of behavioral measures available in the open dataset—such as running speed, whisking, pupil area, and “eigenface” components– to predictivity of neuronal responses.

For the macaque data, the head-fixed and eye-fixation conditions help minimize some of these other potential behavioral contributions. Moreover, we have performed comparisons of eyes-open versus eyes-closed conditions (see Figure 5D). We will also analyze pupil size specifically for the lights-off condition. We do not have access to any other behavioral data from monkeys.

Previous work has explored correlations in activity between areas on various timescales, but this work only considered a narrow scope of timescales.

We appreciate this suggestion. We will perform additional analyses to evaluate predictivity at different temporal scales, as suggested.

The observation that there is some degree of predictability is not surprising, and it is unclear whether changes in observed predictability with analysis conditions are informative of a particular mechanism or just due to differences in the variance of activity under those conditions. Some of these issues could be addressed with further analysis, but some may be due to limitations in the experimental scope of the datasets and would require new experiments to resolve.

Our initial analyses in Fig.6A examined the effect of variance in activity and predictability in mice. As the reviewer intuited, there is a correlation between variance and predictability, at least when presenting a stimulus. Importantly, however, this is not the case when predicting activity in the absence of any stimulus. In the macaque, we cannot compute the variance across stimuli in the checkerboard case (single stimulus), but we will compute it for the conditions of the 4 moving bars. In addition, inspired by the reviewer’s question, we will perform an analysis where we further normalize the variance in activity.

We would like to note that our key contribution is not to merely show that some degree of predictability is possible (which we agree is not surprising) but rather: (i) to use a simple approach to quantify this predictability, (ii) to assess directional differences in predictability, (iii) to evaluate how this predictability depends on neuronal properties and receptive field overlap, (iv) how it depends on the stimuli, and, importantly, (v) to compare predictability during visual stimulation versus absence of visual input.

We agree with the limitations in the datasets. We will include a section within the Discussion to emphasize these limitations.

Reviewer #3:

Neural activity in the visual cortex has primarily been studied in terms of responses to external visual stimuli. While the noisiness of inputs to a visual area is known to also influence visual responses, the contribution of this noisy component to overall visual responses has not been well characterized.

In this study, the authors reanalyze two previously published datasets - a Ca++ imaging study from mouse V1 and a large-scale electrophysiological study from monkey V1-V4. Using regression models, they examine how neural activity in one layer (in mice) or one cortical area (in monkeys) predicts activity in another layer or area. Their main finding is that significant predictions are possible even in the absence of visual input, highlighting the influence of non-stimulus-related downstream activity on neural responses. These findings can inform future modeling work of neural responses in the visual cortex to account for such non-visual influences.

A major weakness of the study is that the analysis includes data from only a single monkey. This makes it hard to interpret the data as the results could be due to experimental conditions specific to this monkey, such as the relative placement of electrode arrays in V1 and V4.

We will add data from at least two more monkeys, as suggested by the reviewer:

● First, we will include a second monkey from the same dataset [1]. The reason this monkey was not included in the original submission is that the dataset for this second monkey consisted of much less data than the original. For example, for the lights-off condition, the number of V4 channels with signal-to-noise ratio greater than 2 (recommended electrodes to use by dataset authors) is 9-12 in this second monkey, compared to 68-74 in the first monkey [1]. However, we will still add results for this second monkey.

● Additionally, we will include data from a new monkey by collaborating with the Ponce lab who will collect new data for this study.

The authors perform a thorough analysis comparing regression-based predictions for a wide variety of combinations of stimulus conditions and directions of influence. However, the comparison of stimulus types (Figure 4) raises a potential concern. It is not clear if the differences reported reflect an actual change in predictive influence across the two conditions or if they stem from fundamental differences in the responses of the predictor population, which could in turn affect the ability to measure predictive relationships. The authors do control for some potential confounds such as the number of neurons and self-consistency of the predictor population. However, the predictability seems to closely track the responsiveness of neurons to a particular stimulus. For instance, in the monkey data, the V1 neuronal population will likely be more responsive to checkerboards than to single bars. Moreover, neurons that don't have the bars in their RFs may remain largely silent. Could the difference in predictability be just due to this? Controlling for overall neuronal responsiveness across the two conditions would make this comparison more interpretable.

This is also a valid concern. As the reviewer noted, we controlled for the number of neurons and degree of self-consistency (Fig. 3A, 3C), and this was always done within their respective stimulus type.

As the reviewer intuits, in Fig. 6A in mice, we show that predictability correlates with neuronal responsiveness. This observation only held during the stimulus condition and not during the gray screen condition. We also showed correlations with self-consistency metrics as a proxy for responsiveness in Fig. 6A and 6C. However, we will directly assess the impact of responsiveness in two ways: (i) by correlating predictability directly with neuronal responsiveness and (ii) by following the same subsampling approach in Fig. 3 to normalize the degree of responsiveness and recompute the predictability metrics.

REFERENCES

(1) Chen, X., Morales-Gregorio, A., Sprenger, J., Kleinjohann, A., Sridhar, S., van Albada, S.J., Grün, S., and Roelfsema, P.R. (2022). 1024-channel electrophysiological recordings in macaque V1 and V4 during resting state. Sci Data 9, 77. https://doi.org/10.1038/s41597-022-01180-1.

(2) Stringer, C., Pachitariu, M., Steinmetz, N., Carandini, M., and Harris, K.D. (2019). High-dimensional geometry of population responses in visual cortex. Nature 571, 361–365. https://doi.org/10.1038/s41586-019-1346-5.

(3) Stringer, C., Pachitariu, M., Carandini, M., and Harris, K. (2018). Recordings of 10,000 neurons in visual cortex in response to 2,800 natural images. (Janelia Research Campus). https://doi.org/10.25378/janelia.6845348.v4 https://doi.org/10.25378/janelia.6845348.v4.