Reshaped Random Projections models outperform Random Projections models.

(A) A short segment of the spiking activity of 100 cortical neurons used for the analysis and comparison of different statistical models of population activity (see Methods). (B) Schematics of the neural circuits that implement the different random projections models we compared: The “standard” Random Projections model, where the coefficients aij that define the projections are randomly selected and fixed whereas the factors λi are learned (see text). The Reshaped Random Projections model, where the coefficients aij that define the projections are tuned and the factors λis are fixed. The backpropagation model, where we tune both the aij s and λis. (C) The predicted probability of individual populations’ activity patterns as a function of the observed probability for RP, Reshaped and Backpropagation models. Gray funnels denote 99% confidence interval. (D) Average performance of models of the three classes is shown as a function of the number of projections, measured by log-likelihood, over 100 sets of randomly selected groups of 50 neurons. Reshaped models outperform RP models and are on par with backpropagation; the shaded area denotes the standard error over 100 models. (E-F) Mean firing rates of projection neurons and mean correlation between projections. Reshaped models show lower correlations and lower firing rates compared to RP and backpropagation models. Standard errors are smaller than marker’s size, hence invisible.

Reshaped RP models that use homeostatic synaptic normalization outperform RP models and bounded RP models.

(A) Schematic drawing of the different models we studied: standard RP model, unconstrained reshaped model, and two types of constrained reshaped models: Bounded models in which each synapse separately obeys Σ ij |aij | < θ during learning, and normalized input reshaped models, where we fix the total synaptic weight of incoming synapses Σ ij |aij | = ϕ. (B) The mean log-likelihood of the models is shown as a function of the total available budget. The normalized input reshaped RP models give optimal results for a wide range of values of available synaptic budget, outperforming the bounded models and the RP model. (C) The mean log-likelihood of the models is shown as a function of the total used budget. Aside from the bounded models, all other models are the same as in (B) by construction. For high available budget values, bounded models show better performance while utilizing a lower synaptic budget, similar to the unconstrained reshape model. (D) Comparison of the performance of 100 individual examples of each model class and their corresponding RP model, where all models relied on the same set of initial projections. Normalized input models outperformed the RP models in all cases (all points are above the diagonal), while all bounded models were worse (points below the diagonal). (E-F) The mean correlation and firing rates of projections as a function of the model’s cost. Normalized reshape models show low correlations and mean firing rates, similar to unconstrained reshaped models. Note that in panels B, C, E, and F, the standard errors are smaller than the marker size, and are therefore invisible.

Homeostatic models firing rates.

(A) Likelihood values of the different models we consider here. Error bars denote standard error over 100 different models. (B) Projections mean firing rates during reshape. Standard error is smaller than marker size, hence cannot be seen (C) A histogram of the projections firing rates of unconstrained models and different homeostatic input models. (D) Projections mean firing rates after reshaping vs. the projection initial firing rate, for different homeostatic models and unconstrained model. Error bars denote standard deviation over 100 different models. (E) The cumulative density function of the projections firing rate relative change during reshape, |FRfinal − FRinitial| /F Rinitial.

Models that rely on projections that use random connectivity show similar performance to models that use the correct connectivity.

(A) Synthetic population activity data is sampled from an RP model with known connectivity (i.e., the “ground truth” model; see Methods). (B) Homeostatic reshaped random projections models that differ in their connectivity are learned to fit the synthetic data. The “True connectivity” model uses projections whose connectivity is identical to the “ground truth” model. The “Random connectivity” model uses projections that are randomly sampled using sparse random connectivity. The “Full connectivity” model is a homeostatic reshaped model that uses projections with full connectivity. (C) The mean log-likelihood of the models is shown as a function of the model’s cost. The true connectivity model is only slightly better than the random connectivity model, with both outperforming the full connectivity model for low model budget values. (D) The mean correlation between the activity of the projection neurons, shown as a function of the model cost. We note that true and random connectivity models are indistinguishable. (E) The firing rates of the projection neurons, shown as a function of the model cost. (F) The performance of homeostatic reshaped RP models, shown as a function of their normalized in-degree of the projections (0–disconnected, 1–fully connected), for different normalization values, shown by the model’s synaptic cost. (G) The performance of the homeostatic reshaped RP models, shown as a function of the synaptic cost normalized by the in-degree of the projections. Curves of different cost values coincide, suggesting a fixed optimal cost/activity ratio. Note that in panels D-G the standard errors over 100 models are smaller than the size of markers and are, therefore, invisible.

A variety of homeostatic synaptic normalization mechanisms produce similar model behavior using a wide range of random projection parameters.

(A) Schematic drawings of the different homeostatic models we compared: Homeostatic input models in which we fixed the total synaptic weight of the incoming synapses Σj |aij | = ϕ; Homeostatic output models in which we fixed the total synaptic weight of the outgoing synapses Σi |aij | = ϕ; and Homeostatic circuit models in which we fixed the total synaptic weight of the whole synaptic circuit Σ ij |aij | = ϕ. (B) The mean log-likelihood of models, shown as a function of the total used synaptic cost. All three homeostatic model variants show similar behavior. (C) Schematic drawing of how projections rotate during reshaping: starting from the initial projections (grey lines), they rotate to their reshaped orientation (black lines) by angle θi. (D) Rotation angles after reshaping, shown for different pairs of models. All four panels show models that initialized with the same set of projections. The different labels specify the constraint type and strength, namely, the specific value of ϕ and θ. (E) The mean rotation angle of the projections due to reshaping, shown as a function of the model synaptic budget.

Sigmoid Random Projections models outperform Step Random Projections models.

(A) Average performance of models as a function of the number of projections, measured by log-likelihood. Sigmoid RP models outperform step RP models. (B-C) Mean firing rates of projection neurons and mean correlation between projections, step RP models show slightly lower correlations and firing rates compared to sigmoid RP models. In all panels shaded area denotes standard error over 100 models.

Homeostatic model variants show similar firing rates and correlation, but different rotation angles.

(A) and (B) Mean correlation between the projection neurons and firing rates as a function of the model cost of the three different homeostatic models we compared. All three homeostatic models show a similar behavior. Note that the standard errors over 100 models are smaller than the marker’s size and are, therefore, invisible. (C) and (D) Histograms of rotation angles of the projections after learning in the case of the unconstrained reshape models and different homeostatic input (C) and homeostatic circuit (D) models. The legend specifies the constraint type and strength, namely, the specific value of ϕ.