Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks

  1. Amirhossein Farzmahdi
  2. Wilbert Zarco
  3. Winrich A Freiwald
  4. Nikolaus Kriegeskorte
  5. Tal Golan  Is a corresponding author
  1. Rockefeller University, United States
  2. Columbia University, United States

Abstract

Primates can recognize objects despite 3D geometric variations such as in-depth rotations. The computational mechanisms that give rise to such invariances are yet to be fully understood. A curious case of partial invariance occurs in the macaque face-patch AL and in fully connected layers of deep convolutional networks in which neurons respond similarly to mirror-symmetric view (e.g., left and right profiles). Why does this tuning develop? Here, we propose a simple learning-driven explanation for mirror-symmetric viewpoint tuning. We show that mirror-symmetric viewpoint tuning for faces emerges in the fully connected layers of convolutional deep neural networks trained on object recognition tasks, even when the training dataset does not include faces. First, using 3D objects rendered from multiple views as test stimuli, we demonstrate that mirror-symmetric viewpoint tuning in convolutional neural network models is not unique to faces: it emerges for multiple object categories with bilateral symmetry. Second, we show why this invariance emerges in the models. Learning to discriminate among bilaterally symmetric object categories induces reflection-equivariant intermediate representations. AL-like mirror-symmetric tuning is achieved when such equivariant responses are spatially pooled by downstream units with sufficiently large receptive fields. These results explain how mirror-symmetric viewpoint tuning can emerge in neural networks, providing a theory of how they might emerge in the primate brain. Our theory predicts that mirror-symmetric viewpoint tuning can emerge as a consequence of exposure to bilaterally symmetric objects beyond the category of faces, and that it can generalize beyond previously experienced object categories.

Data availability

The stimulus set and the source code required for reproducing our results are available at https://gitfront.io/r/afarzmahdi/p666tmWy7YuY/AL-symmetry-manuscript-codes/.

The following data sets were generated

Article and author information

Author details

  1. Amirhossein Farzmahdi

    Laboratory of Neural Systems, Rockefeller University, New York, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-6926-546X
  2. Wilbert Zarco

    Laboratory of Neural Systems, Rockefeller University, New York, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-3599-0476
  3. Winrich A Freiwald

    Laboratory of Neural Systems, Rockefeller University, New York, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-8456-5030
  4. Nikolaus Kriegeskorte

    Zuckerman Mind Brain Behavior Institute, Columbia University, New York, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-7433-9005
  5. Tal Golan

    Zuckerman Mind Brain Behavior Institute, Columbia University, New York, United States
    For correspondence
    golan.neuro@bgu.ac.il
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-7940-7473

Funding

National Eye Institute (R01EY021594)

  • Winrich A Freiwald

National Eye Institute (R01EY029998)

  • Winrich A Freiwald

National Institute of Neurological Disorders and Stroke (RF1NS128897)

  • Nikolaus Kriegeskorte

Naval Research Laboratory (N00014-20-1-2292)

  • Winrich A Freiwald

Charles H. Revson Foundation

  • Tal Golan

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Copyright

© 2024, Farzmahdi et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 909
    views
  • 151
    downloads
  • 1
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Amirhossein Farzmahdi
  2. Wilbert Zarco
  3. Winrich A Freiwald
  4. Nikolaus Kriegeskorte
  5. Tal Golan
(2024)
Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks
eLife 13:e90256.
https://doi.org/10.7554/eLife.90256

Share this article

https://doi.org/10.7554/eLife.90256

Further reading

    1. Neuroscience
    Kayson Fakhar, Fatemeh Hadaeghi ... Claus C Hilgetag
    Research Article

    Efficient communication in brain networks is foundational for cognitive function and behavior. However, how communication efficiency is defined depends on the assumed model of signaling dynamics, e.g., shortest path signaling, random walker navigation, broadcasting, and diffusive processes. Thus, a general and model-agnostic framework for characterizing optimal neural communication is needed. We address this challenge by assigning communication efficiency through a virtual multi-site lesioning regime combined with game theory, applied to large-scale models of human brain dynamics. Our framework quantifies the exact influence each node exerts over every other, generating optimal influence maps given the underlying model of neural dynamics. These descriptions reveal how communication patterns unfold if regions are set to maximize their influence over one another. Comparing these maps with a variety of brain communication models showed that optimal communication closely resembles a broadcasting regime in which regions leverage multiple parallel channels for information dissemination. Moreover, we found that the brain’s most influential regions are its rich-club, exploiting their topological vantage point by broadcasting across numerous pathways that enhance their reach even if the underlying connections are weak. Altogether, our work provides a rigorous and versatile framework for characterizing optimal brain communication, and uncovers the most influential brain regions, and the topological features underlying their influence.

    1. Neuroscience
    Poortata Lalwani, Thad Polk, Douglas D Garrett
    Research Article

    Moment-to-moment neural variability has been shown to scale positively with the complexity of stimulus input. However, the mechanisms underlying the ability to align variability to input complexity are unknown. Using a combination of behavioral methods, computational modeling, fMRI, MR spectroscopy, and pharmacological intervention, we investigated the role of aging and GABA in neural variability during visual processing. We replicated previous findings that participants expressed higher variability when viewing more complex visual stimuli. Additionally, we found that such variability modulation was associated with higher baseline visual GABA levels and was reduced in older adults. When pharmacologically increasing GABA activity, we found that participants with lower baseline GABA levels showed a drug-related increase in variability modulation while participants with higher baseline GABA showed no change or even a reduction, consistent with an inverted-U account. Finally, higher baseline GABA and variability modulation were jointly associated with better visual-discrimination performance. These results suggest that GABA plays an important role in how humans utilize neural variability to adapt to the complexity of the visual world.