1. Neuroscience

Selfee, self-supervised features extraction of animal behaviors

  1. Yinjun Jia  Is a corresponding author
  2. Shuaishuai Li
  3. Xuan Guo
  4. Bo Lei
  5. Junqiang Hu
  6. Xiao-Hong Xu
  7. Wei Zhang  Is a corresponding author
  1. Tsinghua University, China
  2. Chinese Academy of Sciences, China
https://doi.org/10.7554/eLife.76218
Share this article
Cite this article
  1. Yinjun Jia
  2. Shuaishuai Li
  3. Xuan Guo
  4. Bo Lei
  5. Junqiang Hu
  6. Xiao-Hong Xu
  7. Wei Zhang
(2022)
Selfee, self-supervised features extraction of animal behaviors
eLife 11:e76218.
https://doi.org/10.7554/eLife.76218
  2. Download BibTeX
  3. Download .RIS

Abstract

Fast and accurately characterizing animal behaviors is crucial for neuroscience research. Deep learning models are efficiently used in laboratories for behavior analysis. However, it has not been achieved to use an end-to-end unsupervised neural network to extract comprehensive and discriminative features directly from social behavior video frames for annotation and analysis purposes. Here, we report a self-supervised feature extraction (Selfee) convolutional neural network with multiple downstream applications to process video frames of animal behavior in an end-to-end way. Visualization and classification of the extracted features (Meta-representations) validate that Selfee processes animal behaviors in a way similar to human perception. We demonstrate that Meta-representations can be efficiently used to detect anomalous behaviors that are indiscernible to human observation and hint in-depth analysis. Furthermore, time-series analyses of Meta-representations reveal the temporal dynamics of animal behaviors. In conclusion, we present a self-supervised learning approach to extract comprehensive and discriminative features directly from raw video recordings of animal behaviors and demonstrate its potential usage for various downstream applications.

Data availability

Major data used in this study were uploaded to Dryad, including pretrained weights. Data could be accessed via:https://doi.org/10.5061/dryad.brv15dvb8.With the uploaded dataset and pretrained weights, our experiments could be replicated. However, due to its huge size and the limited internet service resources, we are currently not able to share our full training dataset. The full dataset is as large as 400GB, which is hard to upload to a public server and will be difficult for others users to download.For training dataset, it would be available from the corresponding author upon reasonable request (wei_zhang@mail.tsinghua.edu.cn), and then we can discuss how to transfer the dataset. No project proposal is needed as long as the dataset is not used for any commercial purpose.Our Python scripts could be accessed on GitHub: https://github.com/EBGU/SelfeeOther software used in our project include ImageJ(https://imagej.net/software/fiji/) and GraphPad Prism(https://www.graphpad.com/).All data used to plot graphs and charts in the manuscript can be fully accessed on Dryad (DOI 10.5061/dryad.brv15dvb8).

The following data sets were generated
The following previously published data sets were used

Article and author information

Author details

  1. Yinjun Jia

    IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China
    For correspondence
    jyj20@mails.tsinghua.edu.cn
    Competing interests
    The authors declare that no competing interests exist.

  2. Shuaishuai Li

    Institute of Neuroscience, Chinese Academy of Sciences, Beijing, China
    Competing interests
    The authors declare that no competing interests exist.

  3. Xuan Guo

    IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China
    Competing interests
    The authors declare that no competing interests exist.

  4. Bo Lei

    IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China
    Competing interests
    The authors declare that no competing interests exist.

  5. Junqiang Hu

    IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China
    Competing interests
    The authors declare that no competing interests exist.

  6. Xiao-Hong Xu

    Institute of Neuroscience, Chinese Academy of Sciences, Beijing, China
    Competing interests
    The authors declare that no competing interests exist.

  7. Wei Zhang

    IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China
    For correspondence
    wei_zhang@mail.tsinghua.edu.cn
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-0512-3096

Funding

National Natural Science Foundation of China (32022029)

  • Wei Zhang

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Animal experimentation: All mating experiments were approved by the Animal Care and Use Committee of the Institute of Neuroscience, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, China (IACUC No. NA-016-2016)All studies and experimental protocols of CIS and OFT were approved by Institutional Animal Care and Use Committee (IACUC) at Tsinghua University (No. 19-ZY1). Experiments were performed using the principles outlined in the Guide for the Care and Use of Laboratory Animals of Tsinghua University.

Copyright

© 2022, Jia et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 4,077
    views
  • 762
    downloads
  • 15
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Yinjun Jia
  2. Shuaishuai Li
  3. Xuan Guo
  4. Bo Lei
  5. Junqiang Hu
  6. Xiao-Hong Xu
  7. Wei Zhang
(2022)
Selfee, self-supervised features extraction of animal behaviors
eLife 11:e76218.
https://doi.org/10.7554/eLife.76218

Share this article

https://doi.org/10.7554/eLife.76218

Categories and tags

Research organisms

Further reading

    1. Neuroscience

    Rhythmic circuit function is more robust to changes in synaptic than intrinsic conductances

    Zachary Fournier, Leandro M Alonso, Eve Marder
    Research Article

    Circuit function results from both intrinsic conductances of network neurons and the synaptic conductances that connect them. In models of neural circuits, different combinations of maximal conductances can give rise to similar activity. We compared the robustness of a neural circuit to changes in their intrinsic versus synaptic conductances. To address this, we performed a sensitivity analysis on a population of conductance-based models of the pyloric network from the crustacean stomatogastric ganglion (STG). The model network consists of three neurons with nine currents: a sodium current (Na), three potassium currents (Kd, KCa, KA), two calcium currents (CaS and CaT), a hyperpolarization-activated current (H), a non-voltage-gated leak current (leak), and a neuromodulatory current (MI). The model cells are connected by seven synapses of two types, glutamatergic and cholinergic. We produced one hundred models of the pyloric network that displayed similar activities with values of maximal conductances distributed over wide ranges. We evaluated the robustness of each model to changes in their maximal conductances. We found that individual models have different sensitivities to changes in their maximal conductances, both in their intrinsic and synaptic conductances. As expected, the models become less robust as the extent of the changes increases. Despite quantitative differences in their robustness, we found that in all cases, the model networks are more sensitive to the perturbation of their intrinsic conductances than their synaptic conductances.

    1. Neuroscience

    Cognition: When working memory works for our goals

    Jacob A Miller
    Insight

    When navigating environments with changing rules, human brain circuits flexibly adapt how and where we retain information to help us achieve our immediate goals.

    1. Neuroscience

    Cerebellar Purkinje cells control posture in larval zebrafish (Danio rerio)

    Franziska Auer, Katherine Nardone ... David Schoppik
    Research Article

    Cerebellar dysfunction leads to postural instability. Recent work in freely moving rodents has transformed investigations of cerebellar contributions to posture. However, the combined complexity of terrestrial locomotion and the rodent cerebellum motivate new approaches to perturb cerebellar function in simpler vertebrates. Here, we adapted a validated chemogenetic tool (TRPV1/capsaicin) to describe the role of Purkinje cells — the output neurons of the cerebellar cortex — as larval zebrafish swam freely in depth. We achieved both bidirectional control (activation and ablation) of Purkinje cells while performing quantitative high-throughput assessment of posture and locomotion. Activation modified postural control in the pitch (nose-up/nose-down) axis. Similarly, ablations disrupted pitch-axis posture and fin-body coordination responsible for climbs. Postural disruption was more widespread in older larvae, offering a window into emergent roles for the developing cerebellum in the control of posture. Finally, we found that activity in Purkinje cells could individually and collectively encode tilt direction, a key feature of postural control neurons. Our findings delineate an expected role for the cerebellum in postural control and vestibular sensation in larval zebrafish, establishing the validity of TRPV1/capsaicin-mediated perturbations in a simple, genetically tractable vertebrate. Moreover, by comparing the contributions of Purkinje cell ablations to posture in time, we uncover signatures of emerging cerebellar control of posture across early development. This work takes a major step towards understanding an ancestral role of the cerebellum in regulating postural maturation.