Unraveling the developmental dynamic of visual exploration of social interactions in autism
- Psychiatry Department, Faculty of Medicine, University of Geneva, Switzerland
- Faculte de Psychologie et Science de l’Education, University of Geneva, Switzerland
- Instituto de Investigaciones en Matemáticas Aplicadas y en Sistemas, Universidad Nacional Autónoma de México, Ciudad Universitaria, Mexico
- Fondation Pôle Autisme, Switzerland
- College of Medicine, Mohammed Bin Rashid University of Medicine and Health Sciences, United Arab Emirates
- Bosch Sensortec GmbH, Germany
Figures
Figure 1
Proximity Index method illustration.
Referent gaze data distribution (‘reference’) was created using gaze coordinates from 51 typically developing (TD) males (aged 3.48±1.29 years old). Upper row: two example frames with gaze coordinates of TD children (blue dots) used to define the ‘reference’ (delimited by contours) and gaze data from a three-year-old male with autism spectrum disorder (ASD) (whose gaze coordinates are depicted as a red circle). Hotter contour color indicates the area of higher density of distribution of gaze in the TD group, meaning that a particular area was more appealing for a higher number of TD preschoolers for the given frame; the Proximity Index value for the 3-year-old male with ASD for the frame on the left had a value of 0.39 and for the frame on the right a value of 0. Lower row: Proximity Index values for the visual exploration of the 3-year-old boy with ASD over the entire video with the mean Proximity Index value indicated by the dashed red lines.
Figure 2
Mean proximity index (PI) comparison between groups.
Violin plots illustrate the distribution of Proximity Index (PI) values for two groups: typically developing (TD) in blue (n = 51) and autism spectrum disorder (ASD) in red (n = 166). The error bars on each plot represent the 95% confidence intervals around the means. Statistical significance of the differences between means was assessed using a two-sample t-test. The PI values for the TD group were derived using a leave-one-out approach, where the PI for each ASD child was calculated based on the referent gaze data from the 51 TD children in the original sample.
Figure 3
Proximity Index and its relation to behavioral phenotype in children with autism spectrum disorder (ASD).
Loadings on the latent component were derived using partial least squares correlation analysis in the sample of 166 children with ASD. The cross-correlation matrix consisted of the Proximity Index on the imaging (A) side and 12 variables on the behavior (B) side. The behavioral matrix encompassed two domains of autistic symptoms assessed by ADOS-2: Social Affect (SA) and Repetitive and Restricted Behaviors (RRB); six subscales of the PEP-3: Verbal and Preverbal Cognition (VPC), Expressive Language (EL), Receptive Language (RL), Fine Motor Skills (FM), Gross Motor Skills (GM), and Oculomotor Imitation (OMI); and four domains from VABS-II: Communication (COM), Daily Living Skills (DAI), Socialization (SOC), and Motor Skills (MOT). Age was controlled for by regressing it out from both sides (A and B) of the cross-correlation matrix. There was a positive correlation between the Proximity Index and all measures of developmental (PEP-3) and adaptive functioning (VABS-II). Error bars represent the bootstrapping 5th to 95th percentiles. Results that were not robust are indicated by a gray boxplot color.
Figure 4
Focal and ambient fixation modes, between-group comparison, and their relation to the Proximity Index (PI) across ASD and TD groups.
(A1) Relative proportion of focal and ambient fixations in a sample of 51 TD children and 166 ASD children. Box-and-whisker plots illustrate the distribution of fixation proportions. The interquartile range (IQR) is represented by each box, with the median shown as a horizontal line. Whiskers extend to the most extreme data points within 1.5 IQR from the box, as per Tukey's method. Differences between groups were statistically assessed using the Mann-Whitney U test, with asterisks (****) indicating p-values less than 0.0001. (A2 & A3): Scatter plots show the correlation between the proportion of focal (A2) or ambient (A3) fixations and PI. Red points represent ASD individuals and blue points represent TD individuals. Spearman's correlation was used for analysis. Each group's data is fitted with its own linear regression line and includes 95% confidence bands.
Figure 5
Visual salience group differences.
(A) Illustration of the graph-based visual saliency (GBVS) salience model (Full model combining five channels: I-Intensity, O-Orientation, C-Color, F-Flicker, M-Motion). From top to bottom: Saliency map extracted for a given frame, Saliency map overlay on the original image, Original image with 15% most salient parts shown. (B) Box plot depicting mean receiver operating characteristic (ROC) values, derived framewise from full salience maps and fixation coordinates (x,y), for a sample of 51 TD (Typically Developing) and 166 ASD (Autism Spectrum Disorder) children. Boxes indicate the interquartile range (IQR) and medians are shown as horizontal lines within the boxes. Whiskers extend to the farthest data points not exceeding 1.5 times the IQR from the box edges, in line with Tukey's method. Framewise statistical between group differences were evaluated using the Wilcoxon paired test, with asterisks (***) indicating p-values less than 0.001. Effect size is calculated using formula , (Rosenthal, 1991).
Figure 6
Proximity Index and its relation to movie content.
(A) From top to down: In red, the average proximity index (PI) from 166 children with autism spectrum disorder (ASD) over time frames. Red-shaded regions denote a 95% confidence interval of the mean, gray-shaded regions mark the moments of the significant drop in mean values of the PI (below 2.5 SD compared to the theoretical mean of 1); Dark blue: Visual complexity over time frames; Green: Social complexity over time frames; the last panel denotes moments of the movie with the monologue, directed speech, frame switching, or moments involving moving background. (B) PLS-C illustration with PI on the A side and on the B side: Visual complexity, Social Complexity, Monologue, Directed Speech, Frame switch, Moving background and graph-based visual saliency (GBVS), the salience model derived receiver operating characteristic curve (ROC) scores for children with ASD (average ROC framewise). Positive correlation between the Proximity Index and was found between the Proximity Index and monologue, frame switch, moving background and also visual salience. PI negatively correlated with the social and visual complexity, as well as directed speech. Error bars represent the bootstrapping 5th to 95th percentiles.
Figure 7
Proximity Index and its relation to behavioral phenotype in children with autism spectrum disorder (ASD) seen two times a year apart.
Sample comprised 81 children with ASD who had valid eye-tracking recording and a complete set of behavioral phenotype measures a year after the baseline (T2). The PI for this paired longitudinal cohort was established using an age-matched reference group of 29 Typically Developing (TD) children. Loadings on the latent component were derived using PLS correlation analysis. The cross-correlation matrix included the Proximity Index (PI) on the imaging side A and three behavioral variables on the B side. The behavioral matrix accounted for two domains of autistic symptoms as assessed by ADOS-2, Verbal and Preverbal Cognition (VPC) from the PEP-3, and the Adaptive Behavior Composite Score from the VABS-II. Error bars represent the bootstrapping 5th to 95th percentiles. Results that were not robust are indicated by a gray boxplot color. (A) Proximity index (PI) obtained at T1 and phenotype measures obtained a year later (T2). PI at T1 positively correlated with reduced symptoms at T2 (B) Simultaneous PLS-C: both PI and phenotype measures were obtained at T2. PI at T2 positively correlated with symptoms at T2 and positively with adaptive behavior. Loading on the latent component was obtained using the partial least squares correlation analysis. The cross-correlation matrix was composed of the proximity index-PI on the imaging A and three variables on the behavior B side. The behavior matrix included two domains of autistic symptoms assessed by ADOS-2, Verbal and preverbal cognition (VPC) of PEP-3, and the Adaptive Behavior Composite Score of VABS-II.
Figure 8
Characterization of the evolution of visual exploration patterns in young children with autism spectrum disorder (ASD) and the typically developing (TD) group using a sliding window approach.
Panel A: The sliding window approach applied to the available recordings in our ASD group (red) and our TD group (blue); Panel B: gaze dispersion in two groups for the sliding windows n°7 and n°42 (mean age of windows 2.18 and 3.64 years, respectively); each circle represents a window encompassing 20 recordings; Panel C: Comparison of the gaze dispersion between two groups using Mean pairwise distance of gaze coordinates on each frame. The dispersion was calculated across 59 sliding windows spanning 1.88–4.28 years of age on average (here again, every circle represents a window encompassing 20 recordings). The windows with filled circles are those where a statistically significant difference between the two groups was shown using permutation testing. Error bars indicate a 95% confidence interval of the mean. As can be seen on panel C, dispersion values diminished in the TD group with advancing age, while the opposite pattern is observed in the ASD group showing a progressively more dispersed gaze behavior in the ASD group during childhood years.
Appendix 1—figure 1
Stability of the normative distribution regarding the normative sample size.
The continuous function was estimated using a kneed Python package using the average variance (over 5150 frames) of average (over 500 bootstrapped samples without replacement) mean pairwise distance of gaze coordinates on the frame (y-axis) for samples sizes ranging from 10 to 50 (x-axis) as the input: elbow point = 18.
Appendix 2—figure 1
Visual salience group differences across channels.
(A) From left to right: full saliency model with all five channels combined and channels taken separately: I-intensity, O-orientation, D-color, F-flicker, and M-motion channel. From top to bottom: Saliency map extracted for a given frame, Saliency map overlay on the original image, Original image with 15% most salient parts shown. B.
Appendix 3—figure 1
Illustration of the measures of social intensity and visual complexity.
(A) Three frames (denoted as a, b, c) illustrate three levels of social intensity; (B) Visual complexity depicted using the edges of the images detected using the Canny method (Canny, 1986) for the frames a, b, and c.
Appendix 4—figure 1
Proximity Index and its relation to behavioral phenotype in children with autism spectrum disorder (ASD) who were seen two times a year apart (the current figure depicts the initial (T1) visit).
Sample comprised 81 children with ASD who had valid eye-tracking recording and a complete set of behavioral phenotype measures a year after the baseline (T2). The PI for this paired longitudinal cohort was established using an age-matched reference group of 29 Typically Developing (TD) children. PI was obtained at T1 and its correlation with the behavioral phenotype measures was assessed at the same time (T1). Loadings on the latent component were derived using PLS correlation analysis. The cross-correlation matrix included the Proximity Index (PI) on the imaging side A and three behavioral variables B. The behavioral matrix accounted for two domains of autistic symptoms as assessed by ADOS-2, Verbal and Preverbal Cognition (VPC) from the PEP-3, and the Adaptive Behavior Composite Score from the VABS-II. Error bars represent the bootstrapping 5th to 95th percentiles. Results that were not robust are indicated by a gray boxplot color. PI at baseline was positively correlated with developmental and adaptive functionng at baseline.
Appendix 5—figure 1
Evolution of visual exploration patterns in young children with autism spectrum disorder (ASD) and the typically developing (TD) group using a sliding window and bootstrapping approach.
The dispersion in 100 bootstrapped samples of ASD recordings is given in red and the original group dispersion in the TD group is shown in blue. Panel A: ASD bootstrapped samples are matched to the TD group with regards to size (n=51) and chronological age; Panel B: ASD bootstrapped samples are matched to the TD group with regards to size (n=51), chronological age and have the DQ within the normal range (above 80); Panel C: ASD bootstrapped samples are matched to the TD group with regards to size (n=51), and chronological age and have the DQ below the normal range (below 80); Panel D1: Evolution of visual exploration patterns in young children with ASD whose developmental age was matched to the chronological age of the TD group using a sliding window approach. Comparison of the gaze dispersion between two groups using Mean pairwise distance of gaze coordinates on each frame. The dispersion was calculated across 60 sliding windows spanning 2.9–4.3 years of mental age on average (every circle represents a window encompassing 20 recordings); Panel D2: The sliding window approach was applied to the ASD bootstrapped samples that are matched to the TD group with regards to size (n=51) while mental age was aligned with the chronological age of the TD group.
Author response image 1
Age distribution in our cross-sectional sample including 51 TD children and 166 children with ASD.
Tables
Table 1
Description of the cross-sectional sample.
| Measures | ASD (n=166) | TD (n=51) | p-value |
|---|---|---|---|
| Mean±SD | Mean±SD | ||
| Age | 3.37±1.16 | 3.48±1.29 | 0.621a |
| Total Symptom Severity Score (ADOS-2 CSS) | 7.19±1.78 | 1.10±0.300 | <0.001a |
| Social Affect (ADOS-2 SA-CSS) | 6.08±2.06 | 1.18±0.478 | <0.001a |
| Repetitive Behaviors & | |||
| Restricted Interests (ADOS-2 RRB CSS) | 8.63±1.85 | 2.16±1.92 | <0.001a |
| Social Interaction (ADI-R: A) | 14.8±5.70 | 1.04±1.39 | <0.001a |
| Communication (ADI-R: B) | 9.97±3.44 | 1.12±1.35 | <0.001a |
| Repetitive Behaviors & | |||
| Restricted Interests (ADI-R: C) | 4.79±2.22 | 0.314±0.678 | <0.001a |
| Age of onset (ADI-R: D) | 3.60±0.997 | 0.078±0.337 | <0.001a |
| Best Estimate IQ | 83.6±24.0 | 119±16.5 | <0.001a |
| VABS-II Adaptive Behavior | 80.2±10.2 | 103±8.21 | <0.001 |
| VABS-II Communication | 80.2±13.7 | 105±8.94 | <0.001 |
| VABS-II Daily Living Skills | 83.7±11.6 | 101±8.25 | <0.001 |
| VABS-II Socialization | 79.2±9.82 | 101±8.49 | <0.001 |
| VABS-II Motor Skills | 88.4±11.5 | 102±11.2 | <0.001a |
-
Note. p-valuesa are obtained using nonparametric Mann-Whitney tests of differences between the two groups.
Author response table 1
Comparison of mean age across 59 age-matched sliding windows for the TD and the ASD group.
pvalues are obtained using t-test of differences between the ASD and TD group across 59 windows.
| TD | ASD | TD | ASD | TD | ASD | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean | SD | Mean | SD p value | Mean | Mean | SD | Mean | SD p value | Mean | Mean | SD | Mean | SD p value | Mean | |||
| SW 1 | 1.88 | 0.242 | 1.94 | 0.118 | 0.813 | SW 21 | 2.78 | 0.226 | 2.86 | 0.065 | 0.975 | SW 41 | 3.53 | 0.239 | 3.61 | 0.082 | 0.931 |
| SW 2 | 1.93 | 0.224 | 1.97 | 0.111 | 0.931 | SW 22 | 2.82 | 0.215 | 2.90 | 0.060 | 0.911 | SW 42 | 3.57 | 0.236 | 3.64 | 0.088 | 0.919 |
| SW 3 | 1.96 | 0.238 | 2.01 | 0.108 | 0.872 | SW 23 | 2.86 | 0.202 | 2.93 | 0.055 | 0.973 | SW 43 | 3.61 | 0.233 | 3.68 | 0.092 | 0.998 |
| SW 4 | 2.00 | 0.251 | 2.05 | 0.111 | 0.880 | SW 24 | 2.90 | 0.193 | 2.96 | 0.059 | 1.000 | SW 44 | 3.65 | 0.230 | 3.73 | 0.091 | 0.951 |
| SW 5 | 2.04 | 0.268 | 2.09 | 0.113 | 0.986 | SW 25 | 2.93 | 0.187 | 3.00 | 0.062 | 0.916 | SW 45 | 3.68 | 0.220 | 3.75 | 0.085 | 0.952 |
| SW 6 | 2.09 | 0.286 | 2.14 | 0.114 | 0.912 | SW 26 | 2.96 | 0.189 | 3.04 | 0.062 | 0.980 | SW 46 | 3.72 | 0.212 | 3.80 | 0.072 | 0.981 |
| SW 7 | 2.14 | 0.300 | 2.18 | 0.112 | 0.936 | SW 27 | 3.00 | 0.197 | 3.07 | 0.060 | 0.935 | SW 47 | 3.76 | 0.211 | 3.83 | 0.061 | 0.996 |
| SW 8 | 2.18 | 0.307 | 2.22 | 0.100 | 0.933 | SW 28 | 3.04 | 0.201 | 3.11 | 0.057 | 0.958 | SW 48 | 3.79 | 0.209 | 3.86 | 0.054 | 0.950 |
| SW 9 | 2.23 | 0.309 | 2.28 | 0.089 | 0.951 | SW 29 | 3.07 | 0.201 | 3.14 | 0.057 | 0.926 | SW 49 | 3.83 | 0.203 | 3.90 | 0.054 | 0.972 |
| SW 10 | 2.28 | 0.308 | 2.33 | 0.066 | 0.965 | SW 30 | 3.11 | 0.204 | 3.18 | 0.063 | 0.952 | SW 50 | 3.87 | 0.199 | 3.94 | 0.062 | 0.920 |
| SW 11 | 2.33 | 0.308 | 2.43 | 0.080 | 0.948 | SW 31 | 3.14 | 0.209 | 3.21 | 0.062 | 0.957 | SW 51 | 3.90 | 0.197 | 3.97 | 0.061 | 0.961 |
| SW 12 | 2.38 | 0.301 | 2.48 | 0.073 | 0.928 | SW 32 | 3.18 | 0.210 | 3.25 | 0.059 | 0.971 | SW 52 | 3.94 | 0.190 | 4.01 | 0.055 | 0.977 |
| SW 13 | 2.43 | 0.296 | 2.52 | 0.070 | 0.947 | SW 33 | 3.21 | 0.213 | 3.29 | 0.054 | 0.982 | SW 53 | 3.97 | 0.184 | 4.04 | 0.049 | 0.939 |
| SW 14 | 2.48 | 0.285 | 2.57 | 0.065 | 0.992 | SW 34 | 3.25 | 0.217 | 3.33 | 0.054 | 0.962 | SW 54 | 4.01 | 0.179 | 4.07 | 0.047 | 0.998 |
| SW 15 | 2.53 | 0.278 | 2.62 | 0.067 | 0.963 | SW 35 | 3.29 | 0.230 | 3.37 | 0.068 | 0.959 | SW 55 | 4.04 | 0.188 | 4.10 | 0.054 | 0.933 |
| SW 16 | 2.57 | 0.271 | 2.66 | 0.071 | 0.936 | SW 36 | 3.33 | 0.240 | 3.42 | 0.070 | 0.977 | SW 56 | 4.07 | 0.199 | 4.14 | 0.068 | 0.998 |
| SW 17 | 2.61 | 0.260 | 2.70 | 0.072 | 0.998 | SW 37 | 3.38 | 0.243 | 3.45 | 0.067 | 0.929 | SW 57 | 4.11 | 0.205 | 4.18 | 0.069 | 0.976 |
| SW 18 | 2.66 | 0.242 | 2.74 | 0.077 | 0.985 | SW 38 | 3.42 | 0.242 | 3.49 | 0.062 | 0.966 | SW 58 | 4.14 | 0.213 | 4.23 | 0.063 | 0,941 |
| SW 19 | 2.70 | 0.238 | 2.78 | 0.076 | 0.950 | SW 39 | 3.45 | 0.244 | 3.53 | 0.057 | 0.992 | SW 59 | 4.19 | 0.229 | 4.28 | 0.050 | 0.978 |
| SW 20 | 2.74 | 0.233 | 2.83 | 0.069 | 0.937 | SW 40 | 3.49 | 0.242 | 3.56 | 0.066 |
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Unraveling the developmental dynamic of visual exploration of social interactions in autism
eLife 13:e85623.
https://doi.org/10.7554/eLife.85623
