Meta-Research: Large-scale language analysis of peer review reports
- Department of Research in Biomedicine and Health, University of Split School of Medicine, Croatia
- Department d'Informàtica, University of Valencia, Spain
- Department of Social and Political Sciences, University of Milan, Italy
Figures
Figure 1
Words counts in peer review reports.
Word count (mean and 95% confidence interval; LIWC analysis) of peer review reports in four broad areas of research for double-blind review (top) and single-blind review (bottom), and for female reviewers (left) and male reviewers (right). Reports recommending accept (red) were consistently the shortest, and reports recommending major revisions (green) were consistently the longest. See Supplementary file 1 for summary data and mixed model linear regression coefficients and residuals. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.
Figure 2 with 1 supplement
Analytical tone, clout and authenticity and in peer review reports for single-blind review.
Scores returned by LIWC (mean percentages and 95% confidence interval) for analytical tone (A), clout (B) and authenticity (C) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using single-blind review. Reports recommending accept (red) consistently had the most clout, and reports recommending reject (purple) consistently had the least clout. See Supplementary files 2–4 for summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for analytical tone, clout and authenticity. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.
Figure 2—figure supplement 1
Analytical tone, clout and authenticity in peer review reports for double-blind review.
Scores returned by LIWC (mean percentages and 95% confidence interval) for analytical tone (A), clout (B) and authenticity (C) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using double-blind review.
Figure 3 with 1 supplement
Sentiment analysis of peer review reports for single-blind review.
Scores for sentiment analysis returned by LIWC (A; mean percentage and 95% confidence interval, CI), SentimentR (B; mean score and 95% CI), and Stanford CoreNLP (C; mean score and 95% CI) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using single-blind review. See Supplementary files 5–7 for summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment according to LIWC, SentimentR and Stanford CoreNLP analysis.
Figure 3—figure supplement 1
Sentiment analysis of peer review reports for double-blind review.
Scores for sentiment analysis returned by LIWC (A; mean percentage and 95% confidence interval, CI), SentimentR (B; mean score and 95% CI), and Stanford CoreNLP (C; mean score and 95% CI) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using double-blind review.
Figure 4 with 5 supplements
Moral foundations in peer review reports.
Scores returned by LIWC (mean percentage on a log scale) for general morality in peer review reports in four broad areas of research for double-blind review (top) and single-blind review (bottom), and for female reviewers (left) and male reviewers (right). Reports recommending accept (red) consistently had the highest scores. See Supplementary file 8 for lists of the ten most frequent words found in peer review reports for general morality and the five moral foundation variables. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.
Figure 4—figure supplement 1
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for care/harm, one of the five foundations of Moral Foundations Theory.
Figure 4—figure supplement 2
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for fairness/cheating, one of the five foundations of Moral Foundations Theory.
Figure 4—figure supplement 3
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for loyalty/betrayal, one of the five foundations of Moral Foundations Theory.
Figure 4—figure supplement 4
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for authority/subversion, one of the five foundations of Moral Foundations Theory.
Figure 4—figure supplement 5
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for sanctity/degradation, one of the five foundations of Moral Foundations Theory.
Additional files
-
Supplementary file 1
Word count (Figure 1): summary data and mixed model linear regression coefficients and residuals.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp1-v2.docx
-
Supplementary file 2
Analytical tone (Figure 2A): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC analytical tone.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp2-v2.docx
-
Supplementary file 3
Clout (Figure 2B): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC clout.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp3-v2.docx
-
Supplementary file 4
Authenticity (Figure 2C): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC authenticity.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp4-v2.docx
-
Supplementary file 5
Sentiment/LIWC emotional tone (Figure 3A): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (LIWC emotional tone).
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp5-v2.docx
-
Supplementary file 6
Sentiment/SentimentR score (Figure 3B): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (SentimentR scores).
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp6-v2.docx
-
Supplementary file 7
Sentiment/Stanford CoreNLP score (Figure 3C): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (Stanford CoreNLP score).
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp7-v2.docx
-
Supplementary file 8
Ten most frequent words found in peer review reports for general morality and the five moral foundation variables.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp8-v2.docx
-
Transparent reporting form
- https://cdn.elifesciences.org/articles/53249/elife-53249-transrepform-v2.docx
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Meta-Research: Large-scale language analysis of peer review reports
eLife 9:e53249.
https://doi.org/10.7554/eLife.53249
