Standardized mean differences cause funnel plot distortion in publication bias assessments

  1. Peter-Paul Zwetsloot  Is a corresponding author
  2. Mira Van Der Naald
  3. Emily S Sena
  4. David W Howells
  5. Joanna IntHout
  6. Joris AH De Groot
  7. Steven AJ Chamuleau
  8. Malcolm R MacLeod
  9. Kimberley E Wever  Is a corresponding author
  1. University Medical Center Utrecht, Netherlands
  2. University of Edinburgh, United Kingdom
  3. University of Tasmania, Australia
  4. Radboud Institute for Health Sciences, Radboud University Medical Center, Netherlands
  5. The University of Edinburgh, United Kingdom
  6. Radboud University Medical Center, Netherlands

Abstract

Meta-analyses are increasingly used for synthesis of evidence, and often include an assessment of publication bias based on detection of asymmetry in funnel plots. We studied the influence of different normalisation approaches, sample size and intervention effects on funnel plot asymmetry, using empirical datasets and illustrative simulations. We found that funnel plots of the Standardized Mean Difference (SMD) plotted against the standard error (SE) are susceptible to distortion, leading to overestimation of the existence and extent of publication bias. Distortion was more severe when the primary studies had a small sample size and when an intervention effect was present. We show that using the Normalised Mean Difference (when possible), or plotting the SMD against a sample size-based precision estimate, are more reliable alternatives. We conclude that funnel plots using the SMD in combination with the SE are unsuitable for publication bias assessments and can lead to false-positive results.

Article and author information

Author details

  1. Peter-Paul Zwetsloot

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    For correspondence
    P.P.M.Zwetsloot@umcutrecht.nl
    Competing interests
    The authors declare that no competing interests exist.
  2. Mira Van Der Naald

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  3. Emily S Sena

    Center for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
  4. David W Howells

    School of Medicine, University of Tasmania, Hobart, Australia
    Competing interests
    The authors declare that no competing interests exist.
  5. Joanna IntHout

    Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  6. Joris AH De Groot

    Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  7. Steven AJ Chamuleau

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  8. Malcolm R MacLeod

    Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
  9. Kimberley E Wever

    Systematic Review Centre for Laboratory Animal Experimentation, Radboud University Medical Center, Nijmegen, Netherlands
    For correspondence
    kim.wever@radboudumc.nl
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3635-3660

Funding

National Institute of Environmental Health Sciences (National Toxicology Program research funding)

  • Kimberley E Wever

Netherlands Cardiovascular Research Initiative (CVON-HUSTCARE)

  • Steven AJ Chamuleau

National Centre for the Replacement, Refinement and Reduction of Animals in Research (Infrastructure Award)

  • Emily S Sena
  • Malcolm R MacLeod

Alexander Suerman program (PhD student Scholarship)

  • Peter-Paul Zwetsloot

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Copyright

© 2017, Zwetsloot et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 6,108
    views
  • 450
    downloads
  • 144
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Peter-Paul Zwetsloot
  2. Mira Van Der Naald
  3. Emily S Sena
  4. David W Howells
  5. Joanna IntHout
  6. Joris AH De Groot
  7. Steven AJ Chamuleau
  8. Malcolm R MacLeod
  9. Kimberley E Wever
(2017)
Standardized mean differences cause funnel plot distortion in publication bias assessments
eLife 6:e24260.
https://doi.org/10.7554/eLife.24260

Share this article

https://doi.org/10.7554/eLife.24260

Further reading

    1. Epidemiology and Global Health
    2. Microbiology and Infectious Disease
    Gillian AM Tarr, Linda Chui ... Tim A McAllister
    Research Article

    Several areas of the world suffer a notably high incidence of Shiga toxin-producing Escherichia coli. To assess the impact of persistent cross-species transmission systems on the epidemiology of E. coli O157:H7 in Alberta, Canada, we sequenced and assembled E. coli O157:H7 isolates originating from collocated cattle and human populations, 2007–2015. We constructed a timed phylogeny using BEAST2 using a structured coalescent model. We then extended the tree with human isolates through 2019 to assess the long-term disease impact of locally persistent lineages. During 2007–2015, we estimated that 88.5% of human lineages arose from cattle lineages. We identified 11 persistent lineages local to Alberta, which were associated with 38.0% (95% CI 29.3%, 47.3%) of human isolates. During the later period, six locally persistent lineages continued to be associated with human illness, including 74.7% (95% CI 68.3%, 80.3%) of reported cases in 2018 and 2019. Our study identified multiple locally evolving lineages transmitted between cattle and humans persistently associated with E. coli O157:H7 illnesses for up to 13 y. Locally persistent lineages may be a principal cause of the high incidence of E. coli O157:H7 in locations such as Alberta and provide opportunities for focused control efforts.

    1. Epidemiology and Global Health
    Marina Padilha, Victor Nahuel Keller ... Gilberto Kac
    Research Article

    Background: The role of circulating metabolites on child development is understudied. We investigated associations between children's serum metabolome and early childhood development (ECD).

    Methods: Untargeted metabolomics was performed on serum samples of 5,004 children aged 6-59 months, a subset of participants from the Brazilian National Survey on Child Nutrition (ENANI-2019). ECD was assessed using the Survey of Well-being of Young Children's milestones questionnaire. The graded response model was used to estimate developmental age. Developmental quotient (DQ) was calculated as the developmental age divided by chronological age. Partial least square regression selected metabolites with a variable importance projection ≥ 1. The interaction between significant metabolites and the child's age was tested.

    Results: Twenty-eight top-ranked metabolites were included in linear regression models adjusted for the child's nutritional status, diet quality, and infant age. Cresol sulfate (β = -0.07; adjusted-p < 0.001), hippuric acid (β = -0.06; adjusted-p < 0.001), phenylacetylglutamine (β = -0.06; adjusted-p < 0.001), and trimethylamine-N-oxide (β = -0.05; adjusted-p = 0.002) showed inverse associations with DQ. We observed opposite directions in the association of DQ for creatinine (for children aged -1 SD: β = -0.05; p =0.01; +1 SD: β = 0.05; p =0.02) and methylhistidine (-1 SD: β = - 0.04; p =0.04; +1 SD: β = 0.04; p =0.03).

    Conclusion: Serum biomarkers, including dietary and microbial-derived metabolites involved in the gut-brain axis, may potentially be used to track children at risk for developmental delays.

    Funding: Supported by the Brazilian Ministry of Health and the Brazilian National Research Council.