Advertisement

Meta-analysis: Methods, strengths, weaknesses, and political uses

      The general methodology, strengths and weaknesses, and political uses of meta-analysis are examined. As a systematic study of all studies that have been conducted to answer a specific question or hypothesis, meta-analysis is strong in revealing structural flaws and sources of bias in primary research and in posing promising research questions for future study. It cannot exceed, however, the limits of what is reported by primary researchers. Meta-analysis is particularly challenged to quantify the size of a common effect of treatment across reported trials because of (1) the clinical diversity of the trials and (2) the myriad of potential differences among patients with varying characteristics within the trials. Without access to the original data of reported trials, meta-analysis cannot overcome the bias of underpowered trials toward overstatement of the size of main treatment effects, nor the tendency for such trials to falsely conclude there were no statistically significant adverse events. Although severely compromised by ghost-written or honorary-authored reports of primary research, meta-analysis can make use of its methods to focus on the conflicts of interest and likely sources of bias of such research and make known what precautions should be taken by would-be consumers. Examples show how meta-analysis has clarified thinking about the off-label use of selective serotonin reuptake inhibitors for treating child and adolescent depression, use of low-tidal volume respirator assistance for acute lung injury and acute respiratory distress syndrome patients, and the long-term use of COX-2 inhibitors for relieving arthritic pain. Recommendations are made for Congressional action.

      Abbreviations:

      ALI (acute lung injury), ANOVA (analysis of variance), ARDS (acute respiratory distress syndrome), FDA (U.S. Food and Drug Administration), IQ (intelligence quotient), NDA (new drug application), NSAID (nonsteroidal anti-inflammatory drug), RCT (randomized controlled trial), SSRI (selective serotonin reuptake inhibitor)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic and Personal
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Translational Research
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Tippett L.H.C.
        The methods of statistics. Willams and Norgate, London1931
        • Fisher R.A.
        Statistical methods for research workers. 4th ed. Oliver and Boyd, London1932
        • Pearson K.
        On a method of determining whether a sample of size n supposed to have been drawn from a parent population having a known probability integral has probably been drawn at random.
        Biometrika. 1933; 25: 379-410
        • Cochran W.G.
        Problems arising in the analysis of a series of similar experiments.
        J Royal Stat Soc. 1937; 100: 102-118
        • Wolf F.M.
        Meta-analysis. Sage, Beverley Hills, Calif1986 (13)
        • Hunt M.
        How science takes stock. Russell Sage Foundation, New York1997
        • Chalmers T.C.
        • Matta R.J.
        • Smith H.
        • Kunzler A.M.
        Evidence favoring the use of anticoagulants in the hospital phase of acute myocardial infarction.
        N Engl J Med. 1977; 297: 1091-1096
      1. Available at: http://www.cochrane.org/index0.htm. Accessed February 25, 2005.

      2. Available at: http://www.campbellcollaboration.org/guidelines.htm. Accessed February 25, 2005.

        • Cooper H.
        Scientific guidelines for conducting integrative research reviews.
        Rev Educ Res. 1982; 52: 291-302
        • Jadad A.R.
        • Cook D.J.
        • Jones A.
        • Klassen T.P.
        • Tugwell P.
        • Moher M.
        • et al.
        Methodology and reports of systematic reviews and meta-analyses.
        JAMA. 1998; 280: 278-280
        • Moncrieff J.
        Research synthesis.
        Int Rev Psychiatr. 1998; 10: 304-311
        • Robey R.R.
        • Dalebout S.D.
        A tutorial on conducting meta-analyses of clinical outcome research.
        JSLHR. 1998; 41: 1227-1241
        • Rosenthal R.
        Writing meta-analytic reviews.
        Psychol Bull. 1995; 118: 183-192
        • Rothstein H.R.
        • McDaniel M.A.
        Guideline for conducting and reporting meta-analyses.
        Psychol Rep. 1989; 65: 759-770
        • Duriak J.A.
        • Lipsey M.W.
        A practitioner’s guide to meta-analysis.
        Am J Commun Psychol. 1991; 19: 291-352
      3. Furkawa T. Meta-analyses and megatrials: neither is the infallible, universal standard. EBMH Notebook. Available at: www.ebmentalhealth.com. Accessed February 25, 2005.

        • Howard G.S.
        • Maxwell S.E.
        • Fleming K.J.
        The proof of the pudding.
        Psychol Meth. 2000; 5: 315-332
        • Moynihan R.
        Evaluating health services. Milbank Memorial Fund, New York2004
        • Egger M.
        • Smith G.D.
        Meta-analysis.
        BMJ. 1997; 315: 1371-1374
        • Smith G.D.
        • Egger M.
        Meta-analysis.
        BMJ. 1998; 316: 221-225
        • Egger M.
        • Stern J.A.C.
        • Smith G.D.
        Meta-analysis software.
        BMJ. 1998; 316: 221-225
      4. Available at: http://bmj.bmjjournals.com/cgi/collection/systematic_reviews%3Astatistics_descriptions. Accessed February 25, 2005.

        • Furberg C.D.
        • Psaty B.M.
        • Fitzgerald G.A.
        Parecoxib, valdecoxib, and cardiovascular risk.
        Circ J Am Heart Assoc. 2005; 111: 249
        • Cooper H.
        The integrative research review. Sage, Beverley Hills, Calif1997 (15)
        • Dickersin K.
        • Scherer R.
        • LeFebvre C.
        Identifying relevant studies for systematic reviews.
        BMJ. 1994; 309: 1286-1291
        • Thompson S.G.
        • Higgins J.P.T.
        Treating individuals 4.
        Lancet. 2005; 365: 341-346
        • Rosenthal R.
        Combining results of independent studies.
        Psychol Bull. 1978; 85: 185-193
        • Cohen J.
        Statistical power analysis for the behavioral sciences. 2nd ed. Erlbaum, Hillsdale, NJ1988
      5. Bandolier. Describing results of trials and reviews. Available at: http://www.jr2.ox.ac.uk/bandolier/booth/glossary/outputs.html. Accessed July 29, 2005.

        • Rosenthal R.
        Meta-analytic procedures for social research. Sage, Newbury Park, Calif.1991 (34–5)
        • Kraemer H.C.
        • Wilson T.
        • Fairburn C.G.
        • Agras W.S.
        Mediators and moderators of treatment effects in randomized clinical trials.
        Arch Gen Psychiatry. 2002; 59: 877-883
        • Webb E.
        • Campbell D.
        • Schwartz R.
        • Sechrest L.
        • Grove J.
        Nonreactive measures in the social sciences. Houghton Mifflin, Boston, Mass1981
        • Yu J.
        • Cooper H.
        A quantitative review of research design effects on response rates to questionnaires.
        J Marketing Res. 1983; 20: 36-44
        • Ambady N.
        • Rosenthal R.
        Thin slices of expressive behavior as predictors of interpersonal consequences.
        Psychol Bull. 1992; 111: 256-274
        • Rosenthal R.
        • Rubin D.B.
        Interpersonal expectancy effects.
        Behav Brain Stud. 1978; 3: 363-374
        • Lipsey M.
        What do we learn from 400 research studies on the effectiveness of treatment with juvenile delinquents?.
        in: McGuire J. What works reducing re-offending. Wiley, New York1995: 63-111
        • Lipsey M.W.
        • Wilson D.B.
        The efficacy of psychological, educational, and behavioral treatment.
        Am Psychol. 1993; 48: 1181-1209
        • Halvorsen K.T.
        • Burdick E.
        • Colditz G.A.
        • Frazier H.S.
        • Mosteller F.
        Combining results from independent investigations.
        in: Bailar J.C. Mosteller F. Medical uses of statistics. 2nd ed. NEJM Books, Boston, Mass1992: 413-426
        • Chalmers T.C.
        • Lau J.
        Meta-analytic stimulus for changes in clinical trials.
        Stat Methods Med Res. 1993; 2: 162-172
        • Egger M.
        • Smith D.
        • Schneider M.
        • Minder C.
        Bias in meta-analysis detected by a simple, graphical test.
        BMJ. 1997; 315: 629-634
        • Clarke M.
        • Chalmers I.
        Discussion sections in reports of controlled trials published in general medical journals.
        JAMA. 1998; 280: 280-282
        • Petrucci N.
        • Iacovelli W.
        Ventilation with lower tidal volumes versus traditional volumes in adults for acute lung injury and acute respiratory distress syndrome.
        The Cochrane Database Systematic Reviews. 2004; 2 (:CD003844.pub2.DOI:10.1002/14651858.CD003844.pub2.)
        • Geddes J.R.
        • Freemantle N.
        • Mason J.
        • Eccles M.P.
        • Boynton J.
        Selective serotonin reuptake inhibitors (SSRIs) versus other antidepressants for depression.
        The Cochran Database of Systematic Reviews. 2005; 4 (:CD001851.DOI:10.1002/14651858.CD001851.)
        • Garner S.
        • Fidan D.
        • Frankish R.
        • Judd M.
        • Shea B.
        • Towheed T.
        • et al.
        Celecoxib for rheumatoid arthritis.
        The Cochran Database of Systematic Reviews. 2002; 4 (:CD003831.DOI:10.1002/14651858.CD003831.)
        • Wienecke T.
        • Gotzsche P.C.
        Paracetamol versus nonsteroidal anti-inflammatory drugs for rheumatoid arthritis.
        The Cochran Database of Systematic Reviews. 2004; 1 (:CD003789.pub2.DOI:10.1002/14651858.CD003789.pub2.)
        • Garner S.
        • Fidan D.
        • Frankish R.
        • Judd M.
        • Towheed T.
        • Wells G.
        • et al.
        Rofecoxib for rheumatoid arthritis.
        The Cochran Database of Systematic Reviews. 2005; 1 (:CD003685.pub2.DOI:10.1002/14651858.CD003685.pub2.)
        • Greenberg R.P.
        • Bornstein R.F.
        • Zborowski M.J.
        • Fisher S.
        • Greenberg M.D.
        A meta-analysis of fluoxetine outcome in the treatment of depression.
        J Nerv Ment Dis. 1994; 182: 547-551
        • Eichacker P.Q.
        • Banks S.M.
        • Cui X.
        • Natanson C.
        Meta-analysis of ALI and ARDS trials testing low tidal volumes.
        Am J Respir Crit Care Med. 2002; 166: 1510-1514
        • Li H.
        The resolution of some paradoxes relating to reliability and validity.
        J Educ Behav Stat. 2003; 28: 89-95
        • Cooper H.
        • Hedges L.V.
        Potentials and limitations of research synthesis.
        in: Cooper H. Hedges L.V. The handbook of research synthesis. Russell Sage, New York1994
        • Campbell D.
        • Stanley J.
        Experimental and quasi-experimental designs for research. Rand McNally, Chicago, Ill1963
        • Noble J.H.
        Peer review.
        Science. 1974; 185: 916-921
      6. Thornton S. Karl Popper. In: Zalta EN, ed. The Stanford encyclopedia of philosophy. (2002). Available at: http://plato.stanford.edu/archives/win2002/entries/popper/. Accessed February 25, 2005.

        • Acute Respiratory Distress Syndrome Network
        Ventilation with lower tidal volumes compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome.
        N Engl J Med. 2000; 342: 1301-1308
      7. Noble JH. Critique of the ARDS Network experimental design, implementation, and reporting. Testimony presented to the Office of Human Research Protection (OHRP) ARDS Network Trials Expert Review Panel, June 10, 2003. Washington, D.C. Available at: http://www.ahrp.org/testimonypresentations/ARDSNet0603/noble.html. Accessed March 14, 2005.

        • Deans K.J.
        • Minneci P.C.
        • Cui X.
        • Banks S.M.
        • Natanson C.
        • Eichacker P.Q.
        • et al.
        Editorial—Mechanical ventilation in ARDS.
        Crit Care Med. 2005; 33: 1141-1143
        • Solomon S.D.
        • McMurray J.J.V.
        • Pfeffer M.A.
        • Wittes J.
        • Fowler R.
        • Finn P.
        • et al.
        Cardiovascular risk associated with celecoxib in a clinical trial for colorectal adenoma prevention.
        N Engl J Med. 2005; 352: 1071-1080
        • Bresalier R.S.K.
        • Sandler R.S.
        • Quan H.
        • Bolognese J.A.
        • Oxenius B.
        • Horgan K.
        • et al.
        Cardiovascular events associated with rofecoxib in a colorectal adenoma chemoprevention trial.
        N Engl J Med. 2005; 352: 1092-1102
        • Nussmeier N.A.
        • Whelton A.A.
        • Brown M.T.
        • Langford R.M.
        • Hoeft A.
        • Parlow J.L.
        • et al.
        Complications of the COX-2 inhibitors parecoxib and valdecoxib after cardiac surgery.
        N Engl J Med. 2005; 352: 1081-1091
        • Psaty B.M.
        • Furberg C.D.
        COX-2 inhibitors—lessons in drug safety.
        N Engl J Med. 2005; 352: 1133-1135
        • Drazen J.M.
        COX-2 inhibitors—a lesson in unexpected problems.
        N Engl J Med. 2005; 352: 1131-1132
        • Kaufman M.
        FDA panel mulls whether all COX-2 drugs have same risk.
        Washington Post 2005. 2005; (;(Feb. 17):News A03.)
        • Kaufman M.
        Vioxx alternative potentially as risky, official says.
        Washington Post 2005. 2005; (;(Feb 18):News A12.)
        • Halpern S.D.
        • Karlawish J.H.T.
        • Berlin J.A.
        The continuing unethical conduct of underpowered clinical trials.
        JAMA. 2002; 288: 358-362
      8. Erdfelder E, Faul F, Buchner A. G*Power: a general power analysis program. Behav Res Meth Ins C 1996;28:1–11. Available at: http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/how_to_use_gpower.html. Accessed February 25, 2005.

      9. Willis JL. Equality in clinical trials: drugs and gender. Rockville, Md.: U.S. Food and Drug Administration, Office of Special Health Issues; 1997. Available at: http://www.fda/gov/oashi/aids/equal.html. Accessed February 25, 2005.

        • Ioannidis J.P.A.
        Contradicted and initially stronger effects in highly cited clinical research.
        JAMA. 2005; 294: 218-228
        • Sugarman J.
        Editorial—Lying, cheating and stealing in clinical research.
        Clin Trials. 2004; 1: 475-476
      10. Martinson BC, Anderson MS, deVries R. Scientists behaving badly. Nature 2005;435:737–8. Available at: http://www.nature.com/nature/journal/v435/n7043/full/435737a.html. Accessed June 30, 2005.

        • Lemmens T.
        Leopards in the temple.
        Int Comp Health Law Ethics. 2004; (;(Winter):): 641-657
      11. Wolfe SM. Letter to FDA revealing heart dangers in an unpublished clinical trial of Celebrex. Washington, D.C.: Public Citizen Health Research Group; January 31, 2005. Available at: http://www.citizen.org/publications/print_release.cfm?ID=7359. Accessed February 25, 2005.

      12. Barton J, Greenwood JC. Letter to FDA seeking information on antidepressants. Washington, D.C.: U.S. House of Representatives, Committee on Energy and Commerce, Subcommittee on Oversight and Investigations; March 24, 2005. Available at: http://energycommerce.house.gov/108/letters/03242004_1242.htm. Accessed September 30, 2004.

        • Mosholder A.D.
        Written statement summarizing Dr. Andrew Mosholder’s role in the FDA review of pediatric use of antidepressant drugs, including interview by the FDA Office of Internal Affairs regarding disclosure of his findings in the February 1, 2005 edition of the San Francisco Chronicle. U.S. House of Representatives, House Committee on Energy and Commerce, Subcommittee on Oversight and Investigations Hearing, Washington, D.C.2004 (; September 23,)
      13. AG New York v. GlaxoSmithKline, Sec. 38, June 2, 2004.

        • McCabe A.R.
        A precarious balancing act—the role of the FDA as protector of public health and industry wealth.
        Suffolk U Law Rev. 2003; 36: 787-819
        • Garland E.J.
        Facing the evidence.
        Can Med Assoc J. 2004; 170: 489-492
        • Whittington C.J.
        • Kendall T.
        • Fonagy P.
        • Cottrell D.
        • Cotgrove A.
        • Boddington E.
        Selective serotonin reuptake inhibitors in childhood depression.
        Lancet. 2004; 363: 1341-1345
      14. Editorial. Depressing research.
        Lancet. 2004; 363: 1335
        • Wohlfarth T.
        • Lekkerkerker F.
        • van Zwieten B.
        Correspondence.
        Lancet. 2004; 364: 659
        • Fergusson D.
        • Doucette S.
        • Glass K.C.
        • Shapiro S.
        • Healy D.
        • Hebert P.
        • et al.
        Association between suicide attempts and selective serotonin reuptake inhibitors.
        BMJ. 2005; 330: 396
      15. BMJ rapid responses to: Fergusson, Doucette, Glass, Shapiro, Healy, Hebert, et al. BMJ 2005;330:396. Available at: http://bmj.bmjjournals.com/cgi/eletters/330/7488/396. Accessed March 13, 2005.

        • Lau J.
        • Schmid C.H.
        • Chalmers T.C.
        Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care.
        J Clin Epidemiol. 1995; 48: 45-57
        • Healy D.
        • Cattell D.
        Interface between authorship, industry and science in the domain of therapeutics.
        BMJ. 2003; 183: 22-27
        • Rennie D.
        • Flanagin A.
        Authorship! Authorship! Guests, ghosts, grafters, and the two-sided coin.
        JAMA. 1994; 274: 469-471
        • Flanagin A.
        • Carey L.A.
        • Fontanarosa P.B.
        • Phillips S.G.
        • Pace B.P.
        • Lundberg G.D.
        • et al.
        Prevalence of articles with honorary authors and ghost authors in peer-reviewed medical journals.
        JAMA. 1998; 280: 222-224
        • Lexchin J.
        • Bero L.A.
        • Djulbegovic B.
        • Clark O.
        Pharmaceutical industry sponsorship and research outcome and quality.
        BMJ. 2003; 326: 1167-1170
        • Melander H.
        • Ahlqvist-Rastad J.
        • Meijer G.
        • Beermann B.
        Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry.
        BMJ. 2003; 326: 1171-1173
        • Angell M.
        The truth about the pharmaceutical industry. Random House, New York2004
        • Herdelin A.C.
        • Scott D.L.
        Experimental studies of the program of assertive community treatment (PACT).
        J Disabil Policy Stud. 1999; 10: 53-89
        • Schulman K.A.
        • Seils D.M.
        • Timbie J.W.
        • Sugarman J.
        • Dame L.A.
        • Weinfurt K.P.
        • et al.
        A national survey of provisions in clinical trial agreements between medical schools and sponsors.
        N Engl J Med. 2002; 347: 1335-1341
      16. P.L 102-571, Title I, Sec. 102, 105 Stat. 4491.

      17. Food and Drug Administration Modernization Act of 1997, 21 U.S.C. Sec 301 (1997).

        • Taveggia T.
        Resolving research controversy through empirical cumulation.
        Sociol Method Res. 1974; 2: 395-407
        • Hammerschmidt D.E.
        • Franklin M.
        Secrecy in medical journals.
        Minnesota Medicine. 2005; (;(March):. [commentary]): 34-35
      18. Dobson R, Lenzer J. US regulator suppresses vital data on prescription drugs on sale in Britain. The Independent (UK). June 12, 2005. Available at: http://news.independent.co.uk/uk/health_medical/article225491.ece. Accessed July 29, 2005.

      19. Lenzer J, Pyke N. Was Traci Johnson driven to suicide by anti-depressants? That’s a trade secret, say US officials. The Independent (UK). June 19, 2005. Available at: http://news.independent.co.uk/uk/health_medical/article226432.ece. Accessed July 29, 2005.

        • Graham D.J.
        • Campen D.
        • Hui R.
        • Spence M.
        • Cheetham C.
        • Levy G.
        • et al.
        Risk of acute myocardial infarction and sudden cardiac death in patients treated with cyclo-oxygenase 2 selective and non-selective non-steroidal anti-inflammatory drugs.
        Lancet. 2005; 365: 475-481
        • Kotulak R.
        FDA cut off critic’s access to drug database.
        Chicago Tribune 2005. 2005; (;(Feb 20):Health sidebar 01.)
        • Graham J.
        • James F.
        Flaws in drug agency put consumer at risk.
        Chicago Tribune 2005. 2005; (;(Feb 20):Health sidebar 01.)
      20. House of Commons Health Committee. The influence of the pharmaceutical industry: fourth report of session 2004-2005, volume I. London: United Kingdom Parliament. 2005. Available at: http://www.publications.parliament.uk/pa/cm200405/cmselect/cmhealth/42/42.pdf. Accessed April 7, 2005.

        • Eggerston L.
        Drug approval system questioned in US and Canada.
        CMAJ. 2005; 172: 317-318
      21. CBC Health & Science News. Review of arthritis drugs raises questions about role of regulators. December 7, 2004. Available at: http://www.cbc.ca/story/science/national/2004/12/07/arthritis-drugs04207.html. Accessed July 30, 2005.

        • Faunce T.A.
        • Tomossy G.F.
        The UK House of Commons report on the influence of the pharmaceutical industry.
        Monash Bioethics Rev. 2005; 23: 38-42
      22. Richards T. Conduct of drug trials in poor countries must improve. BMJ 2005;330:1466. Available at: http://bmj.bmjjournals.com/cgi/content/full/330/7506/1466-a. Accessed June 30, 2005.

        • Nundy S.
        • Gulhati C.M.
        A new colonialism?—conducting clinical trials in India.
        N Engl J Med. 2005; 352: 16