BMJ  2008;336:1277 (7 June), doi:10.1136/bmj.a144

Head to Head

Should we use large scale healthcare interventions without clear evidence that benefits outweigh costs and harms? No

¹ University of California San Francisco, 3333 California Street, San Francisco, CA 94118, USA , ² San Francisco VA Medical Center, San Francisco, ³ Ottawa Health Research Institute, Ottawa, Canada

Analysis, doi: 10.1136/bmj.39569.510521.AD

Obtaining definitive evidence on the effects of large scale interventions can be difficult. Bernard Crump (doi: 10.1136/bmj.a145) believes that implementation with careful monitoring is justified but Seth Landefeld and colleagues argue that acting without proof is both costly and potentially damaging to health

Large scale healthcare interventions are likely to improve the health of the public if the evidence clearly shows that the benefits outweigh harms and costs. Often, however, the evidence is not compelling, and well intended interventions may fail to improve health, or may even cause harm, while costing dearly. Moreover, when a large scale intervention is implemented without compelling evidence, wishful thinking may replace careful evaluation, and an unproved innovation may become an enduring but possibly harmful standard of care. Such interventions should be implemented, therefore, only when the evidence shows that expected benefits outweigh harms and costs and only when the effects of implementation will be evaluated systematically.¹

Large scale healthcare interventions aim to influence clinical evaluation, treatment, or care of a large group of people. Some interventions are coercive, such as the restrictions on working hours for resident physicians enforced by the US Accreditation Council for Graduate Medical Education. Other interventions, such as pay for performance, use explicit incentives, whereas other initiatives (such as the US Institute for Healthcare Improvement’s 5 Million Lives Campaign to reduce preventable adverse events) are voluntary.

Evaluating evidence

Two questions underlie this debate. Firstly, how can we assess evidence to build knowledge about improvement? The assessment of evidence about a healthcare intervention is, in principle, straightforward, and the methods are well developed.^{2 3 4} In practice, however, challenging issues arise. Can knowledge of the effect of a large scale intervention be built from observational studies as well as experimental trials? Of course: many study designs can provide robust evidence.⁵ However, the debacle over hormone replacement therapy reminds us that reliance on even well done observational studies can mislead both policy makers and clinicians.⁶ Is the evaluation of costs and harms sufficient? Unexpected harms of interventions are understudied, possibly common, and often difficult to discern.^{7 8} How should information about the efficacy of an intervention be put into practice, and how should practice be revised as new evidence emerges? Carefully, because the effects of an intervention may vary among patients, providers, and medical care environments, which often differ from those in studies that established efficacy.^{9 10} For example, policies to promote potentially effective screening for breast or colorectal cancer have proved harmful when applied to patients unlikely to benefit.^{11 12} New evidence often emerges after an intervention has been put into practice, sometimes contradicting earlier judgments about its efficacy, as shown by the growing literature about rapid response teams.¹³ Questions about evidence and knowledge are fundamental to decisions about whether to implement a large scale intervention, and they merit thoughtful consideration by independent judges.

Assessing balance

Secondly, how should we weigh the benefits, costs, and harms of a large scale intervention? Decisions about whether to implement an intervention should be informed by optimistic and pessimistic estimates of its net benefit—that is, best case and worst case scenarios.

If the best case is that an intervention will not have net benefit, the intervention is a non-starter. For example, a policy to reduce falls in hospital by restraining all older patients is a non-starter because it would cause harms that outweigh benefits, even in the best case.¹⁴

If the worst case is that an intervention will have net benefit, and its costs are acceptable, then it is a "no brainer" to implement. For example, washing hands before and after examining each patient will reduce nosocomial infection at low cost, even in the worst case.¹⁵ (Of course, a large scale intervention to put a clearly beneficial behaviour into universal practice is not simple and merits due consideration of the available evidence.)

If the best case is positive but the worst case is not, then more data are needed. For example, the effects of rules restricting residents’ working hours were unknown when they were implemented.¹⁶ In the best case, the restrictions might benefit patients by reducing medical errors related to fatigue. In the worst case, they might harm patients by decreasing access to care, disrupting continuity of care, and increasing errors related to miscommunication, as early evidence suggested.¹⁷ Also, working restrictions are costly. It remains unclear whether rules on working hours can be implemented in such a way that they have net benefit.¹⁸ If the net effects of an intervention are uncertain, large scale implementation of the intervention is dubious, especially without a rigorous plan to determine its effects.

Our patients will predictably benefit from large scale interventions only when the benefits outweigh harms and costs. We can promote such interventions in three ways. Firstly, we must commit ourselves to the goal that we will implement large scale interventions only when it is clear that the benefits outweigh costs and harms. Secondly, we must build the knowledge and methods to achieve this goal. Finally, we must raise the priority for this work so it is supported. The costs and harms of delay—whether by doing nothing or by implementing large scale interventions of unknown benefit—are too high.

Analysis, doi: 10.1136/bmj.39569.510521.AD

---

Competing interests: None declared.

References

1. Auerbach AD, Landefeld CS, Shojania KG. The tension between needing to improve care and knowing how to do it. N Engl J Med 2007;357:608-13.[Free Full Text]
2. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328:1490.[Abstract/Free Full Text]
3. Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions 4.2.6. Chichester: John Wiley, 2006.
4. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005;10(suppl 1):21-34.[Abstract/Free Full Text]
5. Shojania KG, Grimshaw JM. Evidence-based quality improvement: the state of the science. Health Aff (Millwood) 2005;24:138-50.[Abstract/Free Full Text]
6. Barrett-Connor E, Grady D, Stefanick ML. The rise and fall of menopausal hormone therapy. Annu Rev Public Health 2005;26:115-40.[CrossRef][ISI][Medline]
7. Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet 2004;363:1341-5.[CrossRef][ISI][Medline]
8. Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005;116:1506-12.[Abstract/Free Full Text]
9. Dhruva SS, Redberg RF. Variations between clinical trial participants and medicare beneficiaries in evidence used for medicare national coverage decisions. Arch Intern Med 2008;168:136-40.[Abstract/Free Full Text]
10. Weiss NS, Koepsell TD, Psaty BM. Generalizability of the results of randomized trials. Arch Intern Med 2008;168:133-5.[Free Full Text]
11. Walter LC, Davidowitz NP, Heineken PA, Covinsky KE. Pitfalls of converting practice guidelines into quality measures: lessons learned from a VA performance measure. JAMA 2004;291:2466-70.[Abstract/Free Full Text]
12. Walter LC, Eng C, Covinsky KE. Screening mammography for frail older women: what are the burdens? J Gen Intern Med 2001;16:779-84.[CrossRef][ISI][Medline]
13. McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M. Outreach and early warning systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev 2007(3):CD005529.
14. Kannus P, Sievanen H, Palvanen M, Jarvinen T, Parkkari J. Prevention of falls and consequent injuries in elderly people. Lancet 2005;366:1885-93.[CrossRef][ISI][Medline]
15. Boyce JM, Pittet D. Guideline for hand hygiene in health-care settings. Infect Control Hosp Epidemiol 2002;23(12 suppl):S3-40.[CrossRef][ISI][Medline]
16. Steinbrook R. The debate over residents’ work hours. N Engl J Med 2002;347:1296-302.[Free Full Text]
17. Laine C, Goldman L, Soukup JR, Hayes JG. The impact of a regulation restricting medical house staff working hours on the quality of patient care. JAMA 1993;269:374-8.[Abstract]
18. Okie S. An elusive balance—residents’ work hours and the continuity of care. N Engl J Med 2007;356:2665-7.[Free Full Text]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Complexity theory

Tony Delamothe
BMJ 2008 336: 0. [Extract] [Full Text]

Should we use large scale healthcare interventions without clear evidence that benefits outweigh costs and harms? Yes

Bernard Crump
BMJ 2008 336: 1276. [Full Text] [PDF]

Complex interventions or complex systems? Implications for health economic evaluation

Alan Shiell, Penelope Hawe, and Lisa Gold
BMJ 2008 336: 1281-1283. [Extract] [Full Text] [PDF]

Grading quality of evidence and strength of recommendations
BMJ 2004 328: 1490. [Abstract] [Full Text] [PDF]