Unpacking quality indicators: how much do they reflect differences in the quality of care?
==========================================================================================

* Jill Tinmouth

*   Accreditation
*   Ambulatory Care
*   Attitudes

Just over 50 years ago, Avedis Donabedian published his seminal paper, which sought to define and specify the ‘quality of health care’, articulating the now paradigmatic triad of structure, process and outcome for measuring healthcare quality.1 In recent years, we have seen the rapid expansion of increasingly inexpensive information technology capability and capacity, facilitating the collection and analysis of large healthcare data sets. These technological advances fuel the current proliferation of performance measurement in healthcare.2 Increasingly, in an effort to improve care, many cancer health systems, including those in England,3 the USA4 and Canada,5 6 are publicly reporting performance indicators, generally derived from these large data sets. Not surprisingly, differences in prevention, early detection and/or treatment of cancer are often used to explain the observed differences in performance across jurisdictions.6–9 

Given the considerable effort and resource invested in performance measurement as well as potential adverse consequences if done poorly,10 it is important to get it right. Determining the effectiveness of healthcare performance measurement is challenging,11 particularly at the health system level. Often, performance measurement is implemented uniformly across an entire system, making well-designed controlled analysis less feasible or impossible12 13 and leaving evaluations vulnerable to secular trends.14 At the physician level, audit and feedback studies report variable results: meta-analyses show a modest benefit overall,15–17 but an important proportion of interventions were ineffective or minimally effective with a few studies suggesting a negative effect on performance.16 Likely, this heterogeneity is due to the complexity of the endeavour and its many moving parts, which include the behaviour targeted, the recipients of the feedback, their environment, the use of cointerventions and the components of the audit and feedback intervention itself.18 The latter generally comprises performance indicators, often derived from large healthcare data sets; however, who reports these indicators and how they are reported are also critical components of audit and feedback.15 16 19 

In this issue of the *BMJ Quality & Safety*, Abel *et al* 20 illustrate empirically some of the complexities of measuring performance. They examine the properties of 16 primary care ‘diagnostic activity indicators’ related to cancer (ie, performance indicators for diagnosing cancer) among 7000+ large general practices in England. Performance indicators are used to assess the quality of healthcare delivery, sometimes by examining outcomes such as survival or recovery of function and at other times, measuring processes, that is, the intermediate steps that have been shown or are felt to be important in terms of achieving the desirable outcomes. In this interesting analysis, the authors challenge the assumption that variation in indicators necessarily reflects underlying differences in the quality of cancer care. Using mixed models, they parse the aetiologies of the observed variation in the cancer diagnostic activity indicators across the practices in their cohort. They then assessed the reliability or ‘rankability’ (whether a practice can be meaningfully distinguished from others using a particular indicator) of the indicators.

This study used data from the Cancer Service Public Health Profile,3 which is one of 30+ thematic National Public Health Profiles published in an interactive web format by Public Health England. The Cancer Service Profile reports cancer services-related indicators at the general practice level for practices of at least 1000 patients as well as at the health region level. The Profile is intended to assist health system planners to make decisions about services and to stimulate ‘reflective practice’ among providers. While use of these data in this way is supported by links between some of the indicators and outcomes such as cancer survival,21 22 there have been concerns regarding small sample sizes and the impact of underlying case mix as causes of the observed variation.23 This is an increasingly familiar refrain in the ‘big data’ era—similar concerns have been articulated in other contexts and health systems.2 11 

In their study,20 Abel and colleagues found that an important proportion of the observed variation in indicators across practices is related to factors other than quality of healthcare. Depending on the indicator, chance alone accounted for 7%–85% of the observed differences in practice. They then examined the role of case mix, finding that age and sex differences explained an additional 5%–75% of the observed variation across practices beyond the role of chance. Chance played a larger role for indicators that were identified a priori as outcome indicators than for those considered process indicators. Outcome indicators also tended to be less rankable than process indicators. As the authors point out, the findings for the outcome indicators may have resulted from smaller sample sizes. However, these findings might also be expected on the basis of fundamental differences in the nature of these two types of indicators. As has been noted,1 24 25 clinicians and the health system influence healthcare processes to a greater extent than they do patient outcomes, as the latter are subject to factors other than the quality of medical care (and this may be reflected as chance). Put simply, the hospital that a patient attends has a greater impact on the tests and treatments they receive than on whether they live or die.

Touted by some as more sensitive and less vulnerable to differences in case mix,24 the paper20 by Abel *et al* demonstrates empirically that process measures can also be affected by chance and case mix, although to a lesser extent than outcome measures. This finding contributes to what is already known about the limitations of process measures,26–29 including the need for evidence linking processes to outcomes, challenges introduced when defining eligible patient populations and lack of comprehensiveness. In the case of the Cancer Service Profile used by Abel *et al*, only a few of the process indicators in the Profile have been shown to be linked to outcomes such as survival.20 Variation in process indicators, unrelated to underlying quality of care, can be introduced during measurement if exclusions are applied variably when defining eligible populations.29 Finally, many existing process indicators are limited to measuring a specific step in the care process; as such, they may not comprehensively reflect the full process that is tied to outcomes of interest.28 This issue is illustrated in the study by Abel *et al*, where process indicators are limited to practice screening coverage, rates of endoscopy, and referral, and as such do not reflect all the care steps leading to diagnosis of or survival from cancer.

Abel and colleagues’ work does have some limitations. Although they were able to quantify the proportion of the variation that could be attributed to case mix, they used age and sex only. There are other aspects of case mix, such as comorbidity and disease severity, which were not included and could also be a cause of the observed variation across practices. For instance, clinicians might understandably not pursue a new cancer diagnosis as assiduously in a patient who already has advanced medical conditions than a patient of the same age and gender who has no serious medical problems. In addition, social determinants of health could also contribute to the observed variations in care. Although the analysis did not account for these factors, the underlying message would remain the same if it did: much of the observed variation in Cancer Service Profile performance indicators (both outcomes and processes) is unrelated to the underlying quality of healthcare.

A previous editorial30 on hospital standardised mortality ratios suggested that we should consider appending cautionary notes to performance indicators, just as we publish warnings for patients on medication labels. Inappropriate use or interpretation of performance indicators might be considered less worrisome than some medication side effects; however, as pointed out in the editorial, resources could be directed towards non-existing problems and complacency induced among providers or hospitals that do have problems. The study by Abel *et al* 20 adds to the list of caveats needed and helps us understand the properties of different types of indicators in a nuanced way. In general, process indicators for diagnostic performance can be used to reliably discern between primary care practices while outcome indicators cannot. An important reason for this is that chance plays too great a role in the variation we see in outcome indicators. While chance plays this role more commonly with outcome indicators, it is not always the rule—chance variation can be an issue for any type performance indicator in primary care if the number of cases per practice remains relatively small.

As Abel and colleagues20 suggest, further empirical work may address some of the specific shortcomings of these indicators. No doubt some of what we call chance variation today will likely turn out to involve identifiable characteristics of patients, providers or health systems. But, it will probably also remain the case that, just as Donabedian proposed some 50 years ago, no single type of indicator will capture the entirety of the quality of care. We will always need a balanced set of indicators involving structure, processes and outcomes.

## Footnotes

*   Competing interests None declared.

*   Provenance and peer review Commissioned; internally peer reviewed.

## References

1.   Donabedian A . Evaluating the quality of medical care. Milbank Mem Fund Q 1966;44:166–206.[doi:10.2307/3348969](http://dx.doi.org/10.2307/3348969) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.2307/3348969&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=5338568&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1966ZE74800006&link_type=ISI) 

2.   Smith PC , Mossialos E , Papanicolas I . Performance measurement for health system improvement: experiences, challenges and prospects. Copenhagen 2008.
    
    

3.  Public Health England. Cancer Service Public Health Profile. 2016 [https://fingertips.phe.org.uk/profile/cancerservices](https://fingertips.phe.org.uk/profile/cancerservices) 
    
    

4.   Ryerson AB , Massetti GM . CDC’s Public health surveillance of cancer. Prev Chronic Dis 2017;14:160480.[doi:10.5888/pcd14.160480](http://dx.doi.org/10.5888/pcd14.160480) 
    
    

5.  Cancer Quality Council of Ontario. Cancer system quality index (CSQI). 2016 [http://www.csqi.on.ca/](http://www.csqi.on.ca/) 
    
    

6.   Rahal R , Chadder J , DeCaria K , et al . How different is cancer control across Canada? Comparing performance indicators for prevention, screening, diagnosis, and treatment. Curr Oncol 2017;24:124–8.[doi:10.3747/co.24.3578](http://dx.doi.org/10.3747/co.24.3578) 
    
    

7.   Allemani C , Weir HK , Carreira H , et al . Global surveillance of cancer survival 1995-2009: analysis of individual data for 25,676,887 patients from 279 population-based registries in 67 countries (CONCORD-2). Lancet 2015;385:977–1010.[doi:10.1016/S0140-6736(14)62038-9](http://dx.doi.org/10.1016/S0140-6736(14)62038-9) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(14)62038-9&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25467588&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 

8.   Coleman MP , Forman D , Bryant H , et al . Cancer survival in Australia, Canada, Denmark, Norway, Sweden, and the UK, 1995-2007 (the international cancer benchmarking partnership): an analysis of population-based cancer registry data. Lancet 2011;377:127–38.[doi:10.1016/S0140-6736(10)62231-3](http://dx.doi.org/10.1016/S0140-6736(10)62231-3) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(10)62231-3&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=21183212&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000286292800025&link_type=ISI) 

9.   De Angelis R , Sant M , Coleman MP , et al . Cancer survival in Europe 1999-2007 by country and age: results of EUROCARE-5-a population-based study. Lancet Oncol 2014;15:23–34.[doi:10.1016/S1470-2045(13)70546-1](http://dx.doi.org/10.1016/S1470-2045(13)70546-1) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S1470-2045(13)70546-1&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=24314615&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000329135300029&link_type=ISI) 

10.  Shahian DM , Normand SL , Friedberg MW , et al . Rating the raters: the inconsistent quality of health care performance measurement. Ann Surg 2016;264:36–8.[doi:10.1097/SLA.0000000000001631](http://dx.doi.org/10.1097/SLA.0000000000001631) 
    
    

11.  Shahian DM , Normand SL . What is a performance outlier? BMJ Qual Saf 2015;24:95–9.[doi:10.1136/bmjqs-2015-003934](http://dx.doi.org/10.1136/bmjqs-2015-003934) 
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjc6IjI0LzIvOTUiO3M6NDoiYXRvbSI7czoxNjoiL3FoYy8yNy8xLzQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

12.  Ellegård LM , Dietrichson J , Anell A . Can pay-for-performance to primary care providers stimulate appropriate use of antibiotics? Health Econ 2017:1–16.[doi:10.1002/hec.3535](http://dx.doi.org/10.1002/hec.3535) 
    
    

13.  Craig P , Dieppe P , Macintyre S , et al . Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655.[doi:10.1136/bmj.a1655](http://dx.doi.org/10.1136/bmj.a1655) 
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzMzcvc2VwMjlfMS9hMTY1NSI7czo0OiJhdG9tIjtzOjE2OiIvcWhjLzI3LzEvNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

14.  Chen YF , Hemming K , Stevens AJ , et al . Secular trends and evaluation of complex interventions: the rising tide phenomenon. BMJ Qual Saf 2016;25:303–10.[doi:10.1136/bmjqs-2015-004372](http://dx.doi.org/10.1136/bmjqs-2015-004372) 
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI1LzUvMzAzIjtzOjQ6ImF0b20iO3M6MTY6Ii9xaGMvMjcvMS80LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

15.  Hysong SJ , Best RG , Pugh JA . Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci 2006;1:5–3.[doi:10.1186/1748-5908-1-9](http://dx.doi.org/10.1186/1748-5908-1-9) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1186/1748-5908-1-5&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=16722583&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 

16.  Ivers N , Jamtvedt G , Flottorp S , et al . Audit and feedback: effects on professional practice and healthcare outcomes Cochrane Database Syst Rev. 2012:6.
    
    

17.  Jamtvedt G , Young JM , Kristoffersen DT , et al . Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006:CD000259.[doi:10.1002/14651858.CD000259.pub2](http://dx.doi.org/10.1002/14651858.CD000259.pub2) 
    
    

18.  Ivers NM , Sales A , Colquhoun H , et al . No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci 2014;9:14.[doi:10.1186/1748-5908-9-14](http://dx.doi.org/10.1186/1748-5908-9-14) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1186/1748-5908-9-14&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=24438584&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 

19.  van der Veer SN , de Keizer NF , Ravelli AC , et al . Improving quality of care. A systematic review on how medical registries provide information feedback to health care providers. Int J Med Inform 2010;79:305–23.[doi:10.1016/j.ijmedinf.2010.01.011](http://dx.doi.org/10.1016/j.ijmedinf.2010.01.011) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.ijmedinf.2010.01.011&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20189451&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 

20.  Abel G , Saunders CL , Mendonca SC , et al . Variation and statistical reliability of publicly reported primary care diagnostic activity indicators for cancer: a cross-sectional ecological study of routine data. BMJ Qual Saf 2018;27:21–30.[doi:10.1136/bmjqs-2017-006607](http://dx.doi.org/10.1136/bmjqs-2017-006607) 
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjc6IjI3LzEvMjEiO3M6NDoiYXRvbSI7czoxNjoiL3FoYy8yNy8xLzQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

21.  Møller H , Gildea C , Meechan D , et al . Use of the English urgent referral pathway for suspected cancer and mortality in patients with cancer: cohort study. BMJ 2015;351:h5102.[doi:10.1136/bmj.h5102](http://dx.doi.org/10.1136/bmj.h5102) 
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNTEvb2N0MTNfNC9oNTEwMiI7czo0OiJhdG9tIjtzOjE2OiIvcWhjLzI3LzEvNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

22.  Shawihdi M , Thompson E , Kapoor N , et al . Variation in gastroscopy rate in English general practice and outcome for oesophagogastric cancer: retrospective analysis of Hospital Episode Statistics. Gut 2014;63:250–61.[doi:10.1136/gutjnl-2012-304202](http://dx.doi.org/10.1136/gutjnl-2012-304202) 
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiZ3V0am5sIjtzOjU6InJlc2lkIjtzOjg6IjYzLzIvMjUwIjtzOjQ6ImF0b20iO3M6MTY6Ii9xaGMvMjcvMS80LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

23.  Murchie P , Chowdhury A , Smith S , et al . General practice performance in referral for suspected cancer: influence of number of cases and case-mix on publicly reported data. Br J Cancer 2015;112:1791–8.[doi:10.1038/bjc.2015.110](http://dx.doi.org/10.1038/bjc.2015.110) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1038/bjc.2015.110&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25880009&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 

24.  Mant J , Hicks N . Detecting differences in quality of care: the sensitivity of measures of process and outcome in treating acute myocardial infarction. BMJ 1995;311:793–6.[doi:10.1136/bmj.311.7008.793](http://dx.doi.org/10.1136/bmj.311.7008.793) 
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTEvNzAwOC83OTMiO3M6NDoiYXRvbSI7czoxNjoiL3FoYy8yNy8xLzQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

25.  Green J , Wintfeld N , Sharkey P , et al . The importance of severity of illness in assessing hospital mortality. JAMA 1990;263:241–6.[doi:10.1001/jama.1990.03440020075036](http://dx.doi.org/10.1001/jama.1990.03440020075036) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.1990.03440020075036&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=2403601&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1990CG61700031&link_type=ISI) 

26.  Birkmeyer JD , Dimick JB , Birkmeyer NJ . Measuring the quality of surgical care: structure, process, or outcomes? J Am Coll Surg 2004;198:626–32.[doi:10.1016/j.jamcollsurg.2003.11.017](http://dx.doi.org/10.1016/j.jamcollsurg.2003.11.017) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.jamcollsurg.2003.11.017&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15051016&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000220641800019&link_type=ISI) 

27.  Mant J . Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care 2001;13:475–80.[doi:10.1093/intqhc/13.6.475](http://dx.doi.org/10.1093/intqhc/13.6.475) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1093/intqhc/13.6.475&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11769750&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000172628900008&link_type=ISI) 

28.  Rubin HR , Pronovost P , Diette GB . The advantages and disadvantages of process-based measures of health care quality. Int J Qual Health Care 2001;13:469–74.[doi:10.1093/intqhc/13.6.469](http://dx.doi.org/10.1093/intqhc/13.6.469) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1093/intqhc/13.6.469&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11769749&link_type=MED&atom=%2Fqhc%2F27%2F1%2F4.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000172628900007&link_type=ISI) 

29.  Bruckel J , Liu X , Hohmann SF , et al . The denominator problem: national hospital quality measures for acute myocardial infarction. BMJ Qual Saf 2017;26:189–99.[doi:10.1136/bmjqs-2015-004888](http://dx.doi.org/10.1136/bmjqs-2015-004888) 
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI2LzMvMTg5IjtzOjQ6ImF0b20iO3M6MTY6Ii9xaGMvMjcvMS80LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

30.  Marang-van de Mheen PJ , Shojania KG . Simpson’s paradox: how performance measurement can fail even with perfect risk adjustment. BMJ Qual Saf 2014;23:701–5.[doi:10.1136/bmjqs-2014-003358](http://dx.doi.org/10.1136/bmjqs-2014-003358) 
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjIzLzkvNzAxIjtzOjQ6ImF0b20iO3M6MTY6Ii9xaGMvMjcvMS80LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==)