Background: Comparing outcomes across hospitals to learn from best performing hospitals can be valuable. However, reliably identifying best performance is challenging. This study assesses the possibility to distinguish best performing hospitals on single outcomes and consistency of performance on different outcomes. Methods: Data were derived from the Dutch ColoRectal Audit 2013–2015. Outcomes considered were textbook outcome (colon), (circumferential) resection margins, (serious) complications, mortality, and ‘failure to rescue’. To include uncertainty in rankings, random effect logistic regression models were used to calculate expected ranks (ERs), for each hospital and outcome. Rankability was calculated for each outcome, as a measure of reliability of ranking. Furthermore, correlation between ERs on different outcomes was assessed. Correlation was considered weak <0.40, moderate between 0.40 - 0.59 and strong >0.60. Results: The study included 32 143 patients; of whom 11 373 were treated in 2015 across 84 hospitals, 8181 colon and 3192 rectal cancer patients. In this one-year period ‘Postoperative complications’ had the highest rankability for colon (57%) and rectal (41%) surgery. No (group of) hospital(s) had the highest ER(s) on all outcomes. Correlation between ERs of outcomes was moderate in 2 (of 25) and strong in 4 (of 25) combinations. Rankability of colorectal mortality increased from 14% in 2015 to 35% when data over 2013–2015 were used. Conclusion: The highest reliability of identifying best performance based on an outcome was 57%. However, the balance between reliability and relevance of outcomes is vulnerable. No (group of) hospital(s) could be identified as best performer on all outcomes. Performance was not consistent on outcomes.@en

Background: Report cards on the health care system increasingly report provider-specific performance on indicators that measure the quality of health care delivered. A natural reaction to the publishing of hospital-specific performance on a given indicator is to create 'league tables' that rank hospitals according to their performance. However, many indicators have been shown to have low to moderate rankability, meaning that they cannot be used to accurately rank hospitals. Our objective was to define conditions for improving the ability to rank hospitals by combining several binary indicators with low to moderate rankability. Methods: Monte Carlo simulations to examine the rankability of composite ordinal indicators created by pooling three binary indicators with low to moderate rankability. We considered scenarios in which the prevalences of the three binary indicators were 0.05, 0.10, and 0.25 and the within-hospital correlation between these indicators varied between - 0.25 and 0.90. Results: Creation of an ordinal indicator with high rankability was possible when the three component binary indicators were strongly correlated with one another (the within-hospital correlation in indicators was at least 0.5). When the binary indicators were independent or weakly correlated with one another (the within-hospital correlation in indicators was less than 0.5), the rankability of the composite ordinal indicator was often less than at least one of its binary components. The rankability of the composite indicator was most affected by the rankability of the most prevalent indicator and the magnitude of the within-hospital correlation between the indicators. Conclusions: Pooling highly-correlated binary indicators can result in a composite ordinal indicator with high rankability. Otherwise, the composite ordinal indicator may have lower rankability than some of its constituent components. It is recommended that binary indicators be combined to increase rankability only if they represent the same concept of quality of care.@en

Background Despite widespread use of quality indicators, it remains unclear to what extent they can reliably distinguish hospitals on true differences in performance. Rankability measures what part of variation in performance reflects 'true' hospital differences in outcomes versus random noise. Objective This study sought to assess whether combining data into composites or including data from multiple years improves the reliability of ranking quality indicators for hospital care. Methods Using the Dutch National Medical Registration (2007-2012) for stroke, colorectal carcinoma, heart failure, acute myocardial infarction and total hiparthroplasty (THA)/ total knee arthroplasty (TKA) in osteoarthritis (OA), we calculated the rankability for in-hospital mortality, 30-day acute readmission and prolonged length of stay (LOS) for single years and 3-year periods and for a dichotomous and ordinal composite measure in which mortality, readmission and prolonged LOS were combined. Rankability, defined as (betweenhospital variation/between-hospital+within hospital variation)×100% is classified as low (<50%), moderate (50%-75%) and high (>75%). Results Admissions from 555 053 patients treated in 95 hospitals were included. The rankability for mortality was generally low or moderate, varying from less than 1% for patients with OA undergoing THA/TKA in 2011 to 71% for stroke in 2010. Rankability for acute readmission was low, except for acute myocardial infarction in 2009 (51%) and 2012 (62%). Rankability for prolonged LOS was at least moderate. Combining multiple years improved rankability but still remained low in eight cases for both mortality and acute readmission. Combining the individual indicators into the dichotomous composite, all diagnoses had at least moderate rankability (range: 51%-96%). For the ordinal composite, only heart failure had low rankability (46% in 2008) (range: 46%-95%). Conclusion Combining multiple years or into multiple indicators results in more reliable ranking of hospitals, particularly compared with mortality and acute readmission in single years, thereby improving the ability to detect true hospital differences. The composite measures provide more information and more reliable rankings than combining multiple years of individual indicators.@en