Comparing Statistical Methods for Cardiac Surgery Report Cards

The study “Impact of Changing the Statistical Methodology on Hospital and Surgeon Ranking: The Case of the New York State Cardiac Surgery Report Card” investigates how different statistical methodologies affect the identification of quality outliers among hospitals and surgeons. Risk adjustment is a central component in generating health outcome report cards, but it is unclear whether it should be based on standard logistic regression, fixed-effects, or random-effects modeling.

Here’s a detailed look at the study’s findings and the implications of using various statistical approaches:

1. Background and Purpose of Health Outcome Report Cards Health outcome report cards are crucial for improving healthcare quality. They aim to promote an efficient market economy by allowing patients, referring physicians, and third-party payers to select providers based on performance. They also serve as a “catalyst to stimulate and promote internal quality improvement”. The New York State (NYS) cardiac surgery report card, specifically for Coronary Artery Bypass Graft (CABG) surgery, is recognized as a methodologically rigorous and highly respected example.

2. The Challenge of Risk Adjustment and Statistical Methodology While there is a consensus on the need for risk adjustment when comparing outcomes across different hospitals and surgeons, various risk-adjustment models can disagree on identifying high and low-performing providers. Critics often question whether existing models adequately capture patient risk, highlighting the subjective nature of model selection. Moreover, the statistical methodology used to create these report cards can be a significant source of bias.

3. Study Objective and Methodology The study’s primary objective was to determine the robustness of the NYS CABG Surgery Report Card to changes in the underlying statistical methodology. Researchers used data from the NYS Cardiac Surgery Reporting System, which included 51,750 patients undergoing isolated CABG surgery in NYS between 1997 and 1999. The outcome variable was in-hospital mortality.

They fitted three main types of models:

• Standard Logistic Regression: This replicated the NYS Department of Health (DOH) models.
• Fixed-Effects Models: These models included a separate identifier for each hospital or surgeon. The study considered fixed-effects models as the “gold standard” because they provide consistent estimates of parameters even when provider effects are correlated with patient risk factors.
• Random-Effects (Random-Intercept) Models: These models included a random quantity for each provider.

Five different approaches were examined for identifying quality outliers, typically based on the ratio of observed-to-expected mortality rates (O/E ratio) and various confidence interval (CI) calculation methods:

1. NYS DOH Approach: Used the O/E ratio with CIs based on the Poisson distribution.
2. O/E Ratio with Bootstrap CI: Employed bootstrapping to calculate CIs, accounting for uncertainty in both observed and expected mortality rates.
3. Fixed-Effects Model with Poisson CI: O/E ratio derived from the fixed-effects model, with Poisson CIs.
4. Random-Intercept Model (Fixed-Effects Component) with Poisson CI: Used only the fixed-effects portion of the random-effects model to calculate the O/E ratio, with Poisson CIs.
5. Random-Intercept Model with Shrinkage Estimators: Utilized the full random-effects model, with exponentiated shrinkage coefficients to determine adjusted odds ratios.

4. Key Findings on Outlier Identification

• Agreement Among Models for Surgeons: At the surgeon level, the standard logistic regression model, fixed-effects model, and the fixed-effects component of the random-effects model demonstrated near-perfect agreement on the identity of quality outliers when using O/E ratios with Poisson distribution CIs. This high level of agreement (ICC = 1 for pairwise comparisons) was attributed to the small impact of surgeons and hospitals on CABG mortality in this study.
• Shrinkage Estimators Identified the Fewest Outliers: Shrinkage estimators, based on random-effects models, were found to be slightly more conservative in identifying quality outliers. This is because these estimators “shrink” provider effects toward the overall mean, especially for providers with low volumes or performance significantly deviating from the mean. While shrinkage estimators have smaller standard errors and are more precise, this precision comes at the cost of introducing bias, potentially misrepresenting high- and low-performing providers.
• Bootstrap Confidence Intervals Identified the Most Outliers: In contrast, the use of bootstrap confidence intervals resulted in the greatest number of providers being identified as quality outliers. This is because bootstrap CIs were narrower than those based on the Poisson distribution. Bootstrapping provides a nonparametric approach that accounts for uncertainty in both observed and estimated mortality rates without assuming a specific statistical distribution.
• Hospital Outliers: For hospitals, similar trends were observed, though the smaller sample size (33 hospitals vs. 138 surgeons) made interpretation more challenging. The fixed-effects model, for instance, identified more outliers than the standard or random-effects models when using Poisson CIs.

5. Limitations of Standard and Random-Intercept Models A critical limitation of both standard logistic regression and random-intercept models is their assumption that patient risk factors are independent of provider effects. This assumption is often unlikely to be correct, potentially leading to inconsistent and poor estimates of provider quality. Fixed-effects models, however, are designed to produce unbiased estimates of provider effects even when such correlations exist.

6. Conclusion and Implications for Report Cards The study highlights that assigning outlier status to providers has potentially profound implications. Therefore, it is essential for analysts producing healthcare report cards to explore the extent to which their findings vary using different statistical approaches before publicly releasing quality reports. The study provides a framework for evaluating the sensitivity of quality “grades” to the choice of statistical methodology. Given the increasing emphasis on “quality” in decisions such as selective referrals and pay-for-performance measures, establishing “best practices” for constructing report cards is crucial to ensure the validity of quality reporting.

APA Reference: Glance, L. G., Dick, A., Osler, T. M., Li, Y., & Mukamel, D. B. (2006). Impact of changing the statistical methodology on hospital and surgeon ranking: The case of the New York State cardiac surgery report card. Medical Care, 44(4), 311–319. http://www.jstor.org/stable/3768323