This paper, titled “The Kaiser Permanente inpatient risk adjustment methodology was valid in an external patient population,” addresses the critical need for accurate prediction of hospital mortality to effectively measure and compare the quality of patient care. It presents a comprehensive external validation study of an established Kaiser Permanente inpatient risk adjustment methodology. The authors emphasize that external validation is not merely beneficial but necessary to truly prove a predictive model’s utility and confirm that its function is not idiosyncratic to the specific patients, physicians, institutions, or data systems where it was originally derived and internally tested.
The study was conducted as a retrospective cohort study at The Ottawa Hospital (TOH), a tertiary-care teaching facility in Ottawa, Canada, which operates within a publicly funded healthcare system and serves as the sole trauma center for its region, also providing most of the region’s oncological care. The study cohort comprised 188,724 inpatients admitted between January 1998 and April 2002. The inclusion criteria mirrored the original derivation study, focusing on adult admissions (excluding patients under 15 years and delivery-related obstetrical admissions), and further excluded patients transferred to or from other hospitals due to data limitations. The unit of analysis throughout the study was the hospitalization.
The Kaiser Permanente inpatient risk adjustment methodology, which this study sought to validate, was originally derived and internally validated on almost 260,000 hospitalizations across 17 hospitals belonging to the Kaiser Permanente Health Plan. This sophisticated model for predicting hospital mortality included six key covariates:
- Patient age (expressed as a squared natural spline with interaction terms).
- Patient sex.
- Admission urgency (elective or emergent).
- Admission service (medical or surgical).
- Admission diagnosis.
- Severity of acute illness, measured by the Laboratory-based Acute Physiology Score (LAPS).
- Chronic comorbidities, quantified by the COmorbidity Point Score (COPS). The original model demonstrated excellent discrimination (c-statistic 0.88) and good calibration (P-value of Hosmer-Lemeshow statistic 0.66).
In this external validation, the researchers replicated the original methodology in the Canadian setting, with two key modifications. Firstly, due to differences in data systems, comorbidity information was only used from previous hospitalizations and chronic diagnoses from the current admission, as outpatient visit data were unavailable. Secondly, an adjustment was made for the different troponin markers used (troponin-T in Ottawa versus troponin-I in the original study) to ensure consistent LAPS calculation.
The study generated four distinct predictive models to assess the methodology:
- Model A: Used the original parameter estimates from the Kaiser Permanente derivation.
- Model B: Used the same variables as Model A but with parameter estimates recalculated from the Ottawa study data.
- Model C: Substituted COPS with the Elixhauser Index for comorbidity, using data-based parameter estimates.
- Model D: Substituted COPS with the Charlson Comorbidity Score for comorbidity, also using data-based parameter estimates. For models B, C, and D, the Ottawa cohort was randomly split into derivation and validation halves to ensure robust assessment. Model discrimination was assessed using the c-statistic, and calibration was evaluated with the Hosmer-Lemeshow statistic and plots of expected vs. observed mortality rates.
The study’s results demonstrated significant findings:
- The overall inpatient mortality rate in the Ottawa cohort was 3.3%.
- A comparison revealed extensive differences between the Ottawa patient population and the original Kaiser Permanente cohort. The Ottawa group was notably younger, had a lower acuity of illness (LAPS), and significantly fewer documented chronic comorbidities (median COPS of 28.5 vs. 74.1). Crucially, over 80% of the Ottawa cohort had no laboratory tests required for LAPS within 24 hours of admission, contrasting with 100% in the original cohort.
- Model A (original parameter estimates) showed excellent discrimination (c-statistic 0.894) but poor calibration (Hosmer-Lemeshow statistic 4009, P < 0.00001), consistently underestimating observed mortality rates. This suggests that direct application of original parameters to a different population, without recalibration, may lead to inaccurate risk predictions.
- Model B (data-based parameter estimates using COPS) exhibited significantly improved discrimination (c-statistic 0.915) and greatly improved calibration (Hosmer-Lemeshow statistic 18.0, P = 0.02), accurately predicting the risk of hospital death.
- Importantly, discrimination and calibration remained excellent when patient comorbidity was expressed using the Elixhauser Index (Model C, c-statistic 0.901) or the Charlson Index (Model D, c-statistic 0.894). Calibration was also maintained for these models, with expected mortality rates falling within the 95% confidence intervals of observed rates for most risk deciles and all risk strata.
In conclusion, the study unequivocally established that the Kaiser Permanente inpatient risk adjustment methodology is a valid model for predicting hospital mortality risk. Its continued accuracy, even when applied to a very different patient population with less detailed comorbidity data capture, highlights its robustness. A particularly significant finding is that the methodology performed equally well regardless of whether patient comorbidity was summarized using the COPS, Elixhauser Index, or Charlson Index. This implies that institutions without access to the specialized software required for COPS calculation can still effectively utilize this methodology by employing standard comorbidity indices. The authors also note potential limitations, such as the requirement for digitized patient-level data and admission diagnoses, and suggest avenues for future research, including incorporating vital signs and functional status to further enhance the model.
Reference: van Walraven, C., Escobar, G. J., Greene, J. D., & Forster, A. J. (2010). The Kaiser Permanente inpatient risk adjustment methodology was valid in an external patient population. Journal of Clinical Epidemiology, 63(7), 798–803.