Reviewing: Patel, B., Chaussalet, T., & Millard, P. (2008). Balancing the NHS balanced scorecard! European Journal of Operational Research, 185(3), 905–914.
https://doi.org/10.1016/j.ejor.2006.02.056
ARTICLE REVIEW | BLOG POST
Why This Article Matters
Healthcare managers worldwide face a familiar paradox: they are asked to improve multiple performance indicators simultaneously, yet the resources they command are finite and the indicators they monitor are rarely independent of one another. A hospital that aggressively reduces emergency readmissions may inadvertently increase delayed discharges. An institution that shortens outpatient waiting times may find its data quality eroding. These are not hypothetical scenarios; they are the empirical findings of a study published in the European Journal of Operational Research nearly two decades ago—a study that deserves renewed attention in an era when balanced scorecards, key performance indicators, and dashboard-driven management have become ubiquitous in health systems across the globe.
Patel, Chaussalet, and Millard (2008) undertook an ambitious task: to reveal the hidden cause-and-effect relationships among 16 performance indicators in the National Health Service (NHS) balanced scorecard for 2001/2002 and 2002/2003. Their approach combined structural equation modelling (SEM) with causal-loop diagrams (CLDs), a technique borrowed from system dynamics, to show that improving one indicator can compromise others. The article is both a technical contribution and a policy warning, and its central message—that performance frameworks can contain internally conflicting indicators—remains profoundly relevant today.
What the Study Does
The NHS adopted a balanced scorecard approach in 2001, classifying hospital performance indicators into three domains: patient focus (e.g., waiting times, cancelled operations, delayed transfers of care), clinical focus (e.g., emergency readmissions, clinical negligence), and capability and capacity focus (e.g., data quality, staff satisfaction, sickness absence rate). Patel et al. (2008) downloaded official performance ratings data from the Department of Health, identified the 16 indicators common to both years, and used SEM to test whether indicators measured in the first year statistically predicted indicators measured in the second year.
The statistically significant paths (p < .05) were then translated into a causal-loop diagram containing 12 reinforcing loops and 5 balancing loops. Reinforcing loops represent snowball effects: a change in one direction feeds back and amplifies itself. Balancing loops represent counterweights: a change triggers an opposing force that tends to restore equilibrium. The most striking finding was the central role of emergency readmissions, which participated in nearly every loop in the network. When the authors conducted scenario testing—asking what would happen if emergency readmissions were improved—they found that delayed transfer of care and data quality were compromised in every scenario, regardless of the assumptions made about link dominance.
What Makes It Valuable
Three contributions stand out. First, the article demonstrates that a balanced scorecard is not automatically balanced; indicators can work against each other, and improving one may degrade another. This insight, while intuitive in retrospect, was rarely articulated with empirical evidence at the time of publication. Second, the study provides a concrete example of how SEM and system dynamics thinking can be combined to inform health policy. The causal-loop diagram is a powerful visual tool that makes complex statistical relationships accessible to managers who may not be comfortable with regression coefficients. Third, the article raises a legitimate and still-unanswered question about the long-term sustainability of performance improvement policies that focus on individual indicators without considering systemic interactions.
Methodological Reflections
Patel et al. (2008) were working at the frontier of an emerging methodology, and their article opened a door that subsequent researchers have continued to walk through. It is in the spirit of building on their contribution—rather than diminishing it—that the following methodological observations are offered. These are not criticisms of the authors’ competence or intent; they are reflections on how the same research question might be addressed today, given the considerable advances in longitudinal modelling that have occurred since 2008.
1. Data-Driven Model Specification via Modification Indices
The authors built their SEM model through an iterative forward–backward process guided by modification indices (MI) produced by AMOS. At each step, they added parameters suggested by the MI and removed non-significant ones until no further improvements could be achieved. This is a well-established exploratory strategy, and the authors were transparent about it. However, the broader SEM literature has long cautioned against heavy reliance on MI for model specification. MacCallum, Roznowski, and Necowitz (1992) conducted a comprehensive simulation study and concluded that data-driven model modifications may be highly inconsistent across samples and susceptible to capitalisation on chance. Their results demonstrated that, over repeated samples, model modifications guided by MI can behave erratically, raising serious questions about the generalisability of the resulting model to other populations or time periods.
More recently, the regularised SEM literature has reinforced this concern. Jacobucci et al. (2016) noted that specification search using MI carries a low probability of converging on the true global solution, particularly when the degree of model uncertainty is high. Given that Patel et al. (2008) began with 16 variables and no a priori structural model, the number of possible paths was very large (16 × 16 = 256 potential cross-lagged paths), and the risk of identifying sample-specific associations was correspondingly elevated. A cross-validation strategy—fitting the model on a random half of the trusts and testing it on the other half—would have helped assess the stability of the results, though the authors may have been constrained by the relatively small number of trusts available.
2. Two-Wave Cross-Lagged Panel Design
The study employs what is technically known as a cross-lagged panel model (CLPM), estimating the effect of indicators at Time 1 on indicators at Time 2. With only two time points, this is the simplest possible longitudinal design for causal inference. While the time-lag assumption is satisfied (causes precede effects by one year), Hamaker, Kuiper, and Grasman (2015) have shown that the traditional CLPM conflates within-unit dynamics with stable between-unit differences. In their influential critique, they demonstrated that if constructs possess trait-like stability—meaning that some hospitals are consistently better or worse on a given indicator, regardless of year—then the autoregressive paths in the CLPM fail to account for this, and the resulting cross-lagged coefficients may not represent genuine within-unit causal processes. They proposed the Random Intercept Cross-Lagged Panel Model (RI-CLPM) as an alternative that cleanly separates within-unit fluctuations from stable between-unit differences.
Lucas (2023) went further, arguing that the traditional CLPM is almost never the right choice, because even the RI-CLPM can produce spurious effects when measurement error or reliable state variance is present. It should be noted, of course, that both Hamaker et al. (2015) and Lucas (2023) were writing primarily about psychological constructs measured with multi-item scales, and the NHS performance indicators are administrative metrics with potentially higher reliability. Nevertheless, the conceptual point holds: hospitals differ systematically in their structural characteristics (size, teaching status, catchment area deprivation), and these stable differences may confound the cross-lagged associations unless explicitly modelled. Patel et al. (2008) could not have implemented the RI-CLPM, which requires a minimum of three waves, and the data available to them constrained the design. This is a limitation of the data environment, not of the researchers’ judgment.
3. Observed Variables and Measurement Error
The model uses observed (manifest) variables exclusively, without latent variable modelling. In SEM, this is known as path analysis. The key consequence is that all variables are assumed to be measured without error. Newsom (2015) has emphasised that an important weakness of observed-variable cross-lagged models is the assumption of error-free measurement; when measurement error is present, relationships tend to be attenuated, and the consequences of such attenuation can be complex, particularly when the reliability of the two variables differs. NHS performance indicators are derived from administrative data systems, and while they may be more reliable than survey-based measures, they are not immune to coding inconsistencies, missing data patterns, and definitional changes across years. The authors’ decision to exclude three indicators with high proportions of missing data was prudent, but the remaining 16 indicators may still contain non-trivial measurement error that could bias the cross-lagged estimates.
4. Absence of Global Model Fit Statistics
Contemporary SEM reporting standards (e.g., Kline, 2023) require the presentation of multiple global fit indices, typically including the chi-square test statistic with degrees of freedom, RMSEA with confidence intervals, CFI, TLI, and SRMR. The article does not report any of these indices for the final model. Without them, it is difficult for readers to evaluate how well the proposed structure reproduces the observed covariance matrix. It is possible that the model fits the data well—given that it was built iteratively to maximise fit—but the absence of fit statistics means this cannot be independently assessed. Future replications would benefit from reporting the full suite of fit indices, as well as information criteria (AIC, BIC) that would allow comparison with alternative model specifications.
5. From Statistical Association to Causal-Loop Diagram
Perhaps the most conceptually interesting aspect of the article is the translation of SEM coefficients into a causal-loop diagram. CLDs are traditionally constructed through qualitative methods: stakeholder interviews, expert panels, and iterative group model-building sessions (Sterman, 2000). The authors’ approach is novel in that it derives the CLD entirely from quantitative data. This is both a strength and a source of epistemological tension. Statistical significance in SEM establishes that two variables are associated after controlling for other variables in the model; it does not, by itself, establish a causal mechanism. The CLD, however, is read as a causal narrative: “improving emergency readmissions causes data quality to deteriorate.” The gap between statistical association and causal mechanism is a well-known challenge in observational research (Pearl, 2009), and the authors acknowledge it implicitly by noting that their CLD serves as a reference framework rather than a definitive causal model. A mixed-methods approach—combining the data-driven SEM with qualitative validation by NHS managers and clinicians—would have strengthened the causal interpretation considerably.
Closing Thoughts
Patel, Chaussalet, and Millard (2008) asked an important question at a time when few researchers were asking it: do the indicators in a healthcare balanced scorecard actually work together, or do they pull in opposite directions? Their answer—that the NHS balanced scorecard contains conflicting subsets of indicators, with emergency readmissions sitting at the centre of a complex web of reinforcing and balancing loops—remains a powerful insight for healthcare managers and policymakers. The methodological choices they made were reasonable given the data and analytical tools available in the mid-2000s, and the article has earned its place in the literature as a pioneering application of SEM and system dynamics thinking to healthcare performance management.
At the same time, the advances in longitudinal modelling that have occurred since 2008—particularly the development of the RI-CLPM, the growing scepticism towards data-driven specification searches, and the increased expectations for model fit reporting—suggest that the same research question could now be addressed with greater methodological rigour. This is not a failing of the original study; it is the natural progression of science. The authors opened a door, and the field has since developed better tools to walk through it. Researchers who wish to build on this work would do well to adopt a theory-driven model specification, use at least three waves of data, implement the RI-CLPM or its extensions, report comprehensive fit indices, and complement the quantitative analysis with qualitative validation. The question Patel et al. (2008) posed is too important to leave unanswered with only one study.
References
Hamaker, E. L., Kuiper, R. M., & Grasman, R. P. P. P. (2015). A critique of the cross-lagged panel model. Psychological Methods, 20(1), 102–116. https://doi.org/10.1037/a0038889
Jacobucci, R., Grimm, K. J., & McArdle, J. J. (2016). Regularized structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(4), 555–566. https://doi.org/10.1080/10705511.2016.1154793
Kline, R. B. (2023). Principles and practice of structural equation modeling (5th ed.). Guilford Press.
Lucas, R. E. (2023). Why the cross-lagged panel model is almost never the right choice. Advances in Methods and Practices in Psychological Science, 6(1), 1–22. https://doi.org/10.1177/25152459231158378
MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111(3), 490–504. https://doi.org/10.1037/0033-2909.111.3.490
Newsom, J. T. (2015). Longitudinal structural equation modeling: A comprehensive introduction. Routledge.
Patel, B., Chaussalet, T., & Millard, P. (2008). Balancing the NHS balanced scorecard! European Journal of Operational Research, 185(3), 905–914. https://doi.org/10.1016/j.ejor.2006.02.056
Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). Cambridge University Press.
Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. McGraw-Hill.