Interrupted Time Series Design in Healthcare: Methodology and Reporting

The article you shared, “Methodology and reporting characteristics of studies using interrupted time series design in healthcare” by Hudson, Fielding, and Ramsay, is a methodological study itself. This means its primary purpose wasn’t to evaluate a healthcare intervention directly, but rather to examine how other studies evaluating healthcare interventions were conducted and reported, specifically those using a research design called Interrupted Time Series (ITS).

Why this study was needed: In healthcare research, the “gold standard” for determining if an intervention causes a specific outcome is often a Randomised Controlled Trial (RCT). However, RCTs are not always possible due to ethical, practical, or cost reasons. In such situations, researchers turn to quasi-experimental designs, which don’t involve random assignment but still aim to understand cause-and-effect. The ITS design is considered one of the strongest of these quasi-experimental options. An ITS design involves collecting data at multiple, equally spaced time points before and after a specific intervention occurs, to see if the pattern of data changes after the intervention.

The main goal of this methodological study was to systematically investigate:

• How ITS designs were being used in healthcare.
• What their specific design features were.
• How consistently and thoroughly these studies were reporting their methods and results.

Methodology of the Study (How they did their research):

To achieve their aim, the researchers followed a systematic approach:

1. Defining what to include (Inclusion and Exclusion Criteria):
   • They focused on studies that used an Interrupted Time Series (ITS) design.
   • The studies had to have collected a minimum of two data points before the intervention and at least one data point after the intervention. This is crucial for an ITS, as you need enough data to establish a trend before and after the “interruption” (the intervention).
   • They only included studies that assessed a health or healthcare intervention (like programs, policies, or educational initiatives).
   • They were very broad in other aspects: there were no restrictions on the participants, the language of the study, or the type of outcome being measured. This helps ensure their findings are representative of a wide range of ITS studies.
   • They excluded certain types of papers, such as systematic reviews, meta-analyses, and RCTs, as their focus was specifically on ITS designs and their reporting. They also excluded studies that included a control group or did not use a proper ITS analysis.
2. Finding the relevant studies (Search Strategy):
   • They conducted a systematic search in a major medical database called MEDLINE.
   • They specifically looked for studies published in 2015. While this means they didn’t cover all years, the authors felt it would still provide a representative snapshot.
   • A primary researcher (JH) screened all the titles and abstracts identified by the search. To ensure accuracy, two other authors (CR and SF) independently double-checked 10% of these titles and abstracts. If there were no disagreements, the primary researcher continued screening alone. The same process was followed for assessing the full text of potentially relevant articles.
3. Gathering information from the studies (Data Extraction and Analysis):
   • They created a special form to extract data from each of the included studies. This form was tested by all authors to ensure it captured all necessary information consistently.
   • The primary researcher (JH) then extracted the details from all the included studies. Again, to ensure reliability, two other authors (CR and SF) independently extracted data from 10% of randomly selected studies. If there were no disagreements, the primary researcher proceeded with the rest.
   • They collected a wide range of information, including:
     • How the study defined its design (e.g., calling it an ITS, a “before-and-after” study, or a quasi-experimental study).
     • The country where the study was conducted.
     • The objectives of the study (what they aimed to find out).
     • The type and level of the intervention (e.g., a policy change, a new program, and whether it was at a hospital or individual level).
     • The participants involved.
     • The source of the data (e.g., hospital records, national databases).
     • The type of outcome measured.
     • The number of data points collected before and after the intervention and how frequently data was collected (e.g., weekly, monthly, yearly).
     • Crucially, they looked at methodology characteristics, such as how certain statistical issues were handled.
     • How the intervention’s effects were estimated and reported.
     • Details about what was reported in the abstract and discussion sections of the papers.
   • For studies with multiple outcomes, they focused on the primary outcome if one was defined, otherwise they used the first reported outcome.
   • The collected data were then summarized using descriptive statistics, like simple counts and percentages, or medians and percentiles, to provide an overview of the findings.
   • It’s important to note that they did not assess the risk of bias for individual studies. This is because their study was about methodology and reporting, not about the effectiveness of the interventions themselves.

Key Methodological Findings of the Study (What they discovered about ITS studies):

After following this rigorous methodology, the researchers included 116 studies in their analysis. Here’s what they found regarding how ITS studies were being conducted and reported:

• Types of Interventions and Data Frequency:
  • The interventions evaluated were most often programs (35%) and policies (28%).
  • Data was usually collected at monthly intervals (64%).
• Statistical Analysis Methods:
  • Of the studies that reported an analysis method, the most common approach was segmented regression (78%). Segmented regression is a statistical technique used to analyze changes in trends and levels within a time series after an intervention.
  • Other methods were also used, such as ARIMA models and Generalised Estimating Equations.
• Important Methodological Considerations (and how they were addressed): When analyzing ITS data, there are several critical statistical factors that need to be considered because they can affect the accuracy of the results:
  • Autocorrelation: This refers to when data points collected close together in time are correlated with each other (e.g., this month’s data is related to last month’s data).
    • It was considered in only 55% of the studies.
    • Of those that considered it, only 63% reported formal testing for it.
    • If not properly accounted for, autocorrelation can lead to under- or overestimation of intervention effects.
  • Nonstationary or Secular Trend: This is when the data naturally increases or decreases over time regardless of the intervention.
    • Only 8% of studies considered nonstationary trends.
  • Seasonality or Cyclic Patterns: These are regular, repeating patterns in the data, like a yearly fluctuation (e.g., flu cases peaking in winter).
    • Only 24% of studies considered seasonality.
  • Sample Size Calculation: This is determining how many data points are needed to detect a meaningful effect.
    • Only seven studies (6%) reported a sample size calculation.
    • Only one of these calculations was reproducible. This lack of calculation makes it hard to know if the study had enough data to confidently detect a real change.
  • Other considerations: Less commonly reported were outliers in data, handling of missing data, or “transition periods” (a time allowed for the intervention to take full effect).
• Reporting of Intervention Effects:
  • Intervention effects were most commonly reported as a change in slope (84%) (meaning a change in the rate of increase/decrease) and change in level (70%) (meaning an immediate jump or drop in the outcome).
  • However, the variety of ways effect estimates were reported, and sometimes the lack of confidence intervals or standard errors, made it difficult to interpret findings consistently across studies and challenging to combine results in meta-analyses.

In Conclusion: This methodological study highlighted significant problems and inconsistencies in how ITS designs were reported in healthcare research. Key methodological features, such as considering autocorrelation, nonstationary trends, seasonality, and sample size calculations, were often overlooked or poorly reported. This lack of clear and consistent reporting makes it challenging for other researchers to understand, compare, and synthesize findings from ITS studies. The authors strongly recommended the development of formal reporting guidelines for ITS studies through a consensus process to improve the quality of future research in this area.1 kaynak

Reference: Hudson, J., Fielding, S., & Ramsay, C. R. (2019). Methodology and reporting characteristics of studies using interrupted time series design in healthcare. BMC Medical Research Methodology, 19, 130. https://doi.org/10.1186/s12874-019-0777-x