The systematic review by Wang et al. (2015), titled “Methodology and reporting quality of reporting guidelines: systematic review,” published in BMC Medical Research Methodology, addresses the critical issue of reporting standards in health research. This study systematically evaluates the methodological and reporting quality of reporting guidelines published since January 2010, with a specific focus on the application of the 2010 “Guidance for Developers of Health Research Reporting Guidelines” (“the Guidance”) by Moher et al..
1. Background and Rationale for the Study:
The fundamental purpose of investment in health research is to advance scientific understanding and improve health outcomes. However, a significant obstacle to achieving these goals is the insufficient reporting of research methodology and findings. This lack of detail hinders critical appraisal, limits effective dissemination, impedes the applicability of research, and can even mislead patients and practitioners.
To counteract these issues and enhance research quality, experts have developed reporting guidelines. These guidelines are defined as explicit texts, checklists, or flow diagrams that, through a clear methodology, guide authors in reporting specific types of research. Landmark initiatives include the CONSORT (Consolidated Standards of Reporting Trials) Statement for randomized controlled trials (RCTs), developed in 1996. Following CONSORT, numerous other guidelines emerged, such as PRISMA (Transparent reporting of systematic reviews and meta-analysis), STARD (Standards for Reporting of Diagnostic Accuracy Studies), STROBE (Strengthening the Reporting of Observational Studies in Epidemiology), and ARRIVE (Animal Research: Reporting: In Vivo Experiments Guidelines). The adoption of these guidelines has consistently been linked to improved reporting quality in medical research.
Given the growing focus on the methodology behind reporting guidelines, Moher et al. published a review in 2010 of 81 reporting guidelines developed up to December 2009. This earlier review revealed significant information gaps across various aspects of guideline development, particularly concerning consensus methods; for example, 28% of the guidelines provided no information about their consensus processes, and 57% were silent on how feedback after consensus was handled. In response to these identified gaps, the “Guidance for Developers of Health Research Reporting Guidelines” (“the Guidance”) was developed in 2010.
Since January 2010, there has been considerable investment and a substantial increase in new, updated, and expanded reporting guidelines. The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network’s library, for instance, saw its number of guidelines increase from over 90 in December 2009 to over 200 by April 2014. This rapid growth and resource allocation necessitated a systematic evaluation of reporting guidelines published since January 2010 to assess their methodological and reporting quality and to investigate how well “the Guidance” has been applied.
2. Objectives of the Study:
The primary aim of Wang et al.’s systematic review was to:
- Systematically review reporting guidelines published since January 2010.
- Determine the status of their reporting and methodology quality.
- Investigate the application of “the Guidance”.
- Ultimately, understand how much improvement has been achieved and how well “the Guidance” has been applied in practice.
3. Methodology:
The authors employed a rigorous systematic review methodology.
- Search Strategy: A systematic search was conducted across several databases: Cochrane Methodology Register, MEDLINE, and EMBASE. This was supplemented by retrieval from the EQUATOR Network (www.equator-network.org) and, where available, the websites of individual guidelines to find accompanying documents such as checklists, flowcharts, and explanation and elaboration (E&E) documents. The search keywords included “guideline” and “study design”.
- Inclusion and Exclusion Criteria: The review included the most recent versions of reporting guidelines for various fields of medical research and their accompanying documents. Excluded were guidelines for specific research projects (e.g., conducting a questionnaire survey), instructions for authors, guidelines for clinical practitioners reporting diagnostic/treatment information, and any guidelines published before December 2009. This exclusion criterion for publication date was key to focusing on guidelines developed after the initial review by Moher et al. (2009) and the publication of “the Guidance” (2010).
- Study Selection and Data Extraction: Pairs of review authors independently screened titles and abstracts, and all reviewers participated in inclusion decisions. Two reviewers (XQW and QW) independently identified guidelines, with disagreements resolved by a third reviewer (QFW). Data extraction was performed independently by pairs of review authors (XQW and NL, QFW and CLW) using a pre-designed and pilot-tested extraction form. Discrepancies were resolved through discussion or by consulting a third reviewer (QW). The collected outcomes focused on:
- Basic information (title, publication year, authors).
- Summary statistics for reporting outcomes of main interest, including background (version, scope, target research type, users), pre-meeting activities (member identification/recruitment, pre-planned endpoint of consensus, item inclusion criteria), face-to-face consensus meetings (format, number/backgrounds of experts, agenda), post-meeting activities (pilot testing, feedback collection), and other information (checklist, flow diagram, E&E, endorsement, translations, implementation plan, research gap, conflicts of interest).
- Specifically, whether the guidelines were compliant with “the Guidance”, judged by explicit declaration in the text or reference list.
- Data Analysis: Data was summarized descriptively using frequencies, proportions, and medians. Information was presented in tables and narrative summaries. The authors also explored core items from “the Guidance” and conducted a comparison between guidelines that explicitly mentioned following “the Guidance” and those that did not. The study followed the PRISMA statement for reporting.
4. Results:
The systematic search yielded 4329 records, which were narrowed down to 60 included reporting guidelines after screening and retrieving additional guidelines from the EQUATOR network.
- General Characteristics: 60 reporting guidelines for health research were included; 28 were from databases, and 32 were from the EQUATOR network. All were initially published in English. A considerable number were published and updated in recent years.
- Types of Guidelines: 38 (63%) were new guidelines, 17 (28%) were extensions, and 3 (5%) were updates of previous guidelines.
- Target Users and Scope: 39 (65%) guidelines reported their target users. Guidelines covered a broad spectrum of research typologies, with 40 (67%) designed for specific studies or multiple study designs. The most common specific designs were observational studies (11, 18%) and RCTs (9, 15%).
- Methodology for Initial Items: 45 (75%) guidelines described the methods used to generate initial items, often including literature review (42%), referring to existing reporting guidelines (33%), or opinions from experts (27%).
- Consensus Methods:
- 38 (63%) guidelines reported reaching consensus, and 35 (58%) described their consensus methods. However, none defined what constituted consensus.
- Formal methods were used in 23 (38%) guidelines, with Delphi exercise being applied in 13 (22%). Informal methods were identified in 5 (8%).
- Only 11 (18%) guidelines reported the endpoint of the consensus process, which were all terminated after a fixed number of rounds.
- Stakeholders in the consensus process most commonly included content experts (42%), methodologists (32%), and journal editors (23%).
- Face-to-Face Consensus Meetings: 24 guidelines used consensus meetings, with 18 (30%) holding only face-to-face meetings. Few reported the number of invited or attending experts, or detailed meeting agendas or materials.
- Post-Meeting Activities: Pilot tests were conducted in only 12 (20%) guidelines. Of these, 11 (18%) described the pilot methods, but even fewer described feedback requirements (12%) or collection methods (8%).
- Other Information:
- Final items were presented in 58 (97%) guidelines, with a median of 22 items, and 44 (73%) provided an explanation for each item.
- Checklists were provided in 54 (90%) guidelines, and 7 (12%) included flow diagrams.
- Separate Explanation and Elaboration (E&E) documents were developed or under development in only 16 (27%) guidelines.
- Endorsements were reported in 18 (30%) guidelines.
- Funding was obtained for at least 36 (60%) guidelines, but was unclear or not reported in 20 (33%).
- Conflicts of Interest (COIs) were clearly reported in 42 (70%) guidelines; 14 (23%) declared COI, while 28 (47%) did not provide information.
- Application of “the Guidance”: A critical finding was that only 9 (15%) of the 60 guidelines explicitly stated they followed “the Guidance”.
- A comparison of guidelines that mentioned “the Guidance” versus those that did not revealed better conducting and reporting among the former. However, even in these guidelines, certain core aspects—such as discussing the rationale for including items, strategy for producing documents, and feedback requirements—were still poorly conducted. Overall, the reporting of all core items in “the Guidance” was poor, with each item reported in less than 50% of reporting guidelines.
5. Discussion:
The authors noted a significant shift in the landscape of reporting guidelines since Moher et al.’s 2009 review. While previously most guidelines focused on RCTs, there is now increasing attention on other types of health research, including observational studies, animal research, and case reports.
- Gaps in Methodology: Despite the increase in new guidelines, the study highlighted several methodological shortcomings:
- A significant portion of guidelines did not search for existing relevant guidelines (only 37%) or adequately review literature (only 58%), and few provided search details. This is crucial for avoiding duplication and ensuring new guidelines are needed.
- More than 30% of guidelines did not report consensus methods, and for those that did, details were poorly reported, echoing previous findings. The importance of formally defining consensus and its threshold was emphasized.
- Pilot testing of final checklists was infrequent (only 20%). Pilot testing is vital for identifying issues with instructions, clarity, and formatting before formal publication.
- The development of Explanation and Elaboration (E&E) documents was also rare (only 27%). These documents are crucial for enhancing the understanding, use, and dissemination of guidelines by providing rationale and necessary details.
- Many guidelines lacked strategies for implementation, evaluation, and monitoring.
- While funding reporting improved, COI reporting still needed improvement.
- Impact of “the Guidance”: The finding that only 15% of guidelines explicitly followed “the Guidance” is a major concern. Although guidelines that mentioned “the Guidance” showed better reporting on core items, they still had weaknesses, particularly regarding the rationale for item inclusion, document production strategy, and feedback mechanisms. This suggests that “the Guidance” has a positive effect when applied, but its adoption remains low.
6. Limitations of the Study:
The authors acknowledged several limitations:
- They did not search grey literature, which might have led to missing some eligible guidelines.
- Data abstraction largely depended on reported information, though authors were contacted for clarification when information was unclear.
- The comparison focused mainly on core items and the overall trend, rather than an exhaustive analysis of all aspects.
- Not all guidelines listed on the EQUATOR network were included due to definitional differences (e.g., excluding “Toolkits” not designed for reporting research).
7. Conclusions:
The study concludes that while increasing attention has been paid to developing reporting guidelines for a wider range of study designs beyond RCTs, only a limited number of these guidelines (15%) were developed in compliance with “the Guidance”. Despite some identified improvements in reporting, details of consensus methods and other critical aspects remain poorly reported, consistent with previous findings.
The comparison strongly suggests that reporting guidelines mentioning “the Guidance” used more rigorous development methods aligned with recommendations. Therefore, the authors emphasize the critical need for greater attention to the quality of reporting guidelines and recommend that actively promoting the use of “the Guidance” could significantly improve the development and quality of future reporting guidelines. They also suggest that the EQUATOR Network should play a more active role in providing instructions for improving the quality of reporting guidelines.
Reference for the Article:
Wang, X., Chen, Y., Yang, N., Deng, W., Wang, Q., Li, N., Yao, L., Wei, D., Chen, G., & Yang, K. (2015). Methodology and reporting quality of reporting guidelines: systematic review. BMC Medical Research Methodology, 15(74). https://doi.org/10.1186/s12874-015-0069-z