GLOBAL: Guidance for Reporting Bibliometric Analyses

This article addresses a significant and growing concern within scholarly communication: the lack of standardized, evidence-based guidance for reporting bibliometric analyses. Despite their increasing prevalence in peer-reviewed literature and their crucial role in research evaluation for grant proposals, promotions, and institutional assessments, the reporting quality of these studies varies greatly. This inconsistency hinders the reliability, robustness, and reproducibility of bibliometric research.

To mitigate this issue, Ng et al. undertook a scoping review, which is the focus of this paper, as a foundational step towards developing a comprehensive reporting guideline for bibliometric analyses, provisionally named the Guidance List for the repOrting of Bibliometric AnaLyses (GLOBAL). The primary objective of this scoping review was to systematically identify and categorize existing recommendations for reporting bibliometric analyses from a broad array of sources to generate an initial list of candidate items for GLOBAL.

Methodology of the Scoping Review:

The researchers employed a rigorous and comprehensive search strategy:

• Database Searches: They systematically searched five bibliographic databases (MEDLINE, EMBASE, APA PsycINFO, Web of Science Core Collection, and Scopus) from their inception dates up to July 28, 2023. The search strategy was developed in OVID MEDLINE and peer-reviewed using the PRESS tool before being adapted for other databases.
• Preprint Servers and Grey Literature: To capture guidance not published as traditional academic articles, searches were conducted on three common preprint servers (arXiv, OSF Preprints, Zenodo) and a curated list of grey literature sources (e.g., blogs, websites of bibliometrics-related organizations). General Google searches were also performed, reviewing the first 100 results for eligibility.
• Eligibility Criteria: Sources were included if they provided recommendations for reporting a bibliometric analysis and were in English. Bibliometric analyses themselves, conference abstracts, and study protocols were generally excluded, with an exception for bibliometric analyses whose title or abstract specifically mentioned reporting recommendations.
• Screening and Selection: A total of 48,750 documents were identified across all sources. After removing duplicates, 45,411 records remained. To manage this large volume, a combination of human screening and DistillerSR’s AI Screening tool (DAISY) was utilized, with human screeners handling 10% of records and the AI screening the remainder. Full-text screening for all retrieved documents was conducted independently and in duplicate by two reviewers, with conflicts resolved by consensus or third-party arbitration. Ultimately, 23 documents met the inclusion criteria: 13 from databases and 10 from preprint servers and grey literature.
• Data Extraction and Synthesis: A pilot-tested data extraction form was used to collect detailed information from each eligible source, including bibliometric reporting recommendations. All recommendations pertaining to reporting a bibliometric analysis were extracted. These extracted recommendations were then categorized, and similar ones were combined to create candidate items for the GLOBAL reporting guideline. The research team, including bibliometrics specialists and reporting guideline experts, iteratively discussed and reached consensus on the inclusion, section assignment (e.g., ‘title’, ‘abstract’, ‘methods’), and phrasing of each item. Experts also had the opportunity to add items not directly found in the literature but deemed necessary for reporting rigor, such as data cleaning methods.

Key Findings:

The scoping review yielded several significant findings:

• Paucity of Evidence-Based Guidance: A critical discovery was the “paucity of evidence-based studies on bibliometric reporting”. Of the 23 included documents, a striking 17 were opinion-based and did not adhere to the rigorous methodology suggested by the EQUATOR Network’s “How to develop a reporting guideline” toolkit (e.g., systematic literature review, consensus exercise). Only one source (Montazeri et al., 2023) fully adhered to EQUATOR’s toolkit by conducting both a systematic literature search and a Delphi consensus meeting to create a reporting guideline called BIBLIO, though this also had some limitations, such as a small expert panel and missing reporting details. This overwhelming reliance on opinion-based recommendations underscores the urgent need for GLOBAL.
• Preliminary GLOBAL Recommendation List: The synthesis of extracted recommendations resulted in a 32-item preliminary GLOBAL reporting guideline. These items are sub-categorized according to standard manuscript sections: ‘title’ (1 item), ‘abstract’ (1 item), ‘introduction’ (5 items), ‘methods’ (13 items), ‘results’ (4 items), ‘discussion’ (5 items), and ‘other’ (3 items).
• Most Frequently Addressed Items:
  • The most frequently mentioned item (14 times) was ensuring that methods are reported in a replicable and transparent manner. This aligns with the broader academic emphasis on research transparency to enhance credibility and reproducibility.
  • Presenting full database search strategies, including filters and limitations, was addressed in 13 documents.
  • Describing the databases and data sources used, along with their limitations, was cited in 11 sources.
• Least Frequently Addressed Items and Expert Contributions: Six items were mentioned by only one study each, including: defining terms, defining units of analysis, reporting uncertainty/dispersion/heterogeneity and error values, identifying future research directions, describing data availability and accessibility, and using references to support statements. Crucially, the item “describe the data cleaning methods, including any limitations,” was not mentioned in any reviewed source but was added to the preliminary list based on the expert opinions of the research team’s bibliometric specialists. This highlights a critical oversight in current reporting practices, despite data cleaning’s importance for data quality and validity.

Significance and Future Directions:

The GLOBAL reporting guideline is intended to serve as a standardized tool to aid a wide range of stakeholders, including bibliometricians, librarians, policymakers, research evaluators, authors, editors, and peer reviewers. By promoting its widespread adoption, the authors aim to increase the quality, completeness, transparency, and reproducibility of published bibliometric analyses. This is particularly vital as bibliometric analyses continue to grow in popularity and bibliometric databases become more accessible.

The current 32-item preliminary GLOBAL list is a first step. The next phase involves further development through a two-round Delphi study with relevant stakeholders, followed by an international consensus meeting. This multi-stakeholder consensus approach is designed to ensure the guideline is robust and broadly accepted, making GLOBAL the first such guideline developed through an international, multi-stakeholder consensus.

Strengths and Limitations:

The study’s strengths include its independent and duplicate screening and data extraction processes, which enhance reliability. Furthermore, the extensive search across journals, preprint servers, and grey literature ensures a thorough review of existing recommendations. However, a limitation is the exclusion of non-English language documents, which may mean some relevant recommendations were overlooked. Despite comprehensive searches, the authors also acknowledge the possibility that not all relevant recommendations were captured.

In conclusion, the article by Ng et al. clearly establishes the critical need for robust reporting guidelines in bibliometric analysis and presents a well-researched foundation for the GLOBAL guideline, which promises to significantly elevate the standard of reporting in this increasingly important field.

Reference: Ng, J. Y., Liu, H., Masood, M., Syed, N., Stephen, D., Ayala, A. P., Sabé, M., Solmi, M., Waltman, L., Haustein, S., & Moher, D. (2024). Guidance for the Reporting of Bibliometric Analyses: A Scoping Review. OSF. https://doi.org/10.17605/OSF.IO/WYP63

Preliminary GLOBAL Recommendation List:

Title

• 1.1 In the title, identify the study as a bibliometric analysis and indicate the time period and key issues/topic.

Abstract

• 2.1 Abstract should be reflective of the bibliometric analysis, including scope, data collection, analysis, and results.

Introduction

• 3.1 Situate the bibliometric analysis within the context of relevant pre-existing literature, identifying the gap in literature.
• 3.2 Define the aim, scope, rationale, and/or objective of the bibliometric analysis.
• 3.3 Define the research question.
• 3.4 Clearly define all relevant terms and definitions used within the bibliometric analysis.
• 3.5 Describe the intended target audience of the bibliometric analysis (e.g., researchers, public, media, etc.). Describe the ways in which the information included in the report may be used for the target audience.

Methods

• 4.1 Describe the bibliometric methods used.
• 4.2 Define the units of analysis that are analysed (i.e., micro-, meso-, and macro-level) in the bibliometric analysis (e.g., countries, institutions, authors).
• 4.3 Describe the bibliometric data collection methods, including any limitations.
• 4.4 Describe the databases and data sources used, including any limitations.
• 4.5 Present the full search strategies for all databases used, including any filters and limits that were applied.
• 4.6 Describe the data collection time frame.
• 4.7 Describe the search results and selection processes (e.g., inclusion/exclusion). If applicable, use a flow diagram.
• 4.8 Describe the data cleaning methods, including any limitations (This item was not described in any literature but was added based on expert opinion).
• 4.9 Describe the bibliometric data analysis methods used.
• 4.10 Specify the analytical software used and the parameter settings selected.
• 4.11 Describe the bibliometric indicators used.
• 4.12 If applicable, define the calculations/formulas used for indicators in the bibliometric analysis.
• 4.13 Provide sufficient detail in the bibliometric analysis manuscript to ensure full replicability/transparency of methods.

Results

• 5.1 Describe the results and key findings.
• 5.2 Describe the results of bibliometric analysis techniques used.
• 5.3 Visualize the results through the use of figures, graphs, and/or tables. Ensure the visualizations are simple and easy to interpret. Aesthetic bibliometric visualization should not replace a rigorous bibliometric analysis.
• 5.4 If applicable, report the uncertainty/dispersion/heterogeneity depending on the type of analysis and error values of bibliometric indicators.

Discussion

• 6.1 Summarize and discuss study findings.
• 6.2 Elaborate on the applicability and implications of study findings.
• 6.3 Provide context for the results of the bibliometric analysis and situate the study findings in existing literature.
• 6.4 Discuss the strengths, limitations, and potential biases of the bibliometric analysis.
• 6.5 Identify future directions for research.

Other

• 7.1 Disclose any existing or potential conflicts of interest and/or sources of financial or non-financial support.
• 7.2 Describe the availability and accessibility of data.
• 7.3 Use references and citations to support statements and methods used.

This preliminary list is the outcome of a comprehensive scoping review and will be further developed through a Delphi study to achieve consensus among stakeholders, ultimately aiming to increase the quality, transparency, and reproducibility of published bibliometric analyses.