Evaluating Bibliometrics: A Practical Peer Review Guide

The article “Evaluating Bibliometrics Reviews: A Practical Guide for Peer Review and Critical Reading,” authored by Anh-Duc Hoang and published in Evaluation Review, systematically explores the strengths, limitations, and methodological complexities inherent to bibliometric research. Recognizing that bibliometric methods have become fundamental in assessing scholarly contributions, the paper emphasizes the critical importance of structured, transparent, and reproducible evaluation processes.

Hoang introduces the VALOR framework (Verification, Alignment, Logging, Overview, Reproducibility) designed explicitly to ensure rigorous quality control in bibliometric studies that employ multiple databases and analytical tools. Verification focuses on consistent metadata across databases, standardizing author names, affiliations, and citation counts. Alignment highlights the necessity of synchronizing timeframes across different bibliographic databases and managing early-access publications accurately. Logging emphasizes detailed documentation of search strategies, database-specific methodologies, and parameter settings within bibliometric software tools. Overview refers to cross-validation and consistency checks of key findings, such as citation metrics, keyword analyses, and visual network structures. Lastly, Reproducibility mandates providing access to datasets, clearly outlining analytical procedures, and transparently documenting each step to facilitate external verification and reuse.

Additionally, the author proposes practical guidelines for evaluating two primary dimensions of bibliometric research: Performance Analysis and Science Mapping. Performance Analysis, which includes descriptive metrics such as citation counts, h-index, and journal impact, must account for temporal factors like publication indexing delays and differences between local and global citations. The author critically discusses normalization techniques, such as mean-normalized citation scores and source-normalized impact per paper (SNIP), which are essential for fair comparisons across disciplines with distinct citation behaviors.

In terms of Science Mapping, Hoang outlines comprehensive criteria for assessing conceptual structure (keyword co-occurrence, thematic evolution), intellectual structure (co-citation analysis, historiography), and social structure (co-authorship networks). He underscores the importance of thoughtful interpretation beyond mere numerical or visual outputs, encouraging evaluators to critically analyze cluster labeling, interpret thematic trends contextually, and thoroughly explain collaborative dynamics at institutional and international levels.

The paper further integrates these two analytical streams, urging reviewers and readers to critically interpret discrepancies and convergences between performance metrics and network analyses. For example, the position of authors or institutions in citation rankings versus their centrality in collaborative networks can reveal distinct forms of influence or hidden collaboration patterns that simple citation metrics might obscure.

Finally, Hoang highlights several methodological challenges and areas for future advancement, including the integration of qualitative methods with quantitative bibliometric analysis, improving natural language processing tools for deeper conceptual analyses, and the refinement of field-normalized metrics to address biases stemming from interdisciplinary research.

Overall, this article serves as an extensive practical guide, providing peer reviewers, authors, and readers alike with detailed, actionable frameworks and evaluation strategies to enhance the quality, clarity, and reliability of bibliometric research outcomes (Hoang, 2025).

The VALOR Framework for Multi-Source Studies

Hoang’s central contribution is VALOR, a checklist for bibliometric studies that combine databases such as Web of Science, Scopus and Google Scholar: Verification of data consistency (from metadata format to author name disambiguation), Alignment of time synchronisation across sources, Logging of documentation, Overview of result validation, and Reproducibility. Authors should justify why several databases were used and explain how records were merged and deduplicated.

What Reviewers Should Check

The guide separates performance analysis (productivity and impact of sources, authors, institutions, countries and documents) from science mapping, which covers conceptual structure (keyword co-occurrence, thematic evolution), intellectual structure (co-citation, historiography) and social structure (collaboration networks). For each, it lists reviewer questions, such as how influential works were identified, how clusters were defined, and whether the chosen strands match the stated research questions.

Citation metrics require normalisation. Field-normalised indicators such as MNCS suit comparisons within similar fields, while SNIP compares better across disciplines; citation windows should be reported because recent papers are disadvantaged. The paper also recalls the classic laws of Lotka (author productivity), Bradford (core journals) and Zipf (word frequency) as the theoretical base of the method.

Limits Readers Should Keep in Mind

Three challenges recur: data source limitations (selective Web of Science coverage, broader Scopus coverage, wide but noisy Google Scholar coverage), citation dynamics, and interdisciplinary complexity. Biases such as self-citation, citation circles and over-reliance on classic papers cannot be removed entirely. For readers, the guide stresses distinguishing temporary fluctuations from substantial shifts in a field before drawing conclusions. It is an opinion article offering frameworks, not an empirical validation of them.

Reference:
Hoang, A.-D. (2025). Evaluating Bibliometrics Reviews: A Practical Guide for Peer Review and Critical Reading. Evaluation Review. Advance online publication. https://doi.org/10.1177/0193841X251336839