In the age of data-driven science, bibliometric analysis has become a powerful tool for mapping knowledge, identifying research trends, and evaluating scientific impact. However, a recent study by Mehmet Nurullah Kurutkan, published in the International Journal of Business Science & Applications (2024), brings to light a crucial methodological weakness that is often overlooked: the risks and limitations of topic-based search strategies in bibliometric studies.
Why Topic Search Isn’t Always Safe
The study reveals that the widely used “topic” search in Web of Science—which scans the title, abstract, and author keywords—can yield misleading results due to multiple technical and semantic reasons. While convenient, these searches may incorporate irrelevant articles due to database quirks, abbreviation overlaps, or institutional metadata pollution.
Six Key Problems Identified
Kurutkan identifies six recurring issues that threaten the validity of bibliometric outputs:
- Sponsor Coding Confusion: Some terms appear not because they’re the research subject, but because they’re part of a sponsor’s name or affiliation.
- Funding Body Noise: Names of funding agencies containing keywords may falsely pull unrelated publications into the dataset.
- Semantic Confusion from Addresses: Words like “Karl Marx” or other concepts may be picked up because they appear in street or institutional names—not as research content.
- Abbreviation Chaos: Acronyms like “TCCM” may refer to entirely different concepts across disciplines (e.g., mining vs. management theory), introducing heterogeneous and misleading data.
- Keywords Plus Side Effects: Web of Science’s algorithm-generated keywords (Keywords Plus) often add terms based on cited references, not actual article content, creating noise in topic-based searches.
- Word-Number Combinations: Concepts like “Industry 4.0” are highly sensitive to formatting. If a space, dot, or number is misplaced (e.g., “health. 4.0”), unrelated results might slip in.
Why It Matters
These issues are not trivial. Including irrelevant publications in bibliometric datasets can distort co-authorship networks, keyword maps, citation analyses, and thematic clusters. This may lead researchers to draw conclusions based on non-existent patterns, jeopardizing the credibility of their findings and misleading the broader academic community.
Recommendations
The paper offers a detailed checklist for authors, reviewers, and editors to ensure methodological rigor. These include:
- Manually validating top-cited articles.
- Avoiding over-reliance on algorithmically added metadata like Keywords Plus.
- Using full-text phrase searches rather than abbreviations.
- Running pilot searches to detect semantic noise or formatting errors.
- Being transparent and replicable by sharing exact search URLs.
Conclusion
Kurutkan’s work serves as a vital wake-up call for the bibliometric community. As bibliometric outputs increasingly inform policy, funding, and academic evaluations, ensuring methodological soundness is more important than ever. “Topic” may seem like a harmless filter—but as this study shows, when used carelessly, it can open the door to serious analytical errors.
Full Citation
Kurutkan, M. N. (2024). Handicaps and Potential Dangers of Topic Search Strategy in Bibliometric Research. International Journal of Business Science & Applications, 4(2), 93–110.