This article, titled “Is There a Consensus on Consensus Methodology? Descriptions and Recommendations for Future Consensus Research,” by Jane Waggoner, Jan D. Carline, and Steven J. Durning, was published in Acad Med, 91(5) from pages 663–668 on January 19, 2016. The core purpose of the article is to address the significant lack of stringent guidelines for conducting and reporting consensus studies, despite their increasing use and utility, which has led to variability in both practice and results. The authors aimed to synthesize existing approaches and develop a comprehensive set of guidelines and suggestions to improve the reliability and validity of future consensus methodology studies.
Background and Rationale for Consensus Methods
Consensus methods have seen increasing use over the last 60 years as a means to achieve general agreement on topics that lack empirical evidence for future decisions or actions. These topics are often ambiguous or controversial. Beyond addressing evidence gaps, consensus methods are also employed to forecast future events or establish decision protocols, such as creating diagnostic algorithms and treatment plans in clinical settings. The National Institutes of Health (NIH), for instance, utilizes consensus development panels to gather agreement on diagnosing and treating common morbidities in the United States. Despite their widespread application, particularly in healthcare settings over the past 40 years, the absence of clear guidelines has made it challenging to ascertain the reliability and validity of such studies and justify their use in clinical and academic medical decision-making.
Research Methodology
The authors initially sought to identify standards for conducting well-run consensus studies across the Delphi, nominal group process, and consensus development panel approaches. They performed an extensive literature review across several electronic databases, including PubMed, ERIC, Psych Articles, PsycINFO, Web of Science, and Google Scholar. Their search terms included “Delphi,” “nominal group,” “consensus panel,” and “method”.
However, their initial review quickly revealed a “lack of rigor in methodological discussions” surrounding consensus methodology. This prompted a shift in their focus from identifying standards to determining how the majority of researchers were actually conducting, analyzing, and reporting consensus studies. They also investigated how researchers measured the success of their studies, such as evaluating for reliability or conducting follow-up studies. This shift was deemed important because these methodologies are commonly used in creating clinical diagnostic criteria and making decisions in medical education.
From an initial pool of 50 publications dealing with the three main consensus methods, 32 were ultimately used in their final assessment. Articles were included if they formally focused on the nature of the technique or provided sufficient description to identify its components. Those that only briefly described the methods or focused solely on outcomes were excluded. The authors also considered publications that provided supporting arguments, either through statistical analysis or literature support, for the synthesis and development of their guidelines.
Three Main Consensus Methods and Guidelines
The article specifically reviewed three main consensus methods: the nominal group process, consensus development panels, and the Delphi technique. For each, the authors discuss characteristics, advantages, limitations, and propose guidelines for future research and reporting.
1. Nominal Group Process
- Definition: Also known as the expert panel method, this technique allows a group of experts to privately generate ideas or solutions to a specific question. These ideas are then anonymously submitted to a moderator, presented to the entire panel, discussed for clarification, and finally ranked anonymously on a predetermined scale. A predetermined cutoff for consensus is crucial.
- Panel Composition: Composed exclusively of experts, often multidisciplinary.
- Panel Size: Ideally ranges from 5 to 9 members, with some suggesting 5 to 10. Fewer than 5 members may lead to a lack of diverse opinions, while more than 10 can unnecessarily lengthen the process and introduce more differences of opinion.
- Face-to-Face Contact: Yes.
- Literature Provided: As necessary.
- Rounds/Sessions: Typically involves 4 sessions.
- Utility: Examined the appropriateness of interventions in healthcare, impacting patient treatment decisions. It is useful in academic medical research for developing or testing standards (e.g., length of stay after surgery).
- Advantages:
- Anonymity reduces external influence, relying solely on expert opinion.
- Allows time for thought clarification without delaying the study.
- Time-efficient due to focused, on-task expert panels.
- Limitations:
- Expensive and complex to organize, requiring significant resources (time, meeting place, money).
- Face-to-face meetings can pose credibility problems or expose dissenting beliefs, potentially hindering collaboration and quality of results.
- Guidelines for Future Studies and Reporting:
- The research question used to prompt the panel must be clear and concise to elicit valid suggestions.
- The panel size should range from 5 to 10 members, as fewer may lack novel suggestions, and more may hinder consensus (“too many cooks in the kitchen”).
- Groups should be heterogeneous to foster more creative solutions.
- To ensure reproducible results, researchers must explain how groups were chosen and the specific criteria used to determine how and when a consensus was met.
2. Consensus Development Panels
- Definition: Also known as consensus development conferences, these are organized meetings of experts (potentially multidisciplinary) in a specific field to formulate policies and strategic plans. The NIH employs the most well-developed method for these panels to evaluate scientific literature and produce consensus statements accessible to both professionals and the public.
- Panel Composition: Experts in a given field, potentially multidisciplinary.
- Panel Size: Recommended between 5 and 10 members, with Nair et al. suggesting 10 as the optimum number, and the article’s guidelines recommending 8 to 12 members.
- Face-to-Face Contact: Yes.
- Literature Provided: Yes, presented by experts in the field.
- Rounds/Sessions: Variable, as needed.
- Utility: Useful in healthcare for a multidisciplinary approach to problem-solving and policy creation, particularly for debating the state of science related to clinical decisions. This method relies on evidence-based opinions rather than personal experience.
- Advantages:
- Synthesis of the best available information in the field.
- Experts take ownership of the material due to its direct impact, enhancing validity.
- Delivers rapid results.
- Limitations:
- Little research exists regarding the specific methods, advantages, limitations, or reliability/validity of outcomes from these conferences.
- Resource-intensive, requiring significant cost, time, and excellent organization (e.g., obtaining a moderator, meeting locations).
- Potential for bias introduction from overly vocal panel members.
- Guidelines for Future Studies and Reporting:
- Panel composition: Should consist of experts in the field, with the publication clearly reporting how they were chosen and why.
- Panel size: The panel should have between 8 and 12 members, with 10 suggested as optimal.
- Statistical analysis: Must be reasonable for the research question and as rigorous as possible. Researchers need to explain what constituted consensus and how it was assessed.
3. Delphi Technique
- Definition: Defined as a method for achieving consensual agreement among expert panelists through repeated iterations of anonymized opinions and proposed compromise statements from a moderator. Named for the Oracle at Delphi, it uses multiple rounds of questionnaires.
- Origin: Developed by the RAND Corporation in 1948 to forecast military events, it has since become popular across fields, including medical education and clinical settings, largely due to its time and cost efficiency with the advent of electronic communication.
- Panel Composition: Experts in their field, potentially multidisciplinary.
- Panel Size: Recommended between 6 and 11 members. While Dagenais reported increasing reliability up to 11 members, Nair et al. suggest anywhere from 10–30, noting limited reliability below 6 and insignificant increases above 12. The size often reflects the research question rather than statistical reliability.
- Face-to-Face Contact: No, panel members never interact and are unaware of each other’s identities.
- Literature Provided: An overview of literature is typically provided.
- Rounds/Sessions: Requires at least two rounds before consensus is reached. The optimal number of rounds is two. Researchers analyze responses and report back to respondents after each round.
- “Modified Delphi”: The term is often used but lacks a concise, specific definition, with many studies implementing various modifications without explicit definition.
- Utility: Primarily used in academic medicine for establishing decision-making criteria/protocols and curriculum development (e.g., determining important topics in a medical school curriculum).
- Advantages:
- Eliminates bias and influence from face-to-face meetings due to respondent anonymity, allowing for freer expression of opinions.
- Respondents are less likely to “jump on the bandwagon” if their views differ from the majority.
- Multiple rounds ensure thoughtful consideration and improve reliability compared to a single meeting.
- Does not require specified meeting times, offering flexibility and reducing participant fatigue.
- Limitations:
- Judgments in later rounds may be influenced by feedback provided by researchers.
- Involves a lack of collaboration and an increased potential for participant burnout as the number of rounds grows.
- Success is largely dependent on the quality of the questionnaire design.
- Statistical Analysis: While often lacking in detail in many reviewed studies, some literature addresses it. Ways to define consensus include a predetermined agreement percentage (e.g., 80%), a rating scale (e.g., 1 to 5), or a majority of participants rating a topic for inclusion. Examples of rigorous analysis include weighted kappa statistics, ANOVA, Bonferroni adjustments, or determining mean and standard deviations with specific ranges to identify consensus (e.g., means between 2 and 3 indicating uncertainty on a four-point scale). The authors noted a “wide range of criteria that points to consensus” without clear explanations for how these benchmarks are determined.
- Guidelines for Future Studies and Reporting:
- Panel composition: Experts in a given field; researchers must explain how “expert” was defined and the method used for obtaining participants.
- Panel size: 6 to 11 members are recommended.
- Rounds: Two rounds are required and optimal; if a study goes beyond two rounds, the reason for doing so must be explained.
- Statistical analysis: Must be reasonable for the research question and as rigorous as possible. Researchers need to explain what constituted consensus and how it was assessed.
Overall Conclusions and Recommendations
The authors ultimately found no existing standard practice for conducting consensus methodology studies. Their research shifted to focusing on common practices that were described in detail and analyzed stringently. They believe that consensus methods can be highly beneficial in addressing topics in academic medicine, drawing parallels from studies in other fields.
The article concludes by providing a comprehensive set of guidelines and suggestions for future researchers to design and report consensus methodology studies with greater reliability and validity. General recommendations applicable across all methods include:
- A clear description of inclusion criteria for panel members.
- An optimal panel size of 5 to 11 members.
- Ensuring statistical analyses are as rigorous as possible.
- Including the predetermined definition of consensus in the final manuscript.
The authors hope that these guidelines will lead to more stringent standards in the design, methods, and reporting of future consensus research.
Reference: Waggoner, J., Carline, J. D., & Durning, S. J. (2016). Is There a Consensus on Consensus Methodology? Descriptions and Recommendations for Future Consensus Research. Acad Med, 91(5), 663–668. https://doi.org/10.1097/ACM.0000000000001092