Cover image for The Delphi Technique and Consensus Methods: An Introduction (101)

The Delphi Technique and Consensus Methods: An Introduction (101)

In brief: When evidence is limited, health researchers often ask a panel of experts to rate statements over several anonymous rounds, feeding the group’s answers back each time, until agreement is reached: this is the Delphi technique (Keeney et al., 2001; MacFarlane & Zierhut, 2025). The rules matter. In a random sample of 100 Delphi studies, 98 claimed to assess consensus but only 72 defined it (Diamond et al., 2014), and when different published consensus criteria were applied to the results of two child health Delphi processes, the set of outcomes reaching consensus ranged from 5 to 44 (Webbe et al., 2023).

The Delphi technique and consensus methods: an introduction (101)

This guide introduces the Delphi technique and neighbouring formal consensus methods for students and practitioners in health management, nursing, public health and clinical research. It is built on 117 records (86 articles and 31 reviews) from the Web of Science Core Collection, published from 1987 to 2026, and every empirical statement is traced to one of them. The records are the methodological subset of a larger literature: most screened records were single Delphi applications without methodological discussion and were excluded. The guide covers definitions, history, variants, the nominal group technique, the RAND/UCLA appropriateness method and consensus conferences, design choices, the findings of methodological reviews, reporting guidance (CREDES and ACCORD), debates, research gaps and a design checklist.

Try it yourself: Name the consensus method behind short study descriptions, design a Delphi study and see which choices methodological reviews warn about, and test when a set of panel ratings counts as consensus under different rules.
Open the interactive module: Consensus Lab

Learning objectives

By the end of this guide you should be able to:

  1. Define the Delphi technique by its core features and distinguish it from the nominal group technique, the RAND/UCLA appropriateness method, consensus conferences and ordinary surveys.
  2. Classify Delphi variants (classic, modified, e-Delphi, real-time, policy and newer hybrid forms) and say what each changes.
  3. Plan panel selection and size, rounds, rating scale, consensus definition, feedback and the handling of attrition.
  4. Evaluate what methodological reviews and experiments show about Delphi practice, including null and mixed findings.
  5. Apply a design checklist and current reporting guidance (CREDES, ACCORD) to plan, review or report a consensus study.

Key concepts

  • Delphi technique. An approach for gaining consensus among an expert panel, normally through questionnaire rounds in which information is fed back to panel members (Keeney et al., 2001).
  • Core features. Anonymity, iteration, controlled feedback and statistical group response (Bellali & Karamitri, 2011; MacFarlane & Zierhut, 2025).
  • Expert panel. Participants selected for their knowledge and experience of the topic (Guzys et al., 2015); defining “expert” is itself problematic (Trevelyan & Robinson, 2015).
  • Round and controlled feedback. Statements are presented repeatedly, and after each round panellists see the panel’s previous responses and may reconsider (Dietrich et al., 2024).
  • Consensus definition. The rule, ideally fixed in advance, that decides when agreement is reached, most often a percentage agreement (Diamond et al., 2014; Schifano & Niederberger, 2025).
  • Modified Delphi. An adapted Delphi, for example with a physical panel meeting (Boulkedid et al., 2011); modifications are often not well described or justified (Wodnik et al., 2025).
  • Nominal group technique (NGT). A face-to-face small-group method with silent generation, round robin, clarification and voting (McMillan et al., 2016).
  • RAND/UCLA appropriateness method (RAM). A Delphi variant that synthesises evidence and elicits clinical judgement on appropriate treatment (Sparks et al., 2022).

Contents

  • Method, introduction and history
  • Definitions and neighbouring concepts; variants and neighbouring consensus methods
  • Design choices; what methodological reviews found; reporting and quality
  • Themes and bibliometric profile; Delphi and consensus methods in Türkiye
  • Debates, research gaps and implications for practice
  • Summary, discussion questions, further reading, glossary, frequently asked questions and references

How this guide was produced

Database and date. Web of Science Core Collection, Advanced Search, 18 November 2026; no language or year limits; articles and reviews.

Query (verbatim). TI=(Delphi OR "consensus method*" OR "nominal group technique*" OR "RAND/UCLA appropriateness") AND TI=(methodolog* OR guidance OR "reporting guideline*" OR "reporting standard*" OR "reporting quality" OR "best practice*" OR primer OR introduc* OR overview OR critique OR rigour OR rigor OR "how to" OR tutorial OR "systematic review" OR "scoping review" OR "literature review" OR "methodological review" OR "consensus definition*" OR "defining consensus" OR "panel size" OR attrition OR "expert panel selection") AND DT=("Article" OR "Review")

Search development. Earlier versions with broader method terms returned 7,756 and 2,711 records, dominated by single Delphi applications; the final query adds methodological, reporting and review terms to the title (840 records).

Criteria. Included: records that substantively address the Delphi technique or other formal consensus methods (concepts, variants, design choices, reporting guidance, methodological reviews, critiques or empirical tests). Excluded: single Delphi or consensus studies on a clinical or content topic without methodological discussion, protocols without results, and unrelated records.

Flow. See Figure 1: 840 identified, 3 duplicates removed, 0 retracted, 837 screened, 720 excluded (638 single applications, 65 protocols, 17 unrelated), 117 included (86 articles and 31 reviews), published 1987 to 2026.

Screening and extraction. AI-assisted screeners worked on titles and abstracts; the lead reviewer resolved 2 unsure records. Evidence from the abstracts of all 117 included records was extracted into a structured template. Eight records have no abstract in the database and are used only for what their titles state.

Limitations. Single database; title search, so methodological papers with other title wording are missed; abstract-level reading without quality appraisal; AI-assisted screening; English-language bias of Web of Science.

Records identifiedWeb of Science Core Collection, 18 Nov 2026n = 8403 duplicates removed0 retracted publicationsRecords screenedtitle and abstract, AI-assisted screenersn = 837Excluded, n = 720638 single Delphi or consensusstudies on a clinical or contenttopic without methodologicaldiscussion (E1)65 protocols without results (E2)17 unrelated records (E3)Screening codes E1 to E32 unsure records resolvedby the lead reviewerRecords includedn = 11786 articles and 31 reviewspublished 1987 to 2026Evidence was then extracted from theabstracts of all 117 included recordsinto a structured template (definitions,history, design choices, reviews,reporting, strengths and limitations,comparisons, debates, gaps, Türkiye)
Figure 1. Selection of records for this guide. Counts come from the Web of Science export and the screening log. Most excluded records were single Delphi or consensus applications; the included corpus is the methodological subset.

Introduction: why consensus methods matter in health research

Many health decisions cannot wait for trials. When research evidence is limited, inconsistent or absent, decisions and policies need to rest on consensus among interested stakeholders (Logullo et al., 2024), and even with abundant evidence, recommendations and priority-setting may require agreement from multiple, sometimes opposed parties (Gattrell et al., 2024). In the health sciences the Delphi technique is applied mainly in the exploratory or evaluative phases of research to synthesise experts’ explicit and implicit knowledge (Niederberger & Deckert, 2022). It is used to solve clinical problems, draw up protocols and guidelines, set research priorities and plan services (Bellali & Karamitri, 2011), to select quality indicators (Boulkedid et al., 2011) and to agree core outcome sets for trials (Lee et al., 2020).

Its rigour matters because consensus statements guide care where definitive evidence is missing (Blazey et al., 2024). Yet the method’s flexibility brings a lack of detailed guidelines (Sakamoto, 2024), its popularity has produced adaptations that may threaten the rigour of the original format (Keeney et al., 2001), and consensus methods are often poorly reported despite their influence (Gattrell et al., 2024). A Delphi study can therefore look authoritative while hiding arbitrary choices.

History and development

Origins. In the early 1950s logicians and mathematicians at the RAND Corporation ran experiments on the predictive capacities of groups of experts; Delphi’s creators first saw it as a technique, then as a scientific method and later as a tool (Dayé, 2018). RAND pioneered the expert-panel method in the 1950s and 1960s for forecasting technological events, and numerous organisations later adopted it for local and national policy (Normand et al., 1998). The policy Delphi was first proposed in Technological Forecasting and Social Change in 1970 (de Loë et al., 2016), and Delphi reached its culmination in marketing research in the 1970s and 1980s (Nyström & Kaartemo, 2022).

Health research. A 1987 critique in nursing has no abstract in the database (Goodman, 1987). In 1994 consensus decisions in earlier Delphi work were criticised as resting on arbitrary or post hoc rationales (Williams & Webb, 1994), and a 2001 nursing review warned that the technique’s growing popularity had led to adaptations that might threaten the rigour of the original format (Keeney et al., 2001). In 2010 the ASCO Board of Directors approved guideline development using consensus methodology (Loblaw et al., 2012). A 2011 special issue noted that the technique was over half a century old (Rowe & Wright, 2011).

The reporting era. Reviews then documented poor reporting (Boulkedid et al., 2011) and widely varying consensus definitions (Diamond et al., 2014). CREDES was proposed in 2017 (Jünger et al., 2017). Among EQUATOR Network reporting guidelines, the share developed with Delphi was less than 10% in 2011 and 2012 and 29% in 2019 (Banno et al., 2020). In 2024 ACCORD became the first reporting guideline applicable to all consensus-based studies in biomedicine (Gattrell et al., 2024). Recent records add real-time polling (O’Hagan et al., 2026), a language model as co-expert (Nowak et al., 2026) and synthetic panels (Bossler et al., 2026) (Figure 2).

Milestones in Delphi and consensus methods stated in the corpusEarly 1950sRAND experimentson predictive powerof expert groups1950s and 1960sRAND expert panelsfor forecastingtechnological events1970Policy Delphiproposed in TechnolForecast Soc Change1970s to 1980sCulmination ofDelphi in marketingresearch2010ASCO Boardapproves consensusmethodology2011Special issue:Delphi “over halfa century” old2017CREDES proposedfor Delphi studiesin palliative care201929% of reportingguidelines in 2019used Delphi2024ACCORD reportingguideline for allconsensus methods2026LLM as co-expert;synthetic Delphipanels testedNot to scale. Years are as stated in the abstracts; the 2011 entry gives the state at publication, the 2019 entry the share among EQUATOR guidelines published that year.
Figure 2. Timeline of milestones stated in the corpus. Sources, left to right: Dayé, 2018; Normand et al., 1998; de Loë et al., 2016; Nyström & Kaartemo, 2022; Loblaw et al., 2012; Rowe & Wright, 2011; Jünger et al., 2017; Banno et al., 2020; Gattrell et al., 2024; Nowak et al., 2026; Bossler et al., 2026. The 2019 share refers to reporting guidelines in the EQUATOR Network published in that year.

Definitions

Definitions share one idea: a structured, usually anonymous exchange among selected experts, repeated over rounds with feedback, to find out how far they agree. They differ in emphasis (consensus, forecasting, positions or knowledge synthesis) and on the kind of method Delphi is: a qualitative methodology (McPherson et al., 2018; Murry & Hammons, 1995) or one combining quantitative and qualitative methodology (Jenkins & Smith, 1994). Table 1 paraphrases ten definitions.

Source Year Discipline Definition (paraphrased) Emphasised dimension
Keeney et al., 2001 2001 Nursing Gains consensus among an expert panel through rounds of questionnaires with feedback Rounds and feedback
Bellali & Karamitri, 2011 2011 Health care sciences Seeks reliable consensus of a pre-selected panel where data are unavailable, through anonymity, expert input, iteration with controlled feedback and statistical group response Four defining features
Guzys et al., 2015 2015 Health and well-being research A structured anonymous conversation that cultivates collective opinion and highlights disagreement while minimising group dynamics Disagreement and group dynamics
McMillan et al., 2016 2016 Pharmacy practice A multistage self-completed questionnaire with individual feedback to determine consensus from a larger group of experts Questionnaire format
de Loë et al., 2016 2016 Policy research Policy Delphi: iterative stages that reveal positions on an issue Positions
Dayé, 2018 2018 History of operations research and policy sciences A procedure that produces forecasts on technological and social developments Forecasting
Rao et al., 2021 2021 Orthodontics An iterative, multi-stage structured group communication process to reach consensus Structured communication
Niederberger & Deckert, 2022 2022 Medicine and health sciences Originally for structuring group communication; now a consensus method synthesising explicit and implicit expert knowledge Knowledge synthesis
Dietrich et al., 2024 2024 Imaging Panellist selection, anonymity, iterative rounds with controlled feedback and insight into previous panel responses Process principles
MacFarlane & Zierhut, 2025 2025 Genetic counseling A consensus method for expert perspectives when conclusive evidence is unavailable Use when evidence is lacking

The four classic features recur across fields (Bellali & Karamitri, 2011; Fathullah et al., 2023; MacFarlane & Zierhut, 2025). Most health definitions aim at consensus, but the policy Delphi reveals positions (de Loë et al., 2016) and the dissensus Delphi explores divergent ideas (Melander, 2018). No definition says how much agreement counts as consensus; each study must decide. Figure 3 arranges the elements of a Delphi study in sequence.

Anatomy of a Delphi study: from question to agreed output1. Questionlimited or uncertainevidence2. Round oneliterature review,open items, pilot3. Expert panelwho counts as expert,size, diversity4. Rating roundsanonymous ratings,controlled feedbackrepeat5. Stopping rule defined in advanceconsensus threshold, number of rounds or stabilityof responses (not the same as agreement)6. Output: statements, guidelines, indicators, core outcome setswith items that did not reach consensus and dissent reportedCross-cutting concerns at every stepBiassteering group, panel, conformityAttritionresponse and retention ratesReportingCREDES, ACCORD, justify changesConceptual scheme for teaching. Elements follow the cited sources; the arrangement is the author’s synthesis.
Figure 3. Conceptual scheme of a Delphi study. Use when evidence is limited follows MacFarlane and Zierhut (2025); literature reviews as a starting point follow Fathullah et al. (2023) and an exploratory workshop follows Frewer et al. (2011); the problem of defining “expert” and the distinction between consensus and stability follow Trevelyan and Robinson (2015); anonymity, iteration and controlled feedback follow Dietrich et al. (2024); stopping rules follow Diamond et al. (2014); reporting dissent follows Blazey et al. (2024); bias domains follow Nasa et al. (2026); reporting guidance follows Jünger et al. (2017) and Gattrell et al. (2024).

Neighbouring concepts

Waggoner and colleagues reviewed three common consensus methods side by side: the nominal group process, consensus development panels and the Delphi technique (Waggoner et al., 2016). In research partnerships, a scoping review of 34 studies found at least 11 consensus methods, with Delphi variations the most common (Cary et al., 2023). Table 2 and Figure 4 compare nine neighbouring concepts.

Concept What it centres on Evidence from the corpus
Delphi technique Anonymous, iterative questionnaire rounds with controlled feedback Used when conclusive evidence is unavailable (MacFarlane & Zierhut, 2025)
Modified Delphi An adaptation of the classic procedure 49 of 78 quality indicator studies (63%), with a physical panel meeting, usually between rounds (Boulkedid et al., 2011)
e-Delphi Rounds run electronically Controlled, quasi-anonymous feedback by electronic means (Rüetschi & Olarte Salazar, 2022)
Real-time Delphi Results shown in real time; iteration principle changed Computer-based format (Gnatzy et al., 2011)
Nominal group technique Face-to-face small groups with voting Prompt result but requires attendance (McMillan et al., 2016)
RAND/UCLA method Evidence synthesis plus appropriateness ratings A Delphi variant, also run online (Sparks et al., 2022)
Consensus conference Discussion and voting at a meeting Hybrid conference voting on 32 draft statements (Grupp et al., 2023)
Expert panel The group whose judgement is elicited Delphi panels, modified Delphi panels and expert round tables (Evans, 1997)
Survey One questionnaire round without feedback Delphi uses a series of questionnaires (Paré et al., 2013); iteration and feedback set it apart (author’s contrast)
The Delphi technique and its neighbours: which features are central?AnonymousratingIterativeroundsControlledfeedbackFace-to-facediscussionPrespecifiedagreement ruleRemoteparticipationDelphi techniqueModified Delphie-DelphiReal-time DelphiNominal group techniqueRAND/UCLA methodConsensus conferenceExpert panelSurvey (single round)centralpartly or sometimesnot centralAuthor’s synthesis for teaching, not an empirical rating.
Figure 4. Matrix comparing the Delphi technique (shaded row) with eight neighbouring methods and concepts on six features. The ratings are the author’s synthesis, not a finding of the corpus; see Table 2 for cited evidence on each concept.

Variants and neighbouring consensus methods

Delphi variants differ in key epistemological and methodological activities (Niederberger & Deckert, 2022). A review of 764 Delphi and modified Delphi studies from 2008 to 2018 found heterogeneity on core defining characteristics and modifications that sometimes contradicted the method’s core principles (Jandhyala, 2020). Table 3 compares the main forms.

Classic and modified Delphi. The classic method sends sequential questionnaires to a pre-selected panel by mail or e-mail, without geographical limits (Bellali & Karamitri, 2011), and has policy and decision subforms (MacFarlane & Zierhut, 2025). “Modified” covers many adaptations: ASCO’s formal consensus has clinicians rate recommendations drafted from a systematic review (Loblaw et al., 2012), others add a physical meeting between rounds (Boulkedid et al., 2011) or a hybrid consensus conference (Grupp et al., 2023).

e-Delphi and real-time Delphi. Electronic rounds allow quasi-anonymous feedback (Rüetschi & Olarte Salazar, 2022), and software can yield near-immediate results and lower attrition (McPherson et al., 2018), though an Asia-Pacific electronic Delphi still faced problems with recruitment, sample size and instrumentation (Hung et al., 2008). A computer-based real-time Delphi increased efficiency and reduced drop-out; comparing it with conventional Delphi studies, its authors found no significant differences between the formats (Gnatzy et al., 2011). Real-time polling in two rounds of a four-round modified Delphi with 57 round 1 participants preserved anonymity and reduced study duration (O’Hagan et al., 2026).

Policy and dissensus Delphi. A review of 63 policy Delphi studies (1971 to 2014) found little consistency in design and execution (de Loë et al., 2016). Because reaching consensus may limit exploration of radical ideas, a dissensus Delphi can explore divergent views of the future (Melander, 2018).

Hybrid and AI-assisted forms. A hybrid Delphi combining focus groups, the NGT and Delphi yielded highly satisfactory proposals in three real cases (Landeta et al., 2011), and Delphi has been paired with a World Cafe workshop (Maskrey & Underhill, 2014), Q-methodology (Sakamoto, 2024) and fuzzy numbers (Roldán López de Hierro et al., 2021). Language models now prepare items (62 Delphi-ready items from 5 penile cancer guidelines and the literature) (Haas et al., 2026), act as a co-expert (Nowak et al., 2026) or form whole panels: a 60-agent synthetic panel converged under every condition (coefficient of variation reductions of 81 to 91%), regardless of how well it covered the field (Bossler et al., 2026).

Nominal group technique. The NGT gives a prompt result in face-to-face groups and suits consumer and stakeholder views, whereas Delphi is commonly used to develop guidelines with health professionals (McMillan et al., 2016). In 57 studies eliciting survey items, the number of nominal groups ranged from 1 to 41 and group size from 2 to 30 (Harb et al., 2021). Ranking by sum of votes or by voting frequency can change priorities for ideas with the same score (McMillan et al., 2014).

RAND/UCLA method and consensus conferences. The RAM was developed to synthesise evidence and elicit medical experts’ judgement on appropriate treatment (Sparks et al., 2022). In a RAM adaptation with 58 stakeholder organisations, surgeons rated geriatric surgery standards more critically than nonsurgeons in round 1 (91.2% valid versus 100%) but rose to 98.7% after an in-person round 2 (Berian et al., 2018). Consensus conferences vote on draft statements at a meeting (Grupp et al., 2023; Li et al., 2026). Wholly different traditions exist too, such as an Inuit consensus methodology based on successive addition of group members and an unhurried meeting style (Ferrazzi et al., 2019).

Choosing. The choice depends on the research question, the consensus required, time and geography (McMillan et al., 2016); deciding which method is best is the first of five key steps in one primer (Gottlieb et al., 2023).

Method How it works Typical use Citation
Classic Delphi Anonymous questionnaire rounds with controlled feedback Guidelines, priorities, curricula Bellali & Karamitri, 2011
Modified Delphi Classic rounds plus a literature-based start or a meeting Guidelines with limited evidence; quality indicators Loblaw et al., 2012; Boulkedid et al., 2011
e-Delphi Rounds administered electronically Dispersed panels Rüetschi & Olarte Salazar, 2022
Real-time Delphi Results shown in real time Foresight with busy experts Gnatzy et al., 2011
Policy Delphi Reveals positions on an issue Policy questions de Loë et al., 2016
Nominal group technique Silent generation, round robin, clarification, voting Stakeholder views; survey items McMillan et al., 2016; Harb et al., 2021
RAND/UCLA method Evidence synthesis, then appropriateness ratings Appropriateness of care Sparks et al., 2022
Consensus conference Statements discussed and voted at a meeting Practice guidelines Li et al., 2026
Synthetic Delphi Language model agents rate items over rounds Experimental foresight Bossler et al., 2026

Design choices

Every Delphi must settle the selection of experts, their number, panel quality, the iterative rounds and the criteria for completing the process (López-Gómez, 2018). Trevelyan and Robinson identified ten such areas from two parallel Delphi studies (Trevelyan & Robinson, 2015), and primers turn them into steps (Gottlieb et al., 2023; MacFarlane & Zierhut, 2025; Olsen et al., 2021; Wodnik et al., 2025). The numbers below describe practice in particular reviews, not universal norms.

Panel selection and size

Who is an expert? Defining “expert” is problematic (Trevelyan & Robinson, 2015), and panel design and management are central design decisions (Donohoe & Needham, 2009). Experts’ backgrounds and context affect the possibilities for consensus (Melander, 2018), homogeneous panels are a recognised criticism (Flanagan et al., 2016), and a multicountry study treated maximum variation in perspective as a decision point (Wodnik et al., 2025).

Patients and other affected people are often under-represented. In 90 international core outcome set Delphi surveys, the median number of healthcare professionals was 60 against 30 patients, only 31% recruited from low and middle income countries and 7% translated their surveys (Lee et al., 2020). Affected parties such as patients took part in about a quarter of 287 health science Delphi studies (Schifano & Niederberger, 2025), health consumers in 6 of 22 data element studies (27%) (Strekalova et al., 2024), and people with intellectual disability in nine of 55 publications, where family caregivers carried greater weight (Piantedosi & O’Shea, 2025). Translation helps: translating a gastric cancer Delphi survey into 7 languages added 637 participants to the 315 in the source-language survey (Alkhaffaf et al., 2021). There were indications that recruiting patients through treatment centres gave higher round two response rates than recruiting through patient organisations (Barrington et al., 2021).

Panel size. Waggoner and colleagues found 5 to 11 members most beneficial across three consensus methods (Waggoner et al., 2016), but practice varies: a median of 17 members (Q1 11, Q3 31) in 80 quality indicator studies (Boulkedid et al., 2011), a median of 42 at first rating in web-based RAM studies reporting it (Sparks et al., 2022), and about 370 experts invited, with a response rate around 35%, in a national forecasting Delphi (Chakravarti et al., 1998).

Rounds

Three rounds have been called optimal (Trevelyan & Robinson, 2015); ACCORD’s three rounds were completed by 58, 54 and 51 of 72 panellists (Gattrell et al., 2024), while other studies used two, with a third planned for unresolved items (Lurvink et al., 2021), four (O’Hagan et al., 2026) or six feedback rounds (Toillier et al., 2022). What stops the process matters more: of 100 studies, 70 ended after a specified number of rounds and only 23 linked stopping to consensus (Diamond et al., 2014). Round one needs structure to avoid unmanageable data (Trevelyan & Robinson, 2015), and literature reviews are appropriate starting points (Fathullah et al., 2023). More rounds do not always bring convergence: in five national science and technology Delphi surveys, second-round variance was in most cases higher than first-round variance (Sokolov et al., 2025).

Rating scales

Likert scales with four to seven categories have been suggested, an even number avoiding midpoint problems (Trevelyan & Robinson, 2015). The scale can change the result: in a pelvic girdle pain core outcome set study, participants randomised to a 5-point scale produced twice as many consensus outcomes after three rounds as those using a 9-point scale, and reached consensus on all five final outcomes against four (Remus et al., 2021). Open-ended questions are common, but 62% of 287 studies did not report how they were analysed (Schifano & Niederberger, 2025).

Defining consensus

In 100 Delphi studies from 2000 to 2009, 98 claimed to assess consensus but only 72 defined it (64 a priori); percent agreement was the most common definition, with a median threshold of 75%; and of 86 studies concluding that consensus was achieved, only 42 had set a threshold a priori (Diamond et al., 2014). Percentage agreement defined consensus in 81% of 287 later studies (Schifano & Niederberger, 2025), and only 10 of 22 patellofemoral pain statements (45%) reported their threshold (Blazey et al., 2024). Examples include at least 75% agreement (Loblaw et al., 2012), 51.0% (Lurvink et al., 2021), and medians of at least 7 for validity and 4 for feasibility without disagreement (Nathan et al., 2023). Applying different published criteria to the same child health Delphi data produced 5 to 44 consensus outcomes, while mean, median and rate of exceedance gave similar rankings (Webbe et al., 2023). Consensus should not be confused with stability of response (Trevelyan & Robinson, 2015).

Feedback and anonymity

Feedback should show central tendency and dispersion, preferably graphically (Trevelyan & Robinson, 2015), yet only 60% of 80 quality indicator studies reported giving feedback (Boulkedid et al., 2011) and 5 of 12 web-based RAM studies gave none (Sparks et al., 2022). In interviews with 24 participants, some first-time participants struggled to understand the survey, and some interviewees wanted guidance on whose perspective to take when scoring (Biggane et al., 2019). Anonymity aims to stop any individual or group from exerting excessive influence (Dietrich et al., 2024); cognitive biases can let louder voices dominate, and anonymous voting is one mitigation (Nasa et al., 2026). Power dynamics were reported as a challenge in practice (Sterling et al., 2023) and rarely discussed in research partnerships (Cary et al., 2023).

Attrition

High attrition has long been a concern (Evans, 1997; O’Donnell et al., 2025). Well-informed participants and short intervals between rounds reduce it (Trevelyan & Robinson, 2015), and real-time formats were reported to reduce drop-out (Gnatzy et al., 2011). Reporting is weak: response rates for all rounds appeared in 39% of 80 studies (Boulkedid et al., 2011), no web-based RAM study reported response rates with the previous round as denominator (Sparks et al., 2022), and only 11% of 102 aging studies set panel size based on anticipated attrition (Jarrott et al., 2026).

What methodological reviews of Delphi studies found

Many records review how Delphi or consensus studies were done in a field. Table 4 lists the main ones with the numbers their abstracts give. Each row is a separate finding; the counts should not be added, because the reviews cover different fields, periods and definitions.

Review Field n of Delphi studies Key finding Citation
Systematic review, 1978 to 2009 Healthcare quality indicators 80 Response rates for all rounds reported in 39%, feedback in 60%, consensus method in 77% Boulkedid et al., 2011
Systematic review, 2000 to 2009 Multidisciplinary random sample 100 Consensus defined in 72; set a priori with a threshold in 42 of 86 claiming it Diamond et al., 2014
Systematic review, 1971 to 2014 Policy Delphi 63 Little consistency; limited validity and reliability tests; mixed reporting de Loë et al., 2016
Methodological systematic review, 1997 to 2015 Palliative care guidance 30 Considerable differences in rigour and reporting; discrepant terms such as “round” Jünger et al., 2017
Systematic review, 2008 to 2018 Delphi and modified Delphi 764 Heterogeneity on core defining characteristics Jandhyala, 2020
Systematic review, 2017 to 2019 International core outcome sets 90 Fewer patients than professionals; 31% recruited from low and middle income countries Lee et al., 2020
Systematic review, after 2011 EQUATOR reporting guidelines 62 of 244 guidelines Reporting quality of Delphi moderate in most Banno et al., 2020
Systematic review, to 2019 Emergency nursing guidance 22 Modified Delphi in 68.2%; quality high (12/14) but transparency varied Varndell et al., 2021
Literature review, 2009 to 2019 Web-based RAND/UCLA panels 12 Full RAM in 42%; no feedback in 42% Sparks et al., 2022
Scoping review, 2018 to 2021 Health sciences consensus Delphi 287 Modified Delphi 43%; consensus as percentage agreement 81% Schifano & Niederberger, 2025
Scoping review, 2000 to 2024 Sport, exercise and performance psychology 54 Minor inconsistencies in reporting Quartiroli, 2025
Scoping review, searched 2025 Gerontology and geriatrics 102 43% lower than average quality on the DQ4A Jarrott et al., 2026
Systematic review, 2020 to 2024 Pharmaceutical sponsor involvement 82 Threshold rationale and disagreement handling inconsistently reported Brewer et al., 2026

Common patterns. Three findings recur: variable conduct that makes comparisons difficult (de Loë et al., 2016; Jünger et al., 2017; Schifano & Niederberger, 2025); incomplete reporting of response rates, feedback and consensus rules (Boulkedid et al., 2011; Diamond et al., 2014; Sparks et al., 2022); and loose labelling, with “modified Delphi” covering very different designs (Jandhyala, 2020; Jünger et al., 2017) and the Delphi label applied to proposals lacking its key elements (López-Gómez, 2018).

Mixed and positive findings. Emergency nursing Delphis were rated high in overall quality (Varndell et al., 2021), information systems ranking-type Delphis showed positive signs of rigour such as expert anonymity and clear instructions, but reporting of response and retention rates, instrument pretesting and justification of modifications still needed improvement (Paré et al., 2013), and reporting inconsistencies in sport psychology were minor (Quartiroli, 2025). In gerontology, 33% of studies were of higher than average quality (Jarrott et al., 2026).

Participation. Core outcome set Delphis that included patients varied strikingly in scoring, feedback, recruitment, reminders, incentives and piloting (Barrington et al., 2021). In bipolar disorder, five of nine consensus studies used Delphi, with limitations in diversity, co-design and retention (O’Donnell et al., 2025).

Reporting and quality

CREDES. After finding substantial variation in conduct and transparency among 30 palliative care Delphi studies, Jünger and colleagues proposed CREDES (Conducting and REporting of DElphi Studies) (Jünger et al., 2017).

ACCORD. ACCORD followed EQUATOR Network methodology: a systematic review, a Delphi process and meetings (Gattrell et al., 2024). The review of 18 studies found panel composition and consensus definition commonly addressed, and patient and public involvement, the steering committee’s role, conflicts of interest and funding least addressed (van Zuuren et al., 2022). A panel of 72 refined 56 preliminary items to 35 (Gattrell et al., 2024), and an explanation and elaboration document gives rationale, examples and a glossary (Logullo et al., 2024).

Other guidance named in the sources. These include a review of ten existing health science reporting guidelines that identified nine reporting categories and drafted an initial proposal, explicitly not a final guideline (Spranger et al., 2022); preliminary recommendations for web-based RAM (Sparks et al., 2022); the DQ4A rating form for aging research (Jarrott et al., 2026); the TRIAD framework for sponsor involvement (Brewer et al., 2026); and four rigour questions for consensus statements (Blazey et al., 2024). NGT users should document reasons for choices at each decision point (Harb et al., 2021) and report pilot meetings (Liang et al., 2025). Waggoner and colleagues asked that the predetermined consensus definition appear in the manuscript (Waggoner et al., 2016).

Rigour and bias. Gauging rigour remains elusive because of an ongoing epistemological debate and continual modifications (Hasson & Keeney, 2011). A methodological review mapped ten cognitive biases into four domains; its framework still needs prospective validation (Nasa et al., 2026). Users and reviewers should appraise the consensus building behind reporting guidelines carefully (Banno et al., 2020).

Strengths and limitations

Strengths. Delphi has no geographical limits and lets participants express views impersonally (Bellali & Karamitri, 2011), suits dispersed experts when international consensus is needed (Frewer et al., 2011), minimises group dynamics (Guzys et al., 2015) and reduces hierarchies (Freitas & Ferrarini, 2021). Its flexibility is a strength (de Loë et al., 2016).

Limitations. For the policy Delphi, inconsistent use undermines the ability to generate accessible insights (de Loë et al., 2016). A Delphi can take weeks or months and may be complex for lay people (McMillan et al., 2016). Concerns include seed information given to panellists, attrition, expert selection and the definition of consensus (Evans, 1997), and limitations often go undisclosed in published Delphi studies, while one applied linguistics project reported challenges with sampling, timing and power dynamics (Sterling et al., 2023). A Delphi can even change what it studies: in a neurosurgical clinic study, a modified Delphi run before baseline data collection became a catalyst for change, and chronological bias with the Hawthorne effect meant the null hypothesis could not be rejected (Armstrong et al., 2021).

Themes and bibliometric profile

Growth. Nine records were published before 2000 and seven from 2000 to 2009; the count rose to 38 from 2010 to 2019 and 63 from 2020 to 2026, peaking at 13 in 2021 (Figure 5). The 2026 count covers part of the year. Because the search required methodological terms in the title, the curve shows growth of methodological writing, not of Delphi applications.

Included records per publication year, 1987 to 2026 (n = 117)Before 2000: 92000 to 2009: 72010 to 2019: 382020 to 2026: 63051015111199021199512200011200532201010143420154372420201311108920258Alternating shaded bands mark the four periods; counts per period are shown at the top.Orange bar: 2026, which covers part of the year only (search date 18 November 2026). Years without a bar have no records.
Figure 5. Included records per publication year (Web of Science Core Collection, searched 18 November 2026), with counts for four periods. The 2026 count covers part of the year only.

Themes. Keyword matches in titles, abstracts and author keywords (Figure 6) put the expert panel first (73 records), followed by reporting guidance (48), rounds or feedback (44) and systematic or scoping reviews (39); modified Delphi (15), e-Delphi or online formats (13), consensus definition or threshold (12), attrition or response rate (12), the NGT (9), nursing (8), RAND/UCLA (7) and real-time Delphi (2) follow. A record can match several groups, so counts overlap.

Records matching each theme keyword group (a record can match several)020406080Expert panel73Reporting guidance48Rounds or feedback44Systematic or scoping review39Modified Delphi15e-Delphi or online13Consensus definition or threshold12Attrition or response rate12Nominal group technique9Nursing8RAND/UCLA7Real-time Delphi2Keyword matches in title, abstract and author keywords of the 117 included records. Counts overlap.
Figure 6. Number of included records whose title, abstract or author keywords match each theme keyword group. A record can match several groups, so the counts overlap and do not add up to 117.

Journals and categories. Technological Forecasting and Social Change publishes the most records (12), followed by BMJ Open (4); PLOS Medicine, Journal of Clinical Epidemiology and Journal of Medical Internet Research have 3 each; nine journals have 2 each (Journal of Advanced Nursing, PLOS ONE, European Journal of Futures Research, Palliative Medicine, International Journal of Qualitative Methods, BMC Medical Research Methodology, Trials, Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen and Anaesthesia). The leading Web of Science categories are Health Care Sciences & Services (18), Medicine, General & Internal (14), Business (13), Regional & Urban Planning (13), Health Policy & Services (9), Public, Environmental & Occupational Health (9), Nursing (7), Social Sciences, Interdisciplinary (7), Education & Educational Research (6) and Medical Informatics (5); records can belong to several.

Countries. By affiliation address: USA 29, UK 26, Canada 17, Germany 13, Australia 13, Spain 10, the Netherlands 10, and France and Switzerland 5 each. A record can count for several countries, and affiliations are not study settings.

Delphi and consensus methods in Türkiye

One record has a Türkiye-affiliated author, and no record mentions Türkiye, Turkey or Turkish in its title, abstract or keywords. The Türkiye-affiliated record is the literature review and gastric cancer case study on translating Delphi surveys into 7 target languages (Alkhaffaf et al., 2021); its abstract names neither the languages nor any Turkish setting, so the corpus cannot say whether a Turkish version was used. There is no methodological review of Delphi studies in Türkiye, no evaluation of thresholds, panels or attrition in Turkish health research and no study of adapting Delphi questionnaires into Turkish. This absence concerns the methodological literature indexed with these title terms; Turkish applications fall among the excluded single studies or outside the search. The international lessons on translation (Alkhaffaf et al., 2021), patient inclusion (Lee et al., 2020) and prespecified consensus rules (Diamond et al., 2014) apply directly.

Debates and contradictions

1. Consensus as truth or as agreement. A historical analysis argues that Delphi’s core assumption, that convergence over rounds can be read as consensus, appears unjustified for Delphi as a tool (Dayé, 2018). Second rounds in five national surveys generally did not bring convergence (Sokolov et al., 2025), and synthetic panels converged whether or not they covered the field, so agreement was not a sufficient validity criterion (Bossler et al., 2026). Claims of consensus can themselves be contested, as in the debate on renaming fatty liver disease (Fouad et al., 2022), and some authors aim for dissensus instead (Melander, 2018).

2. Consensus thresholds. Sources call for definitions fixed in advance (Waggoner et al., 2016; Webbe et al., 2023) but none settles the level: 75% (Loblaw et al., 2012) and 51.0% (Lurvink et al., 2021) both appear. Since criteria applied to the same data gave 5 to 44 consensus outcomes (Webbe et al., 2023), the threshold is a substantive choice. Criticised as arbitrary in 1994 (Williams & Webb, 1994), threshold rationales were still inconsistently reported in studies from 2020 to 2024 (Brewer et al., 2026).

3. Anonymity versus discussion. Classic Delphi values anonymity to limit group dynamics (Dietrich et al., 2024; Guzys et al., 2015), yet many designs add meetings (Berian et al., 2018; Boulkedid et al., 2011) or hybrid formats with anonymous polling (O’Hagan et al., 2026). After an in-person round, surgeons’ ratings aligned with those of nonsurgeons (Berian et al., 2018); whether this reflects learning or conformity is not answered by the abstract, and social influence is one of four bias domains (Nasa et al., 2026).

4. Expert selection and bias. Who is an expert is contested (Trevelyan & Robinson, 2015); patients, people with lived experience and participants from low and middle income countries are under-represented (Lee et al., 2020; Piantedosi & O’Shea, 2025; Strekalova et al., 2024); and the role of pharmaceutical sponsors is often reported incompletely (Brewer et al., 2026), while seed information given to panellists has long been named as a concern (Evans, 1997).

5. “Modified Delphi” labels. Terms such as “modified Delphi” are understood differently (Jünger et al., 2017), modifications are often not described or justified (Wodnik et al., 2025) and some contradict core principles (Jandhyala, 2020). With modified designs in 43% of health science studies (Schifano & Niederberger, 2025), the label alone says little.

Research gaps

  1. Empirical tests of design choices. More and better empirical studies were called for in 2011 (Rowe & Wright, 2011); studies of rigour are scant and component specific (Hasson & Keeney, 2011), and other rating scales need testing (Remus et al., 2021).
  2. Consensus rules. Only one record in this corpus tests several consensus criteria on the same Delphi data, in two child health core outcome set processes (Webbe et al., 2023); whether the 5 to 44 spread holds in other fields is untested here (author’s note).
  3. Inclusion. Geographical and income differences in prioritisation (Lee et al., 2020), health consumers as experts (Strekalova et al., 2024) and patients’ motives for taking part (Barrington et al., 2021) need study.
  4. Web-based and AI formats. Reporting standards for web-based panels (Sparks et al., 2022) and prospective validation of the bias framework (Nasa et al., 2026) are needed.
  5. Foundations and analysis. Methodological underpinnings (Guzys et al., 2015) and the analysis of NGT data across groups (McMillan et al., 2014) have received little attention.
  6. Policy uptake. How to inform policy uptake of Delphi outputs merits research (Frewer et al., 2011).
  7. Türkiye. No record in this corpus studies consensus methodology in Turkish settings (author’s count).

Implications for practice: a design checklist

For a health researcher planning a Delphi, for example to agree quality indicators for a hospital department, the steps below follow the cited sources; the sequence is the author’s synthesis.

  1. Justify the method and why not the NGT, RAM or a conference (Gottlieb et al., 2023; McMillan et al., 2016).
  2. Write a protocol first, fixing panel, rounds, scale, consensus and stopping rules, with reasons (Harb et al., 2021; Olsen et al., 2021).
  3. Build round one from evidence, such as a literature review, and pilot it (Fathullah et al., 2023; Frewer et al., 2011).
  4. Define and recruit the panel with stated criteria, varied perspectives, patients where relevant and translation for international panels (Alkhaffaf et al., 2021; Waggoner et al., 2016; Wodnik et al., 2025).
  5. Plan for attrition in panel size and timing (Jarrott et al., 2026; Trevelyan & Robinson, 2015).
  6. Choose the rating scale knowingly (Remus et al., 2021).
  7. Prespecify consensus with a rationale, and plan for items without consensus (Brewer et al., 2026; Diamond et al., 2014).
  8. Design feedback with central tendency and dispersion, and brief first-time participants (Biggane et al., 2019; Trevelyan & Robinson, 2015).
  9. Guard against bias with anonymous voting, neutral wording and disclosure of sponsorship (Brewer et al., 2026; Nasa et al., 2026).
  10. Report fully with ACCORD or CREDES, response rates per round and justified modifications (Boulkedid et al., 2011; Gattrell et al., 2024; Jünger et al., 2017).
Item to report What to include Why (evidence from the corpus)
Consensus definition Threshold, timing, rationale Defined in 72 of 100 studies (Diamond et al., 2014)
Stopping rule Rounds planned; link to consensus Linked to consensus in 23 of 100 (Diamond et al., 2014)
Panel Criteria, expertise, stakeholder groups Experience reported in 3 of 22 statements (Blazey et al., 2024)
Response per round Invited and responding, with denominator All rounds reported in 39% of 80 studies (Boulkedid et al., 2011)
Feedback Content and format between rounds Absent in 42% of web-based RAM studies (Sparks et al., 2022)
Governance Steering committee, conflicts of interest, funding Least addressed items (van Zuuren et al., 2022)
Dissent Items without consensus, minority views Acknowledged in 2 of 22 statements (Blazey et al., 2024)

Summary

  • The Delphi technique gathers expert judgement through anonymous, iterative rounds with controlled feedback, mainly when evidence is limited.
  • The NGT uses face-to-face groups, the RAND/UCLA method adds evidence synthesis and appropriateness ratings, and consensus conferences vote at a meeting; modified, e-Delphi, real-time, policy and AI-assisted variants change parts of the classic procedure.
  • Design choices change results: a 5-point scale gave twice as many consensus outcomes as a 9-point scale, and different consensus criteria gave 5 to 44 outcomes from the same data.
  • Reviews find variable conduct and incomplete reporting; in one sample of 100 studies only 72 defined consensus.
  • Patients, people with lived experience and low and middle income countries are under-represented on panels.
  • Convergence is not validity: later rounds may not converge, and synthetic panels converged regardless of field coverage.
  • CREDES and ACCORD (a 35-item checklist) set reporting standards.
  • No record in this corpus addresses consensus methodology in Türkiye.

Discussion questions

  1. A panel’s ratings converge over three rounds. What else would you need to know before calling the result consensus?
  2. One study sets consensus at 51% agreement, another at 75%. How would you judge each threshold?
  3. When does face-to-face discussion improve a consensus process, and when does it threaten it?
  4. Who should count as an expert when agreeing outcomes for patients with a chronic disease?
  5. Applied: draft protocol decisions for a Delphi to agree ten quality indicators for an emergency department in a Turkish hospital.
  6. Applied: could a synthetic panel of language model agents replace human experts in a health policy Delphi?

Further reading

  • Keeney et al. (2001): a critical review of the Delphi in nursing research.
  • McMillan et al. (2016): a practical guide to the NGT and the Delphi technique.
  • Trevelyan and Robinson (2015): ten design areas with concrete advice for a first protocol.
  • Diamond et al. (2014): how often consensus is undefined or defined after the fact.
  • Jünger et al. (2017): the review behind the CREDES standard.
  • Gattrell et al. (2024): the ACCORD reporting guideline and checklist.
  • Schifano and Niederberger (2025): a scoping review of 287 health science Delphi studies.
  • Webbe et al. (2023): how consensus criteria change results from the same data.

Glossary

Term Turkish Short definition
Delphi technique Delphi tekniği Anonymous, iterative questionnaire rounds with controlled feedback
Consensus methods Uzlaşı yöntemleri Formal methods for reaching agreement, such as Delphi, NGT and RAM
Expert panel Uzman paneli Participants selected for their knowledge or experience
Nominal group technique Nominal grup tekniği Face-to-face small-group method ending in voting
Consensus threshold Uzlaşı eşiği The prespecified level of agreement that counts as consensus
Round Tur One cycle of rating before feedback
Feedback Geri bildirim Summary of previous panel responses returned between rounds
Attrition Katılımcı kaybı Loss of panellists between rounds
Modified Delphi Modifiye Delphi A Delphi adapted from the classic form
RAND/UCLA appropriateness method RAND/UCLA uygunluk yöntemi Evidence synthesis plus expert ratings of appropriateness

Frequently asked questions

What is the Delphi technique? A method for gaining consensus among experts through questionnaire rounds with feedback (Keeney et al., 2001).

How many experts and rounds? No fixed rule: 5 to 11 members was found most beneficial in one review (Waggoner et al., 2016), and three rounds have been called optimal (Trevelyan & Robinson, 2015).

What percentage counts as consensus? A design choice to fix in advance; the median threshold in one review was 75% (Diamond et al., 2014).

Delphi or nominal group technique? The NGT meets face to face and is quick; Delphi uses questionnaires over rounds and suits dispersed panels but takes longer (McMillan et al., 2016).

Which reporting guideline? ACCORD for any consensus method in biomedicine (Gattrell et al., 2024); CREDES was proposed for palliative care Delphis (Jünger et al., 2017).

Related on this site: Research designs in health research: an introduction (101) · Coherence in research: an introduction (101) · Consensus methodology: descriptions and recommendations for research

References

  • Alkhaffaf, B., Blazeby, J. M., Metryka, A., Glenny, A.-M., Adeyeye, A., Costa, P. M., del Val, I. D., Gisbertz, S. S., Guner, A., Law, S., Lee, H.-J., Li, Z. Y., Nakada, K., Nuñez, R. M. R., Reim, D., Reynolds, J. V., Vorwald, P., Zanotti, D., Allum, W., . . . Bruce, I. A. (2021). Methods for conducting international Delphi surveys to optimise global participation in core outcome set development: A case study in gastric cancer informed by a comprehensive literature review. Trials, 22(1), Article 410. https://doi.org/10.1186/s13063-021-05338-x
  • Armstrong, K., Nadim, H., Olson, D., & Stutzman, S. (2021). Use of modified Delphi introduces the risk of chronological bias during clinical research interventions. Nurse Researcher, 29(1), 9-13. https://doi.org/10.7748/nr.2020.e1742
  • Banno, M., Tsujimoto, Y., & Kataoka, Y. (2020). The majority of reporting guidelines are not developed with the Delphi method: A systematic review of reporting guidelines. Journal of Clinical Epidemiology, 124, 50-57. https://doi.org/10.1016/j.jclinepi.2020.04.010
  • Barrington, H., Young, B., & Williamson, P. R. (2021). Patient participation in Delphi surveys to develop core outcome sets: Systematic review. BMJ Open, 11(9), Article e051066. https://doi.org/10.1136/bmjopen-2021-051066
  • Bellali, T., & Karamitri, I. (2011). The Delphi research methodology and its applications in the healthcare sciences. Archives of Hellenic Medicine, 28(6), 839-848.
  • Berian, J. R., Baker, T. L., Rosenthal, R. A., Coleman, J., Finlayson, E., Katlic, M. R., Lagoo-Deenadayalan, S. A., Tang, V. L., Robinson, T. N., Ko, C. Y., & Russell, M. M. (2018). Application of the RAND-UCLA appropriateness methodology to a large multidisciplinary stakeholder group evaluating the validity and feasibility of patient-centered standards in geriatric surgery. Health Services Research, 53(5), 3350-3372. https://doi.org/10.1111/1475-6773.12850
  • Biggane, A. M., Williamson, P. R., Ravaud, P., & Young, B. (2019). Participating in core outcome set development via Delphi surveys: Qualitative interviews provide pointers to inform guidance. BMJ Open, 9(11), Article e032338. https://doi.org/10.1136/bmjopen-2019-032338
  • Blazey, P., Scott, A., Ardern, C. L., Davis, J. C., Whittaker, J. L., Losciale, J. M., & Khan, K. M. (2024). Consensus methods in patellofemoral pain: How rigorous are they? A scoping review. British Journal of Sports Medicine, 58(13), 733-744. https://doi.org/10.1136/bjsports-2023-107552
  • Bossler, A., Fullana, A., & Bas, E. (2026). When synthetic Delphi panels work: A graph-native methodology for participatory foresight under structural complexity. European Journal of Futures Research, 14(1), Article 38. https://doi.org/10.1186/s40309-026-00302-x
  • Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011). Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLOS ONE, 6(6), Article e20476. https://doi.org/10.1371/journal.pone.0020476
  • Brewer, C. F., Chase, T. J. G., & Donnely, E. (2026). Guidance on transparent sponsorship reporting in industry associated Delphi studies (TRIAD): Recommendations based on a systematic review. Therapeutic Innovation & Regulatory Science, 60(6), 1601-1611. https://doi.org/10.1007/s43441-026-01015-w
  • Cary, M. A., Plamondon, K., Banner-Lukaris, D., Oelke, N., Sibley, K. M., Baxter, K., Vis-Dunbar, M., Hoens, A. M., Wick, U., Bigsby, S., Wuerstl, K., & Gainforth, H. (2023). Building consensus in research partnerships: A scoping review of consensus methods. Evidence & Policy, 19(3), 485-511. https://doi.org/10.1332/174426421X16645354235140
  • Chakravarti, A. K., Vasanta, B., Krishnan, A. S. A., & Dubash, R. K. (1998). Modified Delphi methodology for technology forecasting: Case study of electronics and information technology in India. Technological Forecasting and Social Change, 58(1-2), 155-165. https://doi.org/10.1016/S0040-1625(97)00081-4
  • Dayé, C. (2018). How to train your oracle: The Delphi method and its turbulent youth in operations research and the policy sciences. Social Studies of Science, 48(6), 846-868. https://doi.org/10.1177/0306312718798497
  • de Loë, R. C., Melnychuk, N., Murray, D., & Plummer, R. (2016). Advancing the state of policy Delphi practice: A systematic review evaluating methodological evolution, innovation, and opportunities. Technological Forecasting and Social Change, 104, 78-88. https://doi.org/10.1016/j.techfore.2015.12.009
  • Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401-409. https://doi.org/10.1016/j.jclinepi.2013.12.002
  • Dietrich, T. J., Mascarenhas, V. V., Cerezal, L., Afonso, P. D., & Sudol-Szopinska, I. (2024). Overview for developing Delphi-based interdisciplinary consensus statements on imaging: Pros and cons. Journal of Ultrasonography, 24(97), 6. https://doi.org/10.15557/jou.2024.0015
  • Donohoe, H. M., & Needham, R. D. (2009). Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions. International Journal of Tourism Research, 11(5), 415-437. https://doi.org/10.1002/jtr.709
  • Evans, C. (1997). The use of consensus methods and expert panels in pharmacoeconomic studies: Practical applications and methodological shortcomings. PharmacoEconomics, 12(2), 121-129. https://doi.org/10.2165/00019053-199712020-00003
  • Fathullah, M. A., Subbarao, A., & Muthaiyah, S. (2023). Methodological investigation: Traditional and systematic reviews as preliminary findings for Delphi technique. International Journal of Qualitative Methods, 22, Article 16094069231190747. https://doi.org/10.1177/16094069231190747
  • Ferrazzi, P., Tagalik, S., Christie, P., Karetak, J., Baker, K., & Angalik, L. (2019). Aajiiqatigiingniq: An Inuit consensus methodology in qualitative health research. International Journal of Qualitative Methods, 18, Article 1609406919894796. https://doi.org/10.1177/1609406919894796
  • Flanagan, T., Ashmore, R., Banks, D., & MacInnes, D. (2016). The Delphi method: Methodological issues arising from a study examining factors influencing the publication or non-publication of mental health nursing research. Mental Health Review Journal, 21(2), 85-94. https://doi.org/10.1108/MHRJ-07-2015-0020
  • Fouad, Y., Dufour, J.-F., Zheng, M.-H., Bollipo, S., Desalegn, H., Gronbaek, H., & Gish, R. G. (2022). The NAFLD-MAFLD debate: Is there a consensus-on-consensus methodology? Liver International, 42(4), 742-748. https://doi.org/10.1111/liv.15197
  • Freitas, C. K., & Ferrarini, A. V. (2021). Delphi as a methodological alternative for research in social sciences: An experience on collaborative consumption. Humanidades & Inovação, 8(49), 46-62.
  • Frewer, L. J., Fischer, A. R. H., Wentholt, M. T. A., Marvin, H. J. P., Ooms, B. W., Coles, D., & Rowe, G. (2011). The use of Delphi methodology in agrifood policy development: Some lessons learned. Technological Forecasting and Social Change, 78(9), 1514-1525. https://doi.org/10.1016/j.techfore.2011.05.005
  • Gattrell, W. T., Logullo, P., van Zuuren, E. J., Price, A., Hughes, E. L., Blazey, P., Winchester, C. C., Tovey, D., Goldman, K., Hungin, A. P., & Harrison, N. (2024). ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLOS Medicine, 21(1), Article e1004326. https://doi.org/10.1371/journal.pmed.1004326
  • Gnatzy, T., Warth, J., von der Gracht, H., & Darkow, I.-L. (2011). Validating an innovative real-time Delphi approach: A methodological comparison between real-time and conventional Delphi studies. Technological Forecasting and Social Change, 78(9), 1681-1694. https://doi.org/10.1016/j.techfore.2011.04.006
  • Goodman, C. M. (1987). The Delphi technique: A critique. Journal of Advanced Nursing, 12(6), 729-734. https://doi.org/10.1111/j.1365-2648.1987.tb01376.x
  • Gottlieb, M., Caretta-Weyer, H., Chan, T. M., & Humphrey-Murto, S. (2023). Educator’s blueprint: A primer on consensus methods in medical education research. AEM Education and Training, 7(4), Article e10891. https://doi.org/10.1002/aet2.10891
  • Grupp, T. M., Rusch, S., Massin, P., Blom, A., Garcia-Rey, E., Cristofolini, L., Janssen, D., Grimm, B., Giurea, A., Jaeger, M., Siccardi, F., & Overgaard, S. (2023). 1st EFORT European Consensus “Medical & Scientific Research Requirements for the Clinical Introduction of Artificial Joint Arthroplasty Devices”: Background, Delphi methodology & consensus process. EFORT Open Reviews, 8(7), 499-508. https://doi.org/10.1530/EOR-23-0054
  • Guzys, D., Dickson-Swift, V., Kenny, A., & Threlkeld, G. (2015). Gadamerian philosophical hermeneutics as a useful methodological framework for the Delphi technique. International Journal of Qualitative Studies on Health and Well-Being, 10, Article 26291. https://doi.org/10.3402/qhw.v10.26291
  • Haas, M., Rinderknecht, E., May, M., Burger, M., Gilfrich, C., Herrmann, A., Brookman-May, S., Eckl, C. A., Kruschwitz, U., Ateia, S., & Donabauer, G. (2026). From divergent guidelines to Delphi-ready items: Tutorial for large language model-assisted guideline harmonization using penile cancer as a worked example. Journal of Medical Internet Research, 28, Article e96886. https://doi.org/10.2196/96886
  • Harb, S. I., Tao, L., Peláez, S., Boruff, J., Rice, D. B., & Shrier, I. (2021). Methodological options of the nominal group technique for survey item elicitation in health research: A scoping review. Journal of Clinical Epidemiology, 139, 140-148. https://doi.org/10.1016/j.jclinepi.2021.08.008
  • Hasson, F., & Keeney, S. (2011). Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change, 78(9), 1695-1704. https://doi.org/10.1016/j.techfore.2011.04.005
  • Hung, H.-L., Altschuld, J. W., & Lee, Y.-F. (2008). Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. Evaluation and Program Planning, 31(2), 191-198. https://doi.org/10.1016/j.evalprogplan.2008.02.005
  • Jandhyala, R. (2020). Delphi, non-RAND modified Delphi, RAND/UCLA appropriateness method and a novel group awareness and consensus methodology for consensus measurement: A systematic literature review. Current Medical Research and Opinion, 36(11), 1873-1887. https://doi.org/10.1080/03007995.2020.1816946
  • Jarrott, S. E., Ogletree, A. M., Turner, S. G., & Demetres, M. (2026). Best practice implementation in gerontology and geriatrics Delphi studies: A scoping review. International Psychogeriatrics, 38(5), Article 100248. https://doi.org/10.1016/j.inpsyc.2026.100248
  • Jenkins, D. A., & Smith, T. E. (1994). Applying Delphi methodology in family-therapy research. Contemporary Family Therapy, 16(5), 411-430. https://doi.org/10.1007/BF02197902
  • Jünger, S., Payne, S. A., Brine, J., Radbruch, L., & Brearley, S. G. (2017). Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative Medicine, 31(8), 684-706. https://doi.org/10.1177/0269216317690685
  • Keeney, S., Hasson, F., & McKenna, H. P. (2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38(2), 195-200. https://doi.org/10.1016/S0020-7489(00)00044-4
  • Landeta, J., Barrutia, J., & Lertxundi, A. (2011). Hybrid Delphi: A methodology to facilitate contribution from experts in professional contexts. Technological Forecasting and Social Change, 78(9), 1629-1641. https://doi.org/10.1016/j.techfore.2011.03.009
  • Lee, A., Davies, A., & Young, A. E. (2020). Systematic review of international Delphi surveys for core outcome set development: Representation of international patients. BMJ Open, 10(11), Article e040223. https://doi.org/10.1136/bmjopen-2020-040223
  • Li, Z., Tobe, S., Sherifali, D., Racey, M., Hua-Stewart, D., Gravel, C. A., Lewis, M., Mehta, S., Hill, M. D., Haider, N., Lindsay, P., Liu, P., & Edwards, J. D. (2026). Consensus methodology for the Canadian brain and heart clinical practice guidelines. CJC Open, 8(3), 336-343. https://doi.org/10.1016/j.cjco.2025.11.018
  • Liang, H.-J., Xiong, Q., & Preston, N. (2025). Methodological reflections to support good practice in using nominal group techniques: Insights from applications in palliative care studies. Palliative Medicine, 39(10), 1099-1108. https://doi.org/10.1177/02692163251368974
  • Loblaw, D. A., Prestrud, A. A., Somerfield, M. R., Oliver, T. K., Brouwers, M. C., Nam, R. K., Lyman, G. H., & Basch, E. (2012). American Society of Clinical Oncology clinical practice guidelines: Formal systematic review-based consensus methodology. Journal of Clinical Oncology, 30(25), 3136-3140. https://doi.org/10.1200/JCO.2012.42.0489
  • Logullo, P., van Zuuren, E. J., Winchester, C. C., Tovey, D., Gattrell, W. T., Price, A., Harrison, N., Goldman, K., Chisholm, A., Walters, K., & Blazey, P. (2024). ACcurate COnsensus Reporting Document (ACCORD) explanation and elaboration: Guidance and examples to support reporting consensus methods. PLOS Medicine, 21(5), Article e1004390. https://doi.org/10.1371/journal.pmed.1004390
  • López-Gómez, E. (2018). The Delphi method in current educational research: A theoretical and methodological review. Educación XX1, 21(1), 17-40. https://doi.org/10.5944/educXX1.15536
  • Lurvink, R. J., Villeneuve, L., Govaerts, K., de Hingh, I. H. J. T., Moran, B. J., Deraco, M., Van der Speeten, K., Glehen, O., Kepenekian, V., & Kusamura, S. (2021). The Delphi and GRADE methodology used in the PSOGI 2018 consensus statement on pseudomyxoma peritonei and peritoneal mesothelioma. EJSO, 47(1), 4-10. https://doi.org/10.1016/j.ejso.2019.03.012
  • MacFarlane, I. M., & Zierhut, H. (2025). An introduction to Delphi studies and consensus methods for genetic counselors. Journal of Genetic Counseling, 34(6), Article e70136. https://doi.org/10.1002/jgc4.70136
  • Maskrey, N., & Underhill, J. (2014). The European Statements of Hospital Pharmacy: Achieving consensus using Delphi and World Cafe methodologies. European Journal of Hospital Pharmacy: Science and Practice, 21(5), 264-266. https://doi.org/10.1136/ejhpharm-2014-000520
  • McMillan, S. S., Kelly, F., Sav, A., Kendall, E., King, M. A., Whitty, J. A., & Wheeler, A. J. (2014). Using the nominal group technique: How to analyse across multiple groups. Health Services and Outcomes Research Methodology, 14(3), 92-108. https://doi.org/10.1007/s10742-014-0121-1
  • McMillan, S. S., King, M., & Tully, M. P. (2016). How to use the nominal group and Delphi techniques. International Journal of Clinical Pharmacy, 38(3), 655-662. https://doi.org/10.1007/s11096-016-0257-x
  • McPherson, S., Reese, C., & Wendler, M. C. (2018). Methodology update: Delphi studies. Nursing Research, 67(5), 404-410. https://doi.org/10.1097/NNR.0000000000000297
  • Melander, L. (2018). Scenario development in transport studies: Methodological considerations and reflections on Delphi studies. Futures, 96, 68-78. https://doi.org/10.1016/j.futures.2017.11.007
  • Murry, J. W., & Hammons, J. O. (1995). Delphi: A versatile methodology for conducting qualitative research. The Review of Higher Education, 18(4), 423-436. https://doi.org/10.1353/rhe.1995.0008
  • Nasa, P., Jain, R., Juneja, D., Ahmed, Z., Suresh, S., Renukappa, S., Chacko, C. J., Schultz, M. J., & Veenith, T. (2026). Causes of bias in Delphi studies and consensus statements: A methodological review. Anaesthesia. Advance online publication. https://doi.org/10.1111/anae.70415
  • Nathan, M., Newburger, J. W., Bell, M., Tang, A., Gongwer, R., Dunbar-Masterson, C., Atz, A. M., Bacha, E., Colan, S., Gaynor, J. W., Kanter, K., Levine, J. C., Ohye, R., Pizarro, C., Schwartz, S., Shirali, G., Tani, L., Tweddell, J., & Gurvitz, M. (2023). Development of the Residual Lesion Score for congenital heart surgery: The RAND Delphi methodology. Cardiology in the Young, 33(6), 977-990. https://doi.org/10.1017/S1047951122003791
  • Niederberger, M., & Deckert, S. (2022). The Delphi technique: Methodology, variants and usage examples. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, 174, 11-19. https://doi.org/10.1016/j.zefq.2022.08.007
  • Normand, S. L. T., McNeil, B. J., Peterson, L. E., & Palmer, R. H. (1998). Methodology matters VIII: Eliciting expert opinion using the Delphi technique: Identifying performance indicators for cardiovascular disease. International Journal for Quality in Health Care, 10(3), 247-260. https://doi.org/10.1093/intqhc/10.3.247
  • Nowak, P. M., Wozniakiewicz, M., Wozniakiewicz, A., Brindell, M., Poznanska, A., Plewka, J., Bis, A., & Nowak, J. (2026). Can artificial intelligence assist experts in evaluating new analytical methodologies? Stop-flow MicroScale Thermophoresis assessed using the RGB model and AI-Delphi framework. Green Analytical Chemistry, 16, Article 100318. https://doi.org/10.1016/j.greeac.2025.100318
  • Nyström, A.-G., & Kaartemo, V. (2022). Developing Delphi methodology for studying future market change. Journal of Business & Industrial Marketing, 37(13), 124-141. https://doi.org/10.1108/JBIM-11-2021-0520
  • O’Donnell, L., Johal, P. K., Mauer-Vakil, D., Ziegler, C., Morton, E., Banner, D., Barnes, S. J., & Michalak, E. E. (2025). Consensus building in bipolar disorder: A scoping review of consensus methods. Journal of Affective Disorders, 385, Article 119339. https://doi.org/10.1016/j.jad.2025.04.170
  • O’Hagan, E., Sheahen, B., Cho, K. N., Turnbull, S., Wong, K. C., Rock, Z., & Chow, C. K. (2026). Integrating real-time polling into modified Delphi methodology: Lessons from a consensus study on cardiac implantable electronic device remote monitoring. BMC Medical Research Methodology, 26(1), Article 73. https://doi.org/10.1186/s12874-026-02807-2
  • Olsen, A. A., Wolcott, M. D., Haines, S. T., Janke, K. K., & McLaughlin, J. E. (2021). How to use the Delphi method to aid in decision making and build consensus in pharmacy education. Currents in Pharmacy Teaching and Learning, 13(10), 1376-1385. https://doi.org/10.1016/j.cptl.2021.07.018
  • Paré, G., Cameron, A.-F., Poba-Nzaou, P., & Templier, M. (2013). A systematic assessment of rigor in information systems ranking-type Delphi studies. Information & Management, 50(5), 207-217. https://doi.org/10.1016/j.im.2013.03.003
  • Piantedosi, D. K., & O’Shea, A. (2025). The role of people with intellectual disability in intellectual disability research: A systematic review of Delphi studies. Journal of Intellectual Disabilities, 29(3), 727-742. https://doi.org/10.1177/17446295231225272
  • Quartiroli, A. (2025). The Delphi technique in sport, exercise, and performance psychology: Extensive scoping review, insights, and recommendations for scholars. Sport, Exercise, and Performance Psychology, 14(1), 57-77. https://doi.org/10.1037/spy0000364
  • Rao, G. K. L., Iskandar, Y. H. P., & Mokhtar, N. (2021). Revisiting the Delphi technique in orthodontics: A systematic review. Journal of Clinical and Diagnostic Research, 15(9), ZE01-ZE05. https://doi.org/10.7860/JCDR/2021/49628.15338
  • Remus, A., Smith, V., & Wuytack, F. (2021). Methodology in core outcome set (COS) development: The impact of patient interviews and using a 5-point versus a 9-point Delphi rating scale on core outcome selection in a COS development study. BMC Medical Research Methodology, 21(1), Article 10. https://doi.org/10.1186/s12874-020-01197-3
  • Roldán López de Hierro, A. F., Sánchez, M., Puente-Fernández, D., Montoya-Juárez, R., & Roldán, C. (2021). A fuzzy Delphi consensus methodology based on a fuzzy ranking. Mathematics, 9(18), Article 2323. https://doi.org/10.3390/math9182323
  • Rowe, G., & Wright, G. (2011). The Delphi technique: Past, present, and future prospects: Introduction to the special issue. Technological Forecasting and Social Change, 78(9), 1487-1490. https://doi.org/10.1016/j.techfore.2011.09.002
  • Rüetschi, U., & Olarte Salazar, C. M. (2022). Methodology of an e-Delphi study to explore future trends in orthopedics education and the role of technology in orthopedic surgeon learning. Journal of Musculoskeletal Surgery and Research, 6(2), 121-137. https://doi.org/10.25259/JMSR_134_2021
  • Sakamoto, F. (2024). Improving Delphi study rigour with the integration of Q-methodology. International Journal of Research & Method in Education, 47(2), 126-139. https://doi.org/10.1080/1743727X.2023.2270432
  • Schifano, J., & Niederberger, M. (2025). How Delphi studies in the health sciences find consensus: A scoping review. Systematic Reviews, 14(1), Article 14. https://doi.org/10.1186/s13643-024-02738-3
  • Sokolov, A., Grebenyuk, A., & Urashima, K. (2025). Biases in expert judgements in large-scale S&T Delphi surveys: How to cope with them? Technological Forecasting and Social Change, 218, Article 124223. https://doi.org/10.1016/j.techfore.2025.124223
  • Sparks, J. B., Klamerus, M. L., Caverly, T. J., Skurla, S. E., Hofer, T. P., Kerr, E. A., Bernstein, S. J., & Damschroder, L. J. (2022). Planning and reporting effective web-based RAND/UCLA appropriateness method panels: Literature review and preliminary recommendations. Journal of Medical Internet Research, 24(8), Article e33898. https://doi.org/10.2196/33898
  • Spranger, J., Homberg, A., Sonnberger, M., & Niederberger, M. (2022). Reporting guidelines for Delphi techniques in health sciences: A methodological review. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, 172, 1-11. https://doi.org/10.1016/j.zefq.2022.04.025
  • Sterling, S., Plonsky, L., Larsson, T., Kytö, M., & Yaw, K. (2023). Introducing and illustrating the Delphi method for applied linguistics research. Research Methods in Applied Linguistics, 2(1), Article 100040. https://doi.org/10.1016/j.rmal.2022.100040
  • Strekalova, Y. A. L., Nelson, J. D., Weber, H. M., Wang, X. R., & Midence, S. M. (2024). Application of the Delphi method to the development of common data elements for social drivers of health: A systematic scoping review. Translational Behavioral Medicine, 14(7), 426-433. https://doi.org/10.1093/tbm/ibae020
  • Toillier, A., Mathé, S., Saley Moussa, A., & Faure, G. (2022). How to assess agricultural innovation systems in a transformation perspective: A Delphi consensus study. Journal of Agricultural Education & Extension, 28(2), 163-185. https://doi.org/10.1080/1389224X.2021.1953548
  • Trevelyan, E. G., & Robinson, N. (2015). Delphi methodology in health research: How to do it? European Journal of Integrative Medicine, 7(4), 423-428. https://doi.org/10.1016/j.eujim.2015.07.002
  • van Zuuren, E. J., Logullo, P., Price, A., Fedorowicz, Z., Hughes, E. L., & Gattrell, W. T. (2022). Existing guidance on reporting of consensus methodology: A systematic review to inform ACCORD guideline development. BMJ Open, 12(9), Article e065154. https://doi.org/10.1136/bmjopen-2022-065154
  • Varndell, W., Fry, M., Lutze, M., & Elliott, D. (2021). Use of the Delphi method to generate guidance in emergency nursing practice: A systematic review. International Emergency Nursing, 56, Article 100867. https://doi.org/10.1016/j.ienj.2020.100867
  • Waggoner, J., Carline, J. D., & Durning, S. J. (2016). Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Academic Medicine, 91(5), 663-668. https://doi.org/10.1097/ACM.0000000000001092
  • Webbe, J., Allin, B., Knight, M., Modi, N., & Gale, C. (2023). How to reach agreement: The impact of different analytical approaches to Delphi process results in core outcomes set development. Trials, 24(1), Article 345. https://doi.org/10.1186/s13063-023-07285-1
  • Williams, P. L., & Webb, C. (1994). The Delphi technique: A methodological discussion. Journal of Advanced Nursing, 19(1), 180-186. https://doi.org/10.1111/j.1365-2648.1994.tb01066.x
  • Wodnik, B. K., Namyalo, P. K., Michaelides, O., Essue, B. M., Kane, S., & Di Ruggiero, E. (2025). Implementation science research priorities for universal health coverage: Methodological lessons from the design and implementation of a multicountry modified Delphi study. Health Policy and Planning, 40(3), 422-427. https://doi.org/10.1093/heapol/czae119

Source verification and use of AI. Author names, publication years, titles, journals, volumes, issues, pages and DOIs in the reference list were taken from the Web of Science records of the included studies, with author names taken from the full-name (AF) field; journal names were normalised from the capitalised Web of Science form, and titles were set in sentence case. Author names were corrected only for obvious export errors (transliterated diacritics and an academic title exported as a given name). Claims attributed to sources were checked against the abstracts in an independent verification at abstract level; full texts were not reviewed, so numbers are reported as the abstracts state them. AI tools were used to support title and abstract screening, to extract evidence from abstracts into a structured template, to draft the text and to generate the figures. Responsibility for the content rests with the author, Prof. Dr. Mehmet Nurullah Kurutkan.

Data and supplementary material. The full list of the 117 included records, the screening log and the verification report are provided in the supplementary file “DE_101_supplementary.xlsx”, available from the author on request.

Cite this page (APA 7)

Kurutkan, M. N. (2026, November 19). The Delphi Technique and Consensus Methods: An Introduction (101). The Health Topics. https://healthtopic.org/topics/delphi-technique-and-consensus-methods-an-introduction-101/

Subscribe to the Health Topics Newsletter!

Google reCaptcha: Invalid site key.