The article, “Methodologies for Improving Response Rates in Surveys of Physicians: A Systematic Review,” by Jonathan B. VanGeest, Timothy P. Johnson, and Verna L. Welch (2007), provides an in-depth systematic review of strategies designed to enhance physician participation in surveys. This research is critically important because physician surveys are a valuable tool in health services and policy research, offering a cost-effective means of gathering information on physicians’ attitudes, knowledge, and practices related to care delivery. These surveys have been utilized to assess a wide array of topics, from routine subjects like compliance with evidence-based practice recommendations to highly sensitive issues such as substance abuse among physicians or attitudes toward euthanasia.
Despite their utility, physician surveys frequently suffer from low response rates, which raises significant concerns about the validity and generalizability of their findings, primarily due to the potential for nonresponse bias. This bias occurs when nonresponding physicians differ systematically from the studied population, a concern supported by research showing differences between responders and nonresponders on demographic or practice-related characteristics. Physicians’ reluctance to participate is attributed to several key factors:
- Lack of time: Physicians are busy, and completing surveys takes away from patient care or other important tasks. This is compounded by the increasing volume and length of surveys they receive.
- Perceived low salience or value: Physicians are less likely to participate if the study’s value is unclear or deemed low.
- Concerns about confidentiality: Worries about the privacy of results can deter participation.
- Potentially biased questions: Surveys with questions that appear biased or limit response choices may lead to nonresponse.
- Private practice office settings: “Gatekeepers” in these settings can also act as a barrier to participation.
To address these challenges, the authors conducted a systematic review of 66 published reports (from 1975 to 2006) focusing on two main categories of strategies: incentive-based interventions and design-based approaches. The methodology involved keyword searches across databases like MEDLINE, Scopus, Sociological Abstracts, and PsychINFO, as well as manual referencing and calculating odds ratios (ORs) to measure effect sizes for identified interventions.
The review yielded several key findings regarding effective strategies:
I. Incentive-Based Interventions:
- Monetary Incentives:
- Even small financial incentives are remarkably effective in improving physician response rates. The weighted overall effect size showed a significant association between monetary incentives and physician response (OR 2.13; 95% CI 1.7–2.6).
- Modest $1 incentives were associated with significantly higher response rates (average OR 2.11) compared to no incentive.
- However, the evidence suggests diminishing returns for serial increments above $1. For example, studies found little difference in effectiveness between $1 and larger amounts.
- Cash payments are more effective than other forms of inducements, such as charity donations, monetary donations to alma maters, nonmonetary incentives, or opportunities to win cash lottery prizes.
- Prepaid monetary incentives are superior to promised incentives. The weighted overall effect size for prepaid versus promised incentives was OR 1.82 (95% CI 1.6–2.1).
- Nonmonetary Incentives:
- Generally, token nonmonetary incentives proved much less effective and often had little or no impact on response rates (weighted overall effect size OR 0.97; 95% CI 0.82–1.14). Examples of ineffective tokens include stickers, pencils (except in one inconsistent case in a second mailing), pens, informational brochures, risk-assessment computer programs, and candy.
- Nonmonetary inducements only work if physicians highly value them. For instance, continuing medical education (CME) credits were effective when offered in conjunction with a small monetary incentive (e.g., $5), although their independent effect was not definitively determined and consistency varied across studies.
- More substantial nonfinancial inducements, such as prize draws for a weekend trip, resulted in a small but significant increase in response (OR 1.29; 95% CI 1.00–1.67), with larger prizes being more effective than many small ones.
II. Design-Based Approaches:
- Questionnaire Design:
- Length of Questionnaire: Shorter questionnaires consistently lead to higher cooperation rates. A simple average of ORs across relevant studies was 2.33, with a weighted overall effect size of OR 2.0 (95% CI 1.1–3.7). Even relatively small differences in length (e.g., under 1,000 words vs. more than 1,000 words) can impact response rates (OR 2.348).
- Format: The use of an attractive business letter format and standard-sized (8.5 in. × 11 in.) questionnaire booklets was associated with higher response rates. However, paper quality and single- versus double-sided printing did not show increased cooperation.
- Item Format: Closed-ended questionnaire formats resulted in a 22% higher cooperation rate compared to open-ended formats in one study, though caution is advised as professionals may be more resistant to closed-ended questions.
- Personalization and Sponsorship:
- Personalization: The inclusion of a personalized cover letter and/or handwritten notes significantly improved response rates (weighted overall effect size OR 1.51; 95% CI 1.1–2.2).
- Direct contact, such as prenotification calls/letters and follow-up contacts, also effectively improved physician response (weighted overall effect size OR 2.3; 95% CI 1.42–3.64). This includes contact by a medical peer, which has been shown to increase participation.
- Sponsorship: Endorsements by opinion leaders or professional associations generally improved participation. However, one study found that endorsement by expert surgeons could surprisingly lead to a lower response rate, suggesting “limits of leadership”.
- Type of Mailing:
- Initial Mailing: Using certified mail (OR 2.085) or courier services like FedEx (OR 1.444) for the initial mailing increased participation compared to first-class mail.
- Return Mailing: Studies consistently found that return envelopes with first-class stamps result in higher physician response compared to franked or business reply envelopes (weighted overall effect size OR 1.3; 95% CI 1.1–1.5).
- Follow-up Mailings: Including a replacement questionnaire with follow-up contact can improve participation.
- Overall, postage/mailing strategies had a weighted OR of 1.4 (95% CI 1.11–1.69).
- Survey Mode:
- Postal and telephone surveys were generally more successful than fax or Web-based approaches. While results were mixed when comparing mail vs. telephone alone, some studies indicated mail surveys had higher response rates, while others favored telephone interviews.
- E-mail surveys resulted in significantly lower physician response rates compared to mailed questionnaires.
- Fax technology, when integrated into a mixed-mode design, can be a cost-effective method to increase participation, with a notable percentage of physicians opting to respond via fax when given a choice.
- Internet-based surveys generally showed significantly lower response rates than traditional mail surveys, and raised methodological issues related to sample representativeness.
- Other Determinants: The number of contacts and length of field periods are crucial determinants, with lengthy field periods often necessary to maximize physician participation.
The authors conclude that researchers should consistently implement these documented design strategies to improve physician participation, thereby enhancing the legitimacy and credibility of their research outcomes. They note that while cost is a factor, the relatively homogeneous nature of physicians can sometimes lead to lower nonresponse bias than anticipated, suggesting that even with limited resources, strategic application of these methods can be effective.
Reference: VanGeest, J. B., Johnson, T. P., & Welch, V. L. (2007). Methodologies for improving response rates in surveys of physicians: A systematic review. Evaluation & the Health Professions, 30(4), 303-321. https://doi.org/10.1177/0163278707307899. P., & Welch, V. L. (2007). Methodologies for improving response rates in surveys of physicians: A systematic review. Evaluation & the Health Professions, 30(4), 303-321. https://doi.org/10.1177/0163278707307899