Are you looking to enhance the quality of your research submissions and increase your chances of publication in top-tier journals like the Academy of Management Journal (AMJ)? This insightful editorial, “PUBLISHING IN AMJ—PART 2: RESEARCH DESIGN,” by Joyce E. Bono and Gerry McNamara, offers critical advice on avoiding common research design pitfalls that frequently lead to manuscript rejection.
As part of a seven-part series on “Publishing in AMJ,” this installment provides essential guidance, moving beyond basic research methodology to address practical design issues observed in rejected manuscripts. The authors emphasize that choosing the appropriate research design is paramount because it directly impacts the confidence readers can place in your conclusions, the strength of your arguments, and your ability to discount alternative explanations. Unlike other aspects, the fundamental design of a study cannot be altered during the revision process, making upfront attention crucial.
The editorial highlights three broad design problems that are frequent sources of rejection:
- Mismatch Between Research Question and Design:
- Cross-sectional data are often unsuitable for research questions that implicitly or explicitly deal with causality or change. For strong causal inferences or to establish change, longitudinal, panel, or experimental data are necessary.
- Inappropriate samples and procedures can undermine a study. It’s not about certain samples (e.g., students vs. executives) or procedures being inherently better, but rather about ensuring they are carefully matched to your specific research question. AMJ encourages experimental research as an excellent way to address causality.
- Measurement and Operationalization Issues (Construct Validity):
- Researchers must clearly define new constructs and map their associations with existing ones to avoid confusion and “jingle jangle fallacies”.
- Inappropriate adaptation or application of existing measures (e.g., selectively picking items from scales without validity evidence, or using measures for completely different constructs) raises significant red flags. Authors must demonstrate the uniqueness and validity of their operationalization.
- Common method variance is a serious threat, particularly in cross-sectional studies where multiple variables are assessed via a single method, potentially biasing observed correlations.
- Inappropriate or Incomplete Model Specification:
- Careful attention to control variables and mediating processes during the design stage is vital. Control variables should meet specific conditions: correlated with both dependent and independent variables, and not a central study variable.
- Operationalizing mediators is critical for theory testing, especially as research areas mature. Failing to assess known, conceptually related mediators can make it difficult to establish a novel contribution.
The authors conclude by noting that while compromising in research is a reality, the core problem for rejected AMJ submissions is making too many compromises at the design stage. They advocate for a pragmatic approach: assume single-study designs may have flaws, and consider conducting a series of studies—each addressing the flaws of the others—to achieve stronger inferences and more generalizable results. This “multiple study and multiple sample” approach is vastly underutilized and encouraged by AMJ.
By honestly assessing and minimizing threats to validity during the design phase, researchers can significantly improve the potential for a positive outcome for their manuscripts.
Reference for this Article:
Bono, J. E., & McNamara, G. (2011). FROM THE EDITORS: PUBLISHING IN “AMJ”—PART 2: RESEARCH DESIGN. The Academy of Management Journal, 54(4), 657–660. https://www.jstor.org/stable/23045105
