Advantages of Starting Research with Theory

This editorial, “From the Editors: Advantages of Starting with Theory” by Jason D. Shaw, published in The Academy of Management Journal, critically examines the pervasive practice of “HARKing” (hypothesizing after the results are known) within deductive hypothesis-driven quantitative research in the field of management. Shaw, as the editor-in-chief of The Academy of Management Journal (AMJ), aims to discuss the likely outcomes of post-result theorizing from the perspective of the journal’s review process, rather than focusing on ethical concerns or inductive theory building.

The author describes HARKing as a process where a researcher obtains a dataset, often with only a general or no specific research question, and then scours the data for unanticipated significant associations. This can involve looking for findings that deviate from conventional wisdom or running numerous models to identify signs of moderation, mediation, or both. Once “novel” and “significant” findings are identified, the researcher then constructs a narrative by searching for applicable theory to frame these findings as if they were predicted a priori using a hypothetico-deductive approach. This practice is common, with surveys indicating that about a third of psychology authors admit to it, and Shaw’s experience suggests it is prevalent in the management community as well.

While some authors might perceive HARKing as a “success-facilitating practice”, Shaw argues from his extensive editorial experience that it is generally a “rejection-creating” practice. He identifies four “telltale signs” of post-result theorizing that often lead to unfavorable commentary and rejection during the review process for quantitative papers:

1. Contorted Theory: This occurs when the theory itself is unfit to support the predictions being made, or authors fail to adequately use the logic, assumptions, and central tenets of the evoked theoretical perspective. It also includes invoking multiple disparate theoretical perspectives to justify various predictions, a phenomenon sometimes referred to as the “too-convenient qualifier”.
2. Poorly Defined Constructs: Retrofitting results to theory can lead to conceptual sloppiness and a lack of precision in defining constructs, which can be easily identified by reviewers with strong theoretical knowledge.
3. Construct-Measurement Mismatch: There is often a disconnect between the constructs in the theoretical model and their actual operationalizations in the study, as variables might have been added, deleted, or modified during the search for a good-fitting empirical model.
4. Theory-Design Mismatch: The study design or available measurements may not adequately capture the mechanisms suggested by the evoked theoretical framework, leading reviewers to suspect that the authors opted for a model that “worked” rather than one logically derived from theory. This often results in a Discussion section that merely restates findings instead of delving into theoretical implications.

Conversely, Shaw advocates for “starting with theory”, highlighting its significant advantages for authors in the hypothetico-deductive genre. This approach leads to less contorted and more coherent predictions, allows for more refined and accurate construct definitions, ensures measurements align with theoretical constructs, and results in a study design capable of properly testing underlying theoretical mechanisms.

Beyond these direct benefits, Shaw identifies two subtler advantages:

1. Clarifying Theoretical Contribution: A strong theoretical frame from the outset helps authors identify and articulate their unique theoretical contribution, enabling them to produce novel and interesting work that challenges, changes, or advances existing theoretical understanding.
2. Improving the Discussion Section: With a clear theoretical foundation, authors can more effectively offer thoughtful reflections on their study’s contribution to theory, articulating how it fundamentally refines understanding of extant theory and its core concepts.

Shaw acknowledges that there are specific situations where post-result theorizing can have a place at AMJ. Unexpected findings can serve as a starting point for further specific deductive theory development and follow-up quantitative studies. Additionally, authors can engage in “tharking”, which involves transparently discussing alternative results discovered in exploratory analyses within a dedicated section of additional findings in the Discussion, potentially enriching the paper and inspiring future theory innovations.

Ultimately, Shaw encourages authors of quantitative papers to carefully consider the advantages of a theory-first approach as an alternative to results-driven retrospective theorizing. He concludes that strong conceptual framing, proper study execution, conceptual clarity, construct-measure matching, and appropriate design features for testing theory will reduce rejection-creating commentary and contribute to higher quality theorizing and minimized biases in the literature.

Why post-result theorising backfires

Jason Shaw, then editor of AMJ, addresses HARKing (hypothesising after the results are known) in deductive, hypothesis-testing research. In the typical story, a researcher scans a correlation matrix or runs many models in search of significant or surprising effects, then searches for a theory that can be presented as if it had predicted them. Surveys suggest that about a third of psychology researchers admit to the practice. Some commentators describe it as a “success-facilitating” strategy; Shaw argues the opposite, calling it a “rejection-creating” practice in the review process.

Warning signs reviewers notice

Contorted theory. A broad theoretical perspective is invoked but its logic and assumptions do not actually drive the predictions, or several unrelated theories are borrowed to justify different hypotheses, including arbitrary moderators that appear “out of the blue”. Poorly defined constructs. Constructs are loosely specified because they were chosen to fit available variables. Construct-measurement mismatch. Variables added, dropped or modified during model searching fit poorly with the concepts in the retrofitted theory, which expert reviewers readily notice. Theory-design mismatch. The study lacks measures of the mechanisms the theory implies, or has non-optimal treatments and missing controls, suggesting the authors moved to a model that “worked” rather than the one the theory suggested.

The advantages of starting with theory

Shaw argues that authentic, a priori theorising produces a more coherent set of predictions, helps authors see which parts of existing theory are well understood and which are contested, and shows where a study can extend theory or offer bold alternatives. It also leads to stronger Discussion sections: when a study starts from a clear theoretical position, authors can explain precisely what it contributes to theory, instead of offering broad statements about a topic or restating results.

Relevance for health management researchers

Health management studies often use large administrative or survey datasets that invite exploratory model searching. Shaw’s editorial is a reminder that such exploration should be reported honestly as exploratory, and that confirmatory papers are judged on whether the theory genuinely came first.

Reference: Shaw, J. D. (2017). From the editors: Advantages of starting with theory. The Academy of Management Journal, 60(3), 819–822. https://doi.org/10.5465/amj.2017.4003