The article, “NEW WAYS OF SEEING: PITFALLS AND OPPORTUNITIES IN MULTILEVEL RESEARCH,” by Paruchuri, Perry-Smith, Chattopadhyay, and Shaw (2018), published in the Academy of Management Journal, addresses the significant rise and increasing importance of multilevel research within the management field. Once considered rare and often lamented in the literature (e.g., O’Reilly, 1990; Staw, 1984), studies that incorporate multilevel theories and models are now “replete” in management journals. Hitt, Beamish, Jackson, and Mathieu (2007) observed about a decade prior that roughly a quarter of management publications were multilevel, and this positive trend has undoubtedly continued.
This proliferation is not merely coincidental but reflects the field’s overarching desire to cultivate more comprehensive and context-rich theories and findings. Several factors have contributed to this movement, including the availability of practical “how-to” guides for developing multilevel theory and analyzing the associated data (e.g., Johns, 2001, 2006; Kozlowski & Klein, 2000), as well as the widespread accessibility of sophisticated statistical software packages. The authors emphasize that this shift is both symbolic and substantive, transforming the multilevel context from a “messy source of error variance” to frequently being “at the heart of theorizing on a variety of topics”. Its influence is particularly evident in the teams literature (Mathieu et al., 2008), where studies examine cross-level effects and contextual moderators, but also extends to areas like strategic human resource management, emotions, and social networks.
A critical contribution of multilevel research, as highlighted by the authors, is its role in helping management distinguish itself from foundational theories borrowed from basic disciplines such as psychology, sociology, and economics. These foundational theories often operate and analyze phenomena at single levels. By developing theories that span multiple organizational levels—such as individuals, teams, and departments—multilevel research generates unique insights into organizational phenomena that are central to understanding how organizations truly operate. These insights are less likely to be produced by researchers in other disciplines who typically theorize and analyze at single levels (Heath & Sitkin, 2001). Such research is particularly valuable for managers who must navigate the inherent complexities of organizing across different levels and require an understanding of how actions at one level influence outcomes at other levels.
Despite these clear successes and the opportunities they present, the article acknowledges that building theory across levels is inherently more complicated than developing single-level theory, requiring scholars to be attuned to a wider array of concerns. The core objective of this editorial is to summarize common pitfalls encountered when theorizing across levels and to offer practical strategies for avoiding them, thereby guiding scholars toward more effective and rigorous multilevel research.
The authors begin by defining three crucial aspects of multilevel research in organizational studies:
- Multilevel Theorizing: This involves identifying and explaining how factors originating from different levels (lower-level, same-level, and/or higher-level constructs) influence an outcome at a particular level. If at least one explanatory factor’s level differs from that of the outcome, it constitutes multilevel theorizing. The authors note that cross-level studies are considered a subset of multilevel studies.
- Multilevel Measurement: This aspect typically focuses on how the properties of a higher-level unit are derived from lower-level units. The article differentiates between:
- Global unit properties: These reside at the construct level and are measured at that same construct level (e.g., team size, firm sales). These are explicitly not considered multilevel measurements.
- Emergent unit properties: These reside at a higher construct level but are measured at a lower level. Kozlowski and Klein (2000) proposed two forms of emergent unit properties: “compositional” and “compilation” models, which describe different ways of aggregating lower-level unit properties to form higher-level construct properties.
- Multilevel Models: These are statistical techniques specifically designed to partition variance at multiple levels. Such models are necessary because data composed of multiple lower-level units nested within higher-level units violate the assumptions of ordinary least squares (OLS) regression. Accordingly, multilevel scholars utilize various techniques, including hierarchical linear modeling, random coefficients regression, mixed effects modeling, mixed determinants modeling, random-effects or fixed-effects modeling, and multilevel regression modeling.
The article then categorizes common pitfalls into two broad types: “failures to surface assumptions” and “misalignment of different components in a study”.
Under “failures to surface assumptions,” the authors detail three pitfalls:
- Pitfall #1: Poorly chosen levels. Organizational phenomena exist across many distinct levels (e.g., intra-individual, individual, team, organization). Choosing levels that are either too proximal (offering little new insight) or too distant (lacking meaningful influence without intermediate levels) can render research less impactful. To avoid this, researchers should develop an overarching premise grounded in theories from each level, fully exploring the logic. For instance, Joshi and Knight (2015) successfully theorized about group deference at the dyadic level, distinct from individual-level conceptualizations.
- Pitfall #2: Lack of clear definitions for the level of constructs. This pitfall involves failing to clearly articulate and justify the appropriate level for each construct. This is particularly common in methodologies like experience sampling, where the distinction between within-person (lower) and between-person (higher) levels is often not explicitly theorized. To avoid this, researchers must clearly articulate the conceptual level of each construct and justify why it is expected to vary (or remain stable) at that particular level. The authors also highlight an opportunity here, exemplified by Nishii, Lepak, and Schneider (2008), who made a theoretical contribution by conceptualizing individual-level variations in reactions to organizational HR practices, demonstrating that members of the same group can have different perceptions.
- Pitfall #3: Assuming homology. This occurs when theoretical logic developed for single-level studies is directly applied across different levels without considering that the underlying assumptions may not hold, or that the dynamics of a construct’s effect might differ across levels. Scholars should be sensitive to and explicitly reconcile the stated and unstated assumptions at each distinct level. Yu and Zellmer-Bruhn (2018), for example, clearly articulated the different processes through which team mindfulness impacts team conflict versus individual social undermining.
Under “misalignment of different components in a study,” the article discusses further pitfalls arising from a lack of alignment among the three multilevel components: theory, measurement, and analytical method:
- Pitfall #4: Misfit between theory and measurement of constructs. This common issue arises when studies theorize about multilevel constructs but only employ measures for higher-level constructs, effectively using a single-level measurement for multilevel theory. Furthermore, the aggregation of lower-level data to measure higher-level units is itself based on specific underlying theories (e.g., Chan, 1998, identified five such aggregation models, each with distinct theoretical assumptions). Researchers must explicitly ensure that their basis of aggregation aligns with their theorization, as a mismatch invalidates empirical testing and misses opportunities for deeper understanding.
- Pitfall #5: Misfit between theory and analytical methods. Multilevel data necessitates analytical methods that account for its multilevel nature. A misalignment occurs if single-level relationships are theorized but multilevel analytical methods are used, representing a “missed opportunity” for richer understanding through multilevel theorizing. Conversely, researchers sometimes employ implicit or “timid” multilevel theorizing but then use a single-level analytical approach, as seen in some studies applying Coleman’s boat model (Coleman, 1990) without empirically exploring the posited lower-level processes. This leads to missing opportunities to fully explore multilevel conceptual implications due to methodological constraints.
In conclusion, Paruchuri and colleagues emphasize that while multilevel approaches offer significant opportunities to enrich understanding of organizational phenomena, they “should be applied wisely”. Researchers must weigh the perils and pitfalls against the value gained from increased complexity, recognizing that simpler, single-level explanations may sometimes be more parsimonious. The authors caution that if a multilevel approach does not inform the research domain or lacks a theoretically justified reason for its data structure and analyses, efforts might be better spent elsewhere. They stress that effective multilevel theorizing must stem from theoretically sound justifications and address a “compelling conceptual problem or puzzle” unique to the nesting, timing, or level of analysis. The editorial serves as both a “handy reference list” of issues to consider and a “list of opportunities for future multilevel research,” viewing every identified shortcoming as an opportunity to advance theory.
Reference: Paruchuri, S., Perry-Smith, J. E., Chattopadhyay, P., & Shaw, J. D. (2018). New ways of seeing: Pitfalls and opportunities in multilevel research. Academy of Management Journal, 61(3), 797–801. https://doi.org/10.5465/amj.2018.4003