This article, authored by Weng Marc Lim, offers a comprehensive typology of validity, a fundamental concept for ensuring the integrity, accuracy, and applicability of research findings in real-world contexts. The purpose of this paper is to aid researchers by delineating the concept of validity and presenting its typology, which is crucial for tackling real-world issues by effectively measuring and testing phenomena.
The article addresses a significant challenge in research: the fragmented understanding of validity’s multifaceted nature and the absence of a cohesive framework to systematically guide researchers in establishing validity across different stages of their studies. By synthesizing expertise, experience, and exposure, Lim provides a structured approach to validate research endeavors, thereby enhancing their quality and impact.
Understanding Validity: What It Is and Is Not
Validity is presented as a multifaceted concept that assesses the adequacy and appropriateness of inferences drawn from research data. It is about the legitimacy and meaningfulness of findings, ensuring they genuinely reflect the phenomena under study. Importantly, the article clarifies what validity is not:
- It is not one-dimensional and cannot be assessed in isolation or through a single method.
- It is not about the consistency of measurements (that is reliability).
- It is not merely about statistical correlation between measures.
- It is not solely an inherent property of a measurement instrument (e.g., a questionnaire) or a test (e.g., path analysis), but also a feature of the inferences made from data.
- Therefore, validity cannot be assumed; it must be rigorously established and argued for within each specific research context.
This nuanced understanding allows researchers to appreciate validity’s complexity and its critical role in upholding research integrity, ensuring that scholarly investigations yield insights that are both methodologically sound and deeply impactful.
A Staged Typology of Validity
The article emphasizes that understanding and applying various types of validity necessitates a staged approach, aligning with the sequential phases of the research process. This typology categorizes validity into six main types:
- Content Validity (1st stage – Pre-test before data collection):
- What it is: This assesses whether the instrument’s indicators (items) adequately and appropriately represent the constructs they aim to measure, based on conceptual soundness and operational coverage. It ensures the comprehensiveness and representativeness of measurement items.
- How to establish: It is established through expert evaluation (academic and/or industry experts), with a minimum of two experts, continuing until data saturation in feedback is reached. Alternatively, a card sorting exercise where experts categorize items without construct names, aiming for a correct hit ratio above 90%, can be used.
- Contribution: It is established by human judgment and contributes to construct validity (as it pertains to concepts), thereby enhancing substantive significance.
- Face Validity (2nd stage – Pilot study before data collection):
- What it is: This ensures that the instrument’s items appear clear, comprehensible, and relevant to potential participants from the target population. It addresses initial impressions and intuitive understandings.
- How to establish: It typically involves a pilot study with potential participants (minimum of two), soliciting feedback on clarity and relevance until data saturation is reached. Cross-checking by asking respondents to explain their understanding of questions is also recommended to avoid miscomprehension.
- Contribution: Like content validity, it is established by human judgment and contributes to construct validity, enhancing substantive significance.
- Convergent Validity (3rd stage – Main study after data collection, measurement model evaluation):
- What it is: This evaluates the extent to which multiple indicators statistically relate to the same construct, confirming they cohesively measure the same underlying concept.
- How to establish:
- For reflective indicators: Assessed using factor loadings (ideally ≥ 0.708) in EFA/CFA, indicating that the construct explains over 50% of the variance in the indicator. The Average Variance Extracted (AVE) should be 0.5 or higher, meaning the construct explains over half the variance in its indicators.
- For formative indicators: Established via a two-step approach: regression analysis (evaluating indicator weights and significance, with weights ≥ 0.50 being strong) and redundancy analysis (correlation between the formative construct and a conceptually identical/similar alternative construct, ideally ≥ 0.708).
- Contribution: It is established by statistics and contributes to construct validity, thereby enhancing statistical significance.
- Discriminant Validity (4th stage – Main study after data collection, measurement model evaluation):
- What it is: This assesses the extent to which measures of different constructs are distinct and not highly correlated, ensuring each construct captures unique phenomena. It is the opposite of convergent validity, focusing on separateness.
- How to establish:
- For reflective constructs: The square root of the AVE for each construct must be greater than its highest correlation with any other construct.
- For formative constructs: Assessed by ensuring indicators uniquely contribute without undue collinearity, typically using Variance Inflation Factor (VIF) values (ideally < 3.3 for conservative, < 5 for moderate, < 10 for liberal thresholds).
- For both: The Heterotrait-Monotrait (HTMT) ratio of correlations should be < 0.85 for conceptually distinct constructs, or < 0.90 for conceptually similar ones. Additionally, correlations between different constructs should ideally be < 0.70 for distinct ones and < 0.80 for similar ones.
- Contribution: It is established by statistics and contributes to construct validity, thereby enhancing statistical significance.
- Nomological Validity (5th stage – Main study after data collection, structural model evaluation):
- What it is: This evaluates the coherence and logical consistency of the inter-construct relationships within a model, ensuring they align with theoretical expectations grounded by established theories or schools of thought. It focuses on the model level.
- How to establish: Involves empirically testing the conceptual model using statistical techniques like Structural Equation Modeling (SEM).
- CB-SEM: Uses goodness-of-fit indices (GFI, AGFI, CFI, NFI, TLI ≥ 0.90; RMSEA < 0.08; relative chi-square < 3).
- PLS-SEM: Uses Standardized Root Mean Square Residual (SRMR) < 0.08.
- Other metrics: Coefficient of determination (R²) for explanatory power and predictive relevance (Q² > 0).
- Contribution: It is established by statistics and contributes to criterion validity (as it pertains to relationships and explaining outcomes), thereby enhancing statistical significance.
- Predictive Validity (6th stage – Main study after data collection, structural model evaluation):
- What it is: This assesses the ability of constructs to accurately forecast outcomes, demonstrating the practical utility of research findings. It operates at the relationship level.
- How to establish: Typically involves regression modeling, evaluating:
- Coefficients: Indicating the direction (þ/) and strength of the relationship.
- Statistical significance: Assessed via p-values (< 0.05 or < 0.01), t-values (e.g., |t| > 1.96 for 95% confidence, two-tailed), and confidence intervals (must not include zero).
- Effect size measures: Such as Cohen’s f² for regression or Pearson’s r for correlations, categorized as small, medium, or large (e.g., f² ≥ 0.02, r ≈ 0.1 to 0.3 for small).
- Contribution: It is established by statistics and contributes to criterion validity, thereby enhancing statistical significance. It also contributes to substantive significance when findings are convincingly extrapolated to reality.
Threats to Validity and Countermeasures
The article also addresses crucial threats to validity, which can compromise a study’s overall credibility and utility.
- Internal Threat to Validity: Common Method Bias (CMB)
- What it is: This bias arises when variations in data stem more from the methodology (e.g., measures, methods) used in data collection than from the constructs themselves, potentially skewing relationships.
- Treatment:
- Procedural: Ensuring anonymity, reducing evaluation concerns, and implementing temporal, psychological, or methodological separation of measurements.
- Statistical: Using Harman’s single-factor test (single factor should explain < 50% variance), full collinearity test (VIFs < 3.3), or the measured latent marker variable (MLMV) method (an unrelated marker variable should not significantly distort primary relationships).
- External Threat to Validity: Generalizability
- What it is: This concerns the applicability of findings beyond the specific conditions of the study, potentially restricted by sample representativeness or unique settings.
- Treatment:
- Methodological: Diversifying samples (e.g., cluster or stratified sampling), engaging in replication studies across varied contexts (industries, populations, time periods), and using triangulation (multiple data sources, methods, perspectives).
- Multiple Evidences: Aggregating and comparing results from various studies, often through systematic reviews (e.g., bibliometric analyses, meta-analyses) to discern patterns and variances for grounded generalizations.
Broad Applicability and Practical Guidance
The principles outlined in this typology are robust and relevant across various disciplines and contexts, including both traditional and digital/online research methodologies. Whether in management, marketing, education, psychology, or sociology, the foundational aspects of validity remain consistent. For example, content validity ensures survey questions for an online survey are as comprehensive as for a paper-and-pen survey, and nomological and predictive validity ensure that models and relationships hold true in e-commerce as they do in shopping malls. While specific methodological adjustments might be needed for digital contexts (e.g., data encryption for integrity, stratified sampling for bias), these fall under broader research methodology, allowing the core validity standards to remain applicable.
Ultimately, this article serves as a practical guide for researchers to systematically establish validity throughout their work, ensuring that research findings are not only methodologically robust but also practically relevant. By providing a structured framework, it aims to elevate research practice beyond conceptual acknowledgment to a fundamental aspect of research integrity, fostering rigor and impact in the pursuit of knowledge.
Reference: Lim, W. M. (2024). A typology of validity: content, face, convergent, discriminant, nomological and predictive validity. Journal of Trade Science, 12(3), 155-179. https://doi.org/10.1108/JTS-03-2024-0016