AI Knowledge Scale

Abstract

The AI Knowledge Scale (Kerstan, Bienefeld, & Grote, 2024) is a psychometric instrument designed to evaluate an individual’s objective, factual understanding of core artificial intelligence (AI) principles, algorithmic operations, and foundational computational concepts. Developed within the context of human-technology interaction and healthcare decision-making, the instrument addresses a critical methodological challenge in psychometrics: distinguishing true declarative comprehension from self-assessed, subjective technological competence. The scale consists of six carefully curated, domain-spanning statements evaluated using a two-faceted response mechanism comprising a binary correctness determination (True versus False) followed by a confidence rating scale (ranging from unsure to sure). To rigorously control for guessing behaviors and pseudo-knowledge, items are scored dichotomously, crediting participants with a point only when a correct answer is accompanied by high subjective certainty. Psychometric evaluation demonstrates sound measurement characteristics, including non-parametric Item Response Theory (IRT) scalability via Mokken scale analysis ($H = 0.34$), an acceptable Molenaar Sijtsma reliability statistic ($ ext{MS} = 0.62$), and robust structural validity within a broader five-factor structural equation model ($chi^2 = 1008.46$,$ ext{df} = 547$,$ ext{CFI} = 0.91$,$ ext{TLI} = 0.90$,$ ext{RMSEA} = 0.04$,$ ext{SRMR} = 0.06$). By establishing a rigorous standard for measuring objective AI literacy, this instrument provides researchers in psychology, human-computer interaction, cognitive science, and risk communication with an empirical tool to investigate how verifiable technological knowledge shapes risk-benefit calculus, comparative trust attitudes, and adoption intentions toward algorithmic systems.

Keywords

Artificial Intelligence Knowledge, Objective Knowledge, AI Literacy, Psychometrics, Item Response Theory, Mokken Scale Analysis, Human-Computer Interaction, Algorithmic Trust, Risk Perception, Technology Acceptance

Authors

The scale was developed by an interdisciplinary research team from the Chair of Work and Organizational Psychology within the Department of Management, Technology, and Economics at ETH Zurich in Switzerland:

• Sophie Kerstan — Department of Management, Technology, and Economics, Work and Organizational Psychology, ETH Zurich, Zurich, Switzerland. Email: [email protected].
• Nadine Bienefeld — Department of Management, Technology, and Economics, Work and Organizational Psychology, ETH Zurich, Zurich, Switzerland.
• Gudela Grote — Department of Management, Technology, and Economics, Work and Organizational Psychology, ETH Zurich, Zurich, Switzerland.

Correspondence regarding the original research study and scale development should be addressed to Sophie Kerstan, ETH Zurich, Department of Management, Technology, and Economics, Work and Organizational Psychology, Weinbergstrasse 56/58, 8092 Zurich, Switzerland.

Purpose

The rapid proliferation of algorithmic systems across critical societal domains—including diagnostic medicine, legal analytics, automated finance, and autonomous transportation—has made the evaluation of public understanding an urgent academic priority. Prior research in cognitive psychology and consumer behavior has long established that subjective knowledge (what people believe they understand) frequently diverges from objective knowledge (verifiable factual competence stored in long-term semantic memory). When assessing public reception toward transformative technologies, relying solely on self-rated familiarity often captures metacognitive biases, such as the Dunning–Kruger effect, or general technological confidence rather than technical comprehension.

The primary purpose of the AI Knowledge Scale is to provide a brief, psychometrically validated instrument that assesses objective declarative knowledge regarding the operational mechanics, capabilities, and intrinsic constraints of modern artificial intelligence. Originally formulated to explore how comparative trust associations between human professionals (such as physicians) and autonomous diagnostic agents relate to risk and benefit perceptions, the scale enables researchers to dissect the cognitive antecedents of algorithmic aversion and algorithmic appreciation.

In applied and clinical settings, understanding a patient’s or provider’s objective AI comprehension is paramount for ethical governance and shared decision-making. When individuals fail to understand that AI systems can perpetuate historical data biases, or mistakenly assume that deep neural networks depend entirely on explicit human-coded rules, their appraisal of clinical risks is distorted. By deploying this instrument, behavioral scientists, policy analysts, and technology developers can:

• Accurately quantify baseline algorithmic literacy among non-expert populations before exposing them to high-stakes algorithmic interventions.
• Control for factual knowledge when modeling the psychological drivers of institutional trust, technophobia, and automation complacency.
• Evaluate the educational efficacy of public communication campaigns, explainable AI (XAI) interfaces, and digital health literacy interventions.
• Empirically examine how objective technological comprehension moderates the relationship between affective heuristics and cognitive risk-benefit trade-offs.

Psychological Construct

The psychological construct evaluated by this instrument is Objective Artificial Intelligence Knowledge. In cognitive psychometrics, objective knowledge represents verified information encoded in an individual’s cognitive schema that can be measured against standardized, independent criteria of truth. This construct stands in sharp contrast to subjective knowledge (perceived knowledge), which reflects self-efficacy, self-confidence, and introspective judgment.

Objective AI knowledge comprises multidimensional semantic networks encompassing fundamental computer science principles, statistical learning architectures, algorithmic boundaries, and sociotechnical realities. Although the scale yields a unidimensional composite score representing general baseline AI literacy, its constituent items strategically target key cognitive facets:

• Deep Neural Network Architectures: Conceptual awareness that advanced deep learning is defined by artificial neural networks structured across hierarchical, multiple hidden layers that autonomously extract intermediate representations from raw inputs.
• Algorithmic Bias and Probabilistic Fallibility: Recognition that machine learning models inherit, amplify, and reflect statistical biases present in their training corpora, debunking the common lay myth that computational agents possess inherent, objective neutrality.
• Computational Infrastructure and Performance Metrics: Elementary technical literacy regarding hardware throughput and processing power, indexed by foundational units such as FLOPS (floating-point operations per second).
• Learning Paradigms and Taxonomy: Understanding the operational demarcation between supervised learning (which requires explicit ground-truth annotations) and unsupervised learning (which discovers latent patterns in unlabeled data without external supervision).
• Philosophical Benchmarks and Machine Intelligence: Accurate comprehension of foundational tests in computing history, such as the Turing test, including what it actually assesses (behavioral indistinguishability from human conversational output rather than demonstrating superior intellect over human intelligence).
• Deterministic versus Heuristic/Inductive Systems: Discarding the outdated mental model that modern AI relies exclusively on symbolic, top-down rule engines pre-programmed by human software engineers, recognizing instead the inductive, statistical nature of contemporary artificial intelligence.

By assessing these multifaceted dimensions, the construct captures an individual’s functional mental model of how autonomous systems operate, diverge from human cognition, and fail.

Theoretical Framework

The conceptual foundation of the AI Knowledge Scale bridges dual-process theory of cognition, the Knowledge-Attitude-Behavior paradigm, and contemporary theories of risk perception and technology acceptance. Dual-process models (such as those advanced by Daniel Kahneman and Keith Stanovich) differentiate between fast, intuitive, affect-driven System 1 processing and deliberative, analytical, logic-driven System 2 processing. When individuals encounter complex, opaque technologies such as artificial intelligence, System 1 typically generates intuitive heuristics grounded in popular media narratives, science fiction tropes, or affective trust.

However, accurate risk-benefit calculus requires System 2 cognitive engagement, which depends entirely on the availability of accurate declarative schemas stored in semantic memory. In the absence of objective technological knowledge, people rely on substitution heuristics, judging the safety or competence of an AI system through peripheral cues (e.g., brand authority, graphical user interfaces, or anthropomorphic cues). As posited in risk analysis frameworks (e.g., Paul Slovic’s “affect heuristic”), perceived risk and perceived benefit are naturally negatively correlated in lay minds, whereas in objective reality, risk and benefit are frequently uncorrelated or positively related. Objective knowledge serves as a critical cognitive moderator: individuals possessing verified factual knowledge are better equipped to decouple affect from analytic assessment, viewing AI technologies through differentiated risk profiles rather than monolithic optimism or pessimism.

Furthermore, the scale incorporates metacognitive calibration theory. In knowledge testing, a major psychometric artifact is the presence of successful guessing, which artificially inflates performance scores and confounds ability estimates. By utilizing a two-tiered testing paradigm inspired by confidence-weighted testing (e.g., Bearth et al., 2019), the scale requires that respondents not only identify the correct truth value of an algorithmic principle but also demonstrate high subjective confidence in their choice. This design models the theoretical position that actionable, functional knowledge requires both accuracy and metacognitive stability; holding a correct belief with high uncertainty functions cognitively like an absence of knowledge when individuals are faced with real-world technological choices.

Validity

The construct, convergent, and structural validity of the AI Knowledge Scale was rigorously demonstrated in empirical investigations involving adult samples from the United States (Kerstan, Bienefeld, & Grote, 2024). The psychometric validation process incorporated multiple complementary methodologies:

• Non-Parametric Item Response Scalability: The unidimensional scalability of the scale was tested using Mokken scale analysis, a non-parametric IRT framework that evaluates whether items conform to a cumulative, hierarchical Guttman-like structure. The total scale yielded a scalability coefficient of $H = 0.34$, with individual item scalability coefficients exceeding acceptable thresholds. In psychometric conventions, an $H$ value between $0.30$ and $0.40$ indicates a valid, albeit weak-to-moderate, unidimensional cumulative scale, confirming that the six items measure a single underlying continuum of objective AI knowledge.
• Convergent and Criterion Validity: Convergent validity was established by correlating AI knowledge scores with broader measures of general scientific literacy and scientific knowledge. As hypothesized, participants with higher objective AI knowledge demonstrated statistically significant positive correlations with general scientific comprehension, supporting the premise that technological literacy is situated within broader cognitive frameworks of scientific reasoning.
• Nomological and Predictive Validity: When evaluated within structural equation models predicting risk and benefit perceptions of AI integration in healthcare, objective AI knowledge demonstrated distinct, theoretically congruent paths. Higher objective knowledge was significantly associated with more nuanced risk perceptions and specific comparative trust orientations between human medical practitioners and diagnostic artificial intelligence algorithms.

Reliability

Evaluating the internal consistency of brief objective knowledge tests presents unique psychometric nuances. Standard parametric reliability coefficients, such as Cronbach’s alpha, often underestimate the true reliability of short, dichotomously scored scales that exhibit varying item difficulties and non-normal distribution patterns. Consequently, the authors evaluated the internal consistency of the AI Knowledge Scale using the Molenaar Sijtsma (MS) statistic within the Mokken scale analysis framework.

The scale achieved a Molenaar Sijtsma coefficient of $ ext{MS} = 0.62$. In non-parametric item response theory and cognitive knowledge measurement, an$ ext{MS}$ value above $0.60$ for a brief six-item inventory represents acceptable reliability, aligning closely with benchmark coefficients observed in established objective knowledge scales measuring specialized scientific and technological domains (e.g., Bearth et al., 2019). The precision of the scale is enhanced by its strict scoring threshold, which converts guesses and unstable beliefs to null scores, thereby maximizing the signal-to-noise ratio in cross-sectional survey research.

Factor Analysis

The structural dimensionality of the AI Knowledge Scale was investigated using Confirmatory Factor Analysis (CFA) within a comprehensive structural equation modeling architecture that encompassed trust associations, risk assessments, and benefit evaluations across human and artificial healthcare agents.

The confirmatory factor model demonstrated acceptable overall goodness-of-fit across standard descriptive and absolute fit indices:

• $\chi^2 = 1008.46$, $ ext{df} = 547$,$p < 0.001$
• Relative Chi-Square ($\chi^2 / ext{df}$) = $1.84$ (well below the conservative threshold of $3.0$, indicating strong parsimony)
• Comparative Fit Index (CFI) = $0.91$
• Tucker-Lewis Index (TLI) = $0.90$
• Root Mean Square Error of Approximation (RMSEA) = $0.04$ ($90% \text{ CI } [0.036, 0.044]$)
• Standardized Root Mean Square Residual (SRMR) = $0.06$

All six items exhibited statistically significant factor loadings on their designated latent objective knowledge factor ($p < 0.001$), confirming that each statement substantially and meaningfully contributes to the measurement of the underlying latent trait without excessive multidimensional error covariance.

Instrument / Measurement Tool

• Instrument Name: AI Knowledge Scale
• Authors: Sophie Kerstan, Nadine Bienefeld, and Gudela Grote
• Year: 2024
• Construct Measured: Objective Artificial Intelligence Knowledge
• Target Population: Adults (18 years and older); applicable across general consumer, organizational, and clinical stakeholder samples
• Administration Format: Self-administered electronic questionnaire (online surveys, computer-assisted web interviews)
• Number of Items: 6 objective test statements
• Response Structure: Two-faceted response format per item:
  • Facet 1 (Correctness): Binary classification (True / False)
  • Facet 2 (Certainty / Confidence): 4-point rating scale measuring subjective certainty:
    • $0$ = Unsure (or $1$ = Unsure depending on index offset)
    • $1$ = Slightly unsure (or $2$ = Fairly unsure)
    • $2$ = Slightly sure (or $3$ = Fairly sure)
    • $3$ = Sure (or $4$ = Sure)
• Scoring Algorithm:
  • Each item is scored dichotomously ($0$ or $1$).
  • A score of $1$ is assigned only if the respondent provides the correct substantive answer (True or False) and indicates the highest confidence level (indicating they are “Sure” / score of $3$).
  • A score of $0$ is assigned if the substantive answer is incorrect, or if the correct substantive answer was selected with lower confidence (unsure, slightly unsure, or slightly sure).
  • Total Score Calculation: Sum of all item scores, yielding a continuous theoretical range from $0$ to $6$. Higher scores reflect greater verified, objective AI knowledge.

Permissions & Fee and Test Year

The AI Knowledge Scale was published in 2024 in the peer-reviewed journal Risk Analysis under an Open Access or standard scientific dissemination framework. The scale items were developed for non-commercial academic research and educational evaluation.

Researchers intending to administer the instrument in academic or educational studies may do so without licensing fees, provided that appropriate bibliographic credit is accorded to the original authors (Kerstan, Bienefeld, & Grote, 2024). Parties interested in commercial deployment, proprietary software integration, or widespread industrial certification testing should contact the corresponding author, Sophie Kerstan ([email protected]), and the Department of Management, Technology, and Economics at ETH Zurich regarding institutional permissions.

References

• Bearth, A., Cousin, M.-E., & Siegrist, M. (2019). The chemical hazard awareness scale: Measuring consumers’ knowledge of chemical hazards. Risk Analysis, 39(12), 2616–2630. https://doi.org/10.1111/risa.13374
• Kerstan, S., Bienefeld, N., & Grote, G. (2024). Choosing human over AI doctors? How comparative trust associations and knowledge relate to risk and benefit perceptions of AI in healthcare. Risk Analysis, 44(4), 939–957. https://doi.org/10.1111/risa.14216
• Mokken, R. J. (1971). A theory and procedure of scale analysis: With applications in political research. Walter de Gruyter. https://doi.org/10.1515/9783110813203
• Sijtsma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item response theory. SAGE Publications. https://doi.org/10.4135/9781412984676
• Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25(1), 44–56. https://doi.org/10.1038/s41591-018-0300-7

Items of the Scale