Problem-Solving Tests

Abstract

The Problem-Solving Tests, developed by J. P. Guilford, Philip R. Merrifield, Paul R. Christensen, and J. W. Frick in 1962 under the auspices of the Aptitudes Research Project at the University of Southern California, represent a landmark psychometric battery designed to operationalize and isolate the discrete intellectual factors underlying human problem-solving performance. Operating within the theoretical architecture of Guilford’s Structure-of-Intellect (SOI) model, the battery sought to determine whether problem solving constitutes a unitary general ability or, conversely, a composite phenomenon driven by distinct cognitive operations, contents, and products. The complete experimental investigation integrated 33 paper-and-pencil tests, comprising 12 newly constructed experimental measures, 3 complex criterion problem-solving tasks, and 18 established marker tests selected to represent previously identified cognitive factors. Administered to a homogeneous cohort of United States Naval Air Cadets and Aviation Officer Candidates (N = 219), the battery utilized both completion-type and multiple-choice formats focused predominantly on semantic and verbal stimuli. Extensive factor analytic evaluation extracted ten previously established intellectual factors and confirmed one newly isolated factor, notably elucidating the roles of cognition of semantic systems (verbal comprehension), convergent production of semantic transformations, divergent production, and evaluation in structured problem sequences. While criterion variance demonstrated that complex problem solving relies heavily on the preparation, production, and redefinition of semantic structures, empirical findings indicated that “problem solving” does not manifest as an autonomous, monolithic psychometric entity. Rather, it reflects the coordinated deployment of specialized aptitudes across identifiable task phases. This article provides a comprehensive academic analysis of the Problem-Solving Tests, delineating their theoretical foundation, psychometric properties, factor structure, and historical contribution to cognitive measurement.

Keywords

Problem-Solving Tests, J. P. Guilford, Structure-of-Intellect, Aptitudes Research Project, Factor Analysis, Cognitive Psychology, Semantic Problem Solving, Convergent Production, Divergent Thinking, Psychometrics, Naval Personnel Selection

Authors

The Problem-Solving Tests were formulated, investigated, and documented by key researchers associated with the Aptitudes Research Project at the University of Southern California (USC):

• Philip R. Merrifield: Co-principal investigator and psychometrician at the Psychological Laboratory, University of Southern California, who contributed extensively to factorial differentiation of higher-order cognitive processing and creativity.
• Joy Paul Guilford: Principal investigator, Professor of Psychology at USC, former president of the American Psychological Association (APA), and primary architect of the Structure-of-Intellect model.
• Paul R. Christensen: Research associate and long-time collaborator at the Aptitudes Research Project, specializing in the measurement of fluency, flexibility, and creative aptitudes.
• J. W. Frick: Research associate, psychometric consultant, and statistician within the Aptitudes Research Project.

Inquiries regarding the historical datasets, research monographs, and foundational test protocols are curated within the historical archives of the Aptitudes Research Project at the University of Southern California and documented in the historical records of the United States Office of Naval Research (Personnel and Training Branch).

Purpose

The primary purpose of the Problem-Solving Tests was to subject the multi-stage phenomenon of human problem solving to systematic factor-analytic scrutiny in order to determine its constituent intellectual dimensions. In the early 1960s, prevailing theories of human intelligence remained divided between unitary models dominated by Spearman’s general factor (g) and multi-factorial models such as Thurstone’s Primary Mental Abilities. Educational and military selection programs frequently treated “problem-solving ability” as a monolithic trait, assuming that general intelligence tests sufficiently captured an individual’s capacity to navigate complex, novel, and ambiguous decision environments. Merrifield, Guilford, Christensen, and Frick (1962) contested this assumption, proposing that problem solving represents an integrated sequence of distinct intellectual operations rather than an indivisible psychological trait.

The practical necessity for this research arose directly from the complex demands placed on military personnel, particularly aviation candidates and naval flight officers. The United States Navy required psychometric instruments that could identify specific cognitive strengths and deficits in candidates facing high-stakes operational dilemmas, emergency troubleshooting, strategic route planning, and rapid tactical decision-making. Conventional intelligence assessments failed to provide diagnostic clarity regarding why certain trainees excelled at identifying core issues but faltered in generating viable alternatives, or why others generated innovative strategies but failed to evaluate their operational feasibility. By constructing targeted, factor-pure tests alongside realistic, complex criterion problems, the authors intended to identify the exact cognitive components that predict success across discrete stages of problem resolution.

From a theoretical standpoint, the tests were engineered to evaluate four hypothesized phases of problem solving derived from classical models (such as John Dewey’s reflective thinking paradigm): preparation, analysis, production, and verification. The authors sought to answer three foundational psychometric questions:

• Can complex problem-solving criteria be fully accounted for by established Structure-of-Intellect factors?
• Are there unique, emergent cognitive factors that appear only when an individual is confronted with complex, multi-step problem structures?
• What specific roles do semantic operations—particularly the cognition, production, and evaluation of semantic systems, transformations, and implications—play in resolving verbal dilemmas?

Through this empirical battery, Merrifield and colleagues aimed to provide researchers, industrial-organizational psychologists, and human-factors engineers with an analytical taxonomy of cognitive operations capable of informing personnel selection, training program design, and aptitude diagnosis.

Psychological Construct

The psychological construct operationalized by the Problem-Solving Tests is defined as the multi-faceted cognitive processing required to resolve an impasse when no standard, habitual response is immediately available. Rather than conceptualizing problem solving as a single construct, the authors framed it as an umbrella meta-construct comprising distinct, interactive intellectual abilities classified under the Structure-of-Intellect (SOI) taxonomy. The primary focus of the 1962 battery was positioned within the semantic content domain—involving verbal ideas, meanings, and conceptual relationships—and encompassed several discrete cognitive dimensions:

1. Cognition of Semantic Systems (CMS)

This dimension reflects the ability to comprehend, structure, and grasp complex interrelated verbal concepts as a coherent, structured whole. In problem solving, this ability functions as the primary vehicle for problem structuring and situational awareness. Individuals high in CMS rapidly comprehend complicated instructions, grasp the overall scope of an organizational dilemma, and recognize how individual environmental constraints interconnect.

2. Cognition of Semantic Units (CMU)

Classically recognized as verbal comprehension, this ability involves recognizing the meanings of isolated words and conceptual units. Within the problem-solving construct, CMU serves as a baseline foundational facility; without an adequate grasp of precise vocabulary and concepts, higher-order problem interpretation is impeded.

3. Convergent Production of Semantic Transformations (NMT)

Often termed conceptual redefinition or insight, NMT represents the capacity to produce a single, uniquely determined solution that requires shifting the function, meaning, or usage of a familiar object or concept. In functional fixedness paradigms, an individual must break away from traditional semantic associations to repurpose an existing tool or concept in service of problem resolution.

4. Convergent Production of Semantic Implications (NMI)

This ability involves deducing necessary, uniquely determined conclusions or intermediate steps from a given set of antecedents. In structured problem solving, NMI governs sequential deduction, logical planning, and the generation of chained procedural actions necessary to navigate from an initial problem state to a goal state.

5. Divergent Production of Semantic Systems (DMS) and Units (DMU)

These dimensions represent ideational fluency, conceptual flexibility, and the ability to generate a wide variety of alternative solutions, hypotheses, or approaches when a single, predetermined solution path does not exist. In the early stages of ill-defined problem solving, high divergent production allows an individual to escape cognitive ruts and explore broad search spaces.

6. Evaluation of Semantic Relations (EMR) and Implications (EMI)

These evaluative abilities pertain to judgment, critical appraisal, and error detection. Problem solvers must continuously monitor intermediate outputs, appraise whether hypothesized steps satisfy problem constraints, identify logical inconsistencies in proposed plans, and decide whether a final solution is acceptable. Evaluative factors ensure that ideational production does not degenerate into unfeasible speculation.

Theoretical Framework

The theoretical architecture undergirding the Problem-Solving Tests is anchored directly in J. P. Guilford’s Structure-of-Intellect (SOI) model, synthesized with classical psychological paradigms of reflective thought, particularly those articulated by John Dewey (1910) and Gestalt psychology (Karl Duncker, Max Wertheimer).

Guilford’s Structure-of-Intellect Model

Guilford’s SOI theory organizes human intelligence across a three-dimensional morphological model: Operations, Contents, and Products. Any cognitive task can be classified according to the specific intersection of these three facets:

• Operations (What the mind does): Cognition (C), Memory (M), Divergent Production (D), Convergent Production (N), and Evaluation (E).
• Contents (The nature of the information): Figural/Visual (F), Symbolic (S), Semantic (M), and Behavioral (B).
• Products (The organizational form of information): Units (U), Classes (C), Relations (R), Systems (S), Transformations (T), and Implications (I).

Under this theoretical framework, human problem solving is not a discrete cell within the SOI matrix; rather, it is an emergent dynamic process that draws iteratively upon multiple cells within the semantic content category. The Problem-Solving Tests were deliberately configured to target specific intersections, with particular emphasis on systems (interrelated parts), transformations (restructuring), and implications (extrapolations and consequences).

The Problem-Solving Process Cycle

Merrifield et al. (1962) mapped the theoretical SOI factors directly onto the sequential phases of problem resolution, which they conceptualized as a continuous feedback loop:

• Phase 1: Input and Preparation: The environment presents an ambiguous situation or goal discrepancy. The individual must perceive and decode incoming stimuli. Here, Cognition factors (specifically CMU and CMS) dominate as the problem space is internalized.
• Phase 2: Analysis and Definition: The problem solver distinguishes essential variables from irrelevant noise, identifying the structural requirements of the goal state. Cognition of Semantic Transformations (CMT) and Evaluation of Semantic Units (EMU) are critical in filtering misleading paths.
• Phase 3: Generation of Hypotheses (Production): The individual searches cognitive memory stores to retrieve or construct potential solutions. If the problem is closed-ended, Convergent Production (NMT, NMI) generates the precise sequence required; if the problem is open-ended or ill-defined, Divergent Production (DMS, DMT) yields alternative options.
• Phase 4: Verification and Execution (Evaluation): Proposed solutions are mentally simulated and checked against initial constraints. Evaluative operations (EMR, EMI) determine whether the proposed solution satisfies the objective or whether the individual must loop back to redefine the problem.

By mapping factorially isolated tests to this cycle, the authors established a theoretical bridge between differential psychometrics and cognitive process models.

Validity

The empirical validation of the Problem-Solving Tests relied on a multi-pronged factor-analytic and criterion-related framework documented in the 1962 monograph. The experimental study was conducted with a rigorously defined, homogeneous sample of 219 Naval Air Cadets and Aviation Officer Candidates undergoing rigorous pre-flight training at the Naval Air Station, Pensacola, Florida. This sample represented a high-aptitude adult population with minimal demographic attrition, providing an optimal baseline for cognitive ability testing.

Construct and Factorial Validity

Construct validity was established through factor loading patterns across the 33 variables. Marker tests with known historical loadings (established through previous Aptitudes Research Project investigations) were embedded alongside experimental tests to define the factor space. Construct validity was confirmed when newly developed experimental tests demonstrated high, statistically significant loadings on their hypothesized target factors while displaying minimal cross-loadings on non-target factors. For example:

• Cognition of Semantic Systems (CMS): Marker tests such as the General Information Test and Verbal Comprehension reliably defined the axis, against which experimental measures of system comprehension loaded between .40 and .68.
• Convergent Production of Semantic Transformations (NMT): Demonstrated construct separation from general verbal ability; tests requiring conceptual redefinition loaded uniquely on NMT (.38 to .56), demonstrating that redefinition represents an ability distinct from vocabulary knowledge or rote memory.

Criterion-Related Validity

To determine the validity of the factor battery in predicting complex, ecologically valid problem solving, Merrifield et al. incorporated three elaborate criterion tasks:

1. Missing Information: A test requiring candidates to analyze real-world situational problems, identify missing information essential for resolution, and outline steps needed to acquire it.
2. Predicaments Test: An assessment requiring individuals to resolve complex, unexpected, and resource-constrained practical emergencies.
3. Planning Test: A procedural sequencing test requiring candidates to devise step-by-step navigational and logistical routes under tight operational rules.

Multiple regression and canonical correlation analyses revealed that variance in these criterion measures was substantially accounted for by a combination of SOI factors. Specifically, the criterion tasks exhibited multiple correlations (R) ranging from .45 to .62 with the factor battery. Crucially, the analysis demonstrated that no single general factor accounted for success on the criteria. Performance on Missing Information was primarily driven by Cognition of Semantic Systems and Evaluation, whereas success on Predicaments relied heavily on Convergent Production of Semantic Transformations (NMT) and Ideational Fluency (DMU). This differential predictive validity confirmed the authors’ core thesis: complex problem solving is task-specific in its cognitive composition.

Reliability

Given the rigorous standards of Guilford’s Aptitudes Research Project, all 33 tests utilized in the Problem-Solving Tests battery underwent rigorous reliability screening. Because the majority of the tests were timed, speeded, or split into independently timed halves, internal consistency and split-half reliability methodologies were systematically employed.

Internal Consistency and Split-Half Estimates

For the majority of the completion-type and multiple-choice tests, reliability was evaluated using split-half correlation coefficients (odd-even item splits), corrected via the Spearman-Brown prediction formula. For measures administered under unspeeded power conditions, Kuder-Richardson Formula 20 (KR-20) was computed. Across the battery, the reported reliability coefficients were predominantly robust for short, highly specialized cognitive instruments:

• Marker tests representing established factors demonstrated high reliability, typically ranging from r = .78 to .91 (e.g., Verbal Comprehension, r = .89; General Information, r = .84).
• The newly developed experimental problem-solving tests, designed to be concise and targeted, exhibited reliability coefficients ranging between r = .62 and .81.
• The three complex criterion problem-solving tasks exhibited split-half reliabilities of .68 (Missing Information), .72 (Predicaments Test), and .77 (Planning Test).

Measurement Error and Test Limitations

The investigators noted that certain completion-type divergent production tests yielded somewhat lower inter-rater scoring reliability (ranging between .70 and .82) due to the subjective judgment required to classify creative or non-standard completions. However, rigorous scoring manuals, categorical protocols, and pre-calibrated scoring keys substantially mitigated subjective error variance. Tests operating within the convergent production and evaluative domains displayed objective scoring fidelity with near-perfect scorer agreement (r > .98).

Factor Analysis

The mathematical and psychometric backbone of the Merrifield et al. (1962) study was a large-scale exploratory factor analysis (EFA). The 33 test variables administered to the 219 naval candidates yielded an intercorrelation matrix consisting of Pearson product-moment correlation coefficients.

Factor Extraction and Rotational Procedures

Initial factor extraction was performed using Thurstone’s centroid method (a prevailing computational technique of the era conceptually analogous to iterated principal axis factoring). Following the extraction of 14 centroid factors, the authors performed orthogonal and oblique rotations toward simple structure using both the Zimmerman graphical rotational technique and analytical criteria aligning with Thurstone’s principles of simple structure. Factors were rotated until psychological meaningfulness was maximized with respect to the theoretical Structure-of-Intellect hypothecated coordinates.

Identified Factors and Factor Structure

The rotational solution converged on eleven clearly defined cognitive factors, ten of which confirmed established SOI constructs, while one represented an emerging factorial dimension:

• Factor CMU (Cognition of Semantic Units): Marked by standard vocabulary and verbal comprehension instruments (loadings ranging from .55 to .78).
• Factor CMS (Cognition of Semantic Systems): Highly defined by tests requiring the comprehension of complex textual sequences, situational overviews, and contextual relationships.
• Factor CMT (Cognition of Semantic Transformations): Defined by tests requiring the detection of hidden shifts in meaning or metaphorical insights.
• Factor CMI (Cognition of Semantic Implications): Marked by tests requiring candidates to anticipate problems, foresee consequences, and project future outcomes from present premises.
• Factor DMU (Divergent Production of Semantic Units): Loaded heavily with open-ended ideational fluency tests.
• Factor DMR (Divergent Production of Semantic Relations): Marked by tasks requiring the rapid production of diverse relational links between word pairs.
• Factor DMS (Divergent Production of Semantic Systems): Defined by tests requiring the creation of alternate organizational schemes or multifaceted plots.
• Factor DMT (Divergent Production of Semantic Transformations): Representing original flexibility, marked by unusual uses and adaptive restructuring tasks.
• Factor NMU (Convergent Production of Semantic Units): Involving naming and unique identity generation tasks.
• Factor NMT (Convergent Production of Semantic Transformations): Defined by functional redefinition tests, where an object’s conventional use must be altered to solve a physical or conceptual dilemma.
• Factor EMI (Evaluation of Semantic Implications): Characterized by tests requiring candidates to evaluate the logical soundness of conclusions, detect operational flaws, and verify deductive sequences.

Residual variance indicated that no single general problem-solving factor accounted for intercorrelations among the criterion tasks. Instead, each criterion loaded across a differentiated profile of these 11 factors, definitively supporting Guilford’s multi-component model.

Instrument / Measurement Tool

The complete Problem-Solving Tests battery represents a comprehensive, multi-part, paper-and-pencil assessment protocol. Rather than a single standalone questionnaire, it is structured as an experimental battery comprising 33 distinct subtests classified into three functional categories:

1. Battery Structure

• 12 Experimental Problem-Solving Tests: Newly devised measures targeting hypothesized but previously unconfirmed semantic operations (e.g., test tasks requiring problem formulation, functional redefinition, and procedural sequencing).
• 3 Criterion Tasks: Complex, multi-layered problem simulations (Missing Information, Predicaments, Planning Test) designed to mimic real-world operational challenges.
• 18 Marker Tests: Standardized, highly reliable reference tests from the Aptitudes Research Project archive used to anchor previously known SOI factors.

2. Administration and Testing Format

• Administration Method: Group-administered, proctored paper-and-pencil examination divided into multiple modular test booklets.
• Time Requirements: Administered across multiple testing sessions totaling approximately 6 to 8 hours to minimize cognitive fatigue among candidates. Individual subtests are strictly timed, with working times ranging from 2 to 15 minutes per section.
• Item Formats:
  • Multiple-Choice: Objective 4-choice or 5-choice items evaluating comprehension, deduction, and logical evaluation.
  • Completion-Type / Constructed Response: Short-answer verbal responses requiring candidates to state a solution, identify a missing variable, or invent an alternative use for an object.

3. Scoring Protocols

• Objective Scoring: Multiple-choice tests and convergent completion tests are scored using binary keys (1 = correct, 0 = incorrect), corrected for guessing where applicable using standard formula scoring: (S = R – frac{W}{n-1}).
• Categorical and Quality Scoring: Divergent production and complex problem-solving items are evaluated based on pre-established classification guides, measuring either the total number of relevant, non-redundant responses (ideational fluency) or the qualitative level of insight and transformation demonstrated.

Permissions & Fee and Test Year

The Problem-Solving Tests battery was formally published in 1962 in the monograph “The Role of Intellectual Factors in Problem Solving” (Psychological Monographs: General and Applied, Vol. 76, No. 10, Whole No. 529), authored by Philip R. Merrifield, Joy Paul Guilford, Paul R. Christensen, and J. W. Frick. The research was supported by the United States Navy under Contract Nonr-228(20) with the Office of Naval Research.

The individual subtests developed by Guilford’s Aptitudes Research Project were historically distributed and licensed through the Sheridan Psychological Services (later Orange Fortune). The full test booklets and experimental marker batteries are protected historical intellectual property. They are not available in the public domain for unrestricted commercial use, clinical assessment, or standardized psychological diagnosis. However, qualified researchers and psychometric historians may access test descriptions, sample items, and factorial factor matrices via the original APA publication, academic library repositories, or by contacting archival holdings at the University of Southern California and the American Psychological Association. Appropriate institutional permissions and academic qualifications are required for any contemporary adaptation or experimental replication.

References

• Dewey, J. (1910). How we think. D. C. Heath & Co. https://doi.org/10.1037/10903-000
• Duncker, K. (1945). On problem-solving. Psychological Monographs, 58(5), i–113. https://doi.org/10.1037/h0093599
• Guilford, J. P. (1956). The structure of intellect. Psychological Bulletin, 53(4), 267–293. https://doi.org/10.1037/h0040755
• Guilford, J. P. (1959). Three faces of intellect. American Psychologist, 14(8), 469–479. https://doi.org/10.1037/h0046827
• Guilford, J. P. (1967). The nature of human intelligence. McGraw-Hill.
• Merrifield, P. R., Guilford, J. P., Christensen, P. R., & Frick, J. W. (1962). The role of intellectual factors in problem solving. Psychological Monographs: General and Applied, 76(10), 1–21. https://doi.org/10.1037/h0093850
• Thurstone, L. L. (1938). Primary mental abilities. University of Chicago Press.
• Wertheimer, M. (1945). Productive thinking. Harper & Brothers.

Items of the Scale