Daily Experience of Simulation Training (DEST)

Abstract

The Daily Experience of Simulation Training (DEST), developed by Chiu, Cross, Wheeler, Evans, and Goulet (2023), is an ecologically sensitive, multi-component psychometric instrument designed to capture daily fluctuations in self-efficacy, pedagogical feedback perceptions, and qualitative professional reflections among child welfare professionals undergoing immersive simulation-based education. Constructed to evaluate cohort-level learning trajectories at the Child Protection Training Academy (CPTA)—a collaborative venture established by the University of Illinois at Springfield (UIS) and the Illinois Department of Children and Family Services (DCFS)—the DEST operationalizes professional growth in high-stakes child welfare contexts. The core quantitative engine of the instrument, the Daily Experience of Simulation Training – Confidence Subscale (DEST-CS), comprises 13 distinct items evaluated on a 7-point Likert scale (ranging from 1 = lowest confidence to 7 = highest confidence), embedded within a broader 20-item daily tracking battery that includes multi-source trainer and standardized actor feedback appraisal alongside an open-ended narrative reflection log. Psychometric evaluations conducted across longitudinal time points during intensive, week-long investigative simulation boot camps demonstrate outstanding reliability and structural stability. Specifically, principal component analyses with varimax rotation across six distinct assessment waves yielded a persistent unifactorial confidence construct, accounting for substantial portions of total variance ranging from 71.4% (eigenvalue = 9.28) to 76.5% (eigenvalue = 9.94). Internal consistency reliability estimates were extraordinarily high, with Cronbach’s alpha coefficients exceeding α = .95 at every single administration point. The DEST bridges the gap between traditional pre-/post-training evaluation models and dynamic, intra-individual process tracking, affording researchers and workforce developers an empirically validated framework for monitoring deliberate skill acquisition, procedural confidence, and experiential learning in child protective services.

Keywords

Child Welfare, Professional Competence, Simulation Training Effectiveness, Child Protection Trainees, Self-Efficacy Measurement, Longitudinal Daily Tracking, Program Evaluation, Protective Services, Deliberate Practice, Factor Analysis, Psychometrics, Experiential Learning

Authors

The Daily Experience of Simulation Training (DEST) was conceptualized, validated, and published by an interdisciplinary team of researchers and child protection clinical educators associated with the University of Illinois system:

• Yu-Ling Chiu, Ph.D. – Children and Family Research Center (CFRC), School of Social Work, University of Illinois at Urbana-Champaign. (Corresponding author: [email protected]).
• Theodore P. Cross, Ph.D. – Children and Family Research Center (CFRC), School of Social Work, University of Illinois at Urbana-Champaign.
• Amy B. Wheeler, Ph.D. – Child Protection Training Academy, Center for State Policy and Leadership, University of Illinois at Springfield.
• Susan M. Evans, M.A. – Child Protection Training Academy, University of Illinois at Springfield.
• Betsy P. Goulet, D.P.A. – Department of Public Administration and Child Protection Training Academy, University of Illinois at Springfield.

Purpose

Child protection casework is characterized by immense cognitive load, high emotional volatility, ambiguous investigative boundaries, and life-altering decision-making. Historically, public child welfare agencies have relied heavily on didactic classroom instruction to onboard new child protection investigators. However, empirical literature in public administration and social work education has repeatedly highlighted the failure of traditional didactic pedagogy to bridge the “transfer of learning” chasm—leaving newly minted investigators vulnerable to acute workplace anxiety, premature turnover, procedural error, and severe burnout. To transform this training paradigm, the Child Protection Training Academy (CPTA) integrated experiential learning and clinical simulation paradigms originally pioneered within aerospace and medical education, constructing mock physical environments (such as standardized residential home labs, forensic interview rooms, and simulated courtrooms) with professional standardized actors (Goulet et al., 2020).

The primary purpose of the Daily Experience of Simulation Training (DEST) is to provide a robust, daily repeated-measures psychometric evaluation protocol specifically engineered for week-long simulation academies. Unlike static pre-test/post-test instruments that mask non-linear learning curves, temporary dips in confidence (such as the “unconscious incompetence to conscious incompetence” transition), or daily instructional bottlenecks, the DEST measures day-to-day fluctuations in trainee confidence and learning experience. Its administrative and clinical purposes include:

• Longitudinal Cohort Tracking: Dynamically monitoring the micro-development of investigative confidence across discrete foundational skills (e.g., assessing immediate child safety, framing hypothesis testing versus confirmation bias, conducting forensic collateral interviews, engaging hostile caregivers, and surviving courtroom cross-examination).
• Multisource Ecological Feedback: Systematically capturing trainee appraisal of multi-role instruction, differentiating the perceived educational value of academic classroom trainers, simulation lab facilitators, and standardized patient/client actors who provide direct feedback from the perspective of parents, paramours, or secondary caregivers.
• Integrated Qualitative Metacognition: Promoting end-of-day self-reflection through an integrated reflective log, prompting novice investigators to articulate key operational takeaways, synthesize corrective debriefings, and internalize investigative behavioral scripts.
• Programmatic Quality Assurance: Furnishing child welfare administrators and simulation curriculum designers with rapid-cycle, formative evaluation data to adjust instructional pacing, identify failing simulation scenarios, and calibrate case complexity to trainee capabilities.

Psychological Construct

The DEST operationalizes a multidimensional yet structurally integrated assessment of task-specific professional self-efficacy, instructional feedback utility, and metacognitive reflection within an immersive occupational learning ecology. In psychometrics, professional self-efficacy does not represent a generalized, diffuse personality trait; rather, it manifests as a contextualized belief in one’s capability to mobilize cognitive, behavioral, and emotional resources required to meet specific situational demands.

1. Investigative Self-Efficacy and Procedural Confidence (DEST-CS)

The foundational quantitative core of the instrument (Items 1–13) measures task-specific self-efficacy across critical operational domains encountered by child protection investigators. Rather than assessing abstract legal knowledge, these items measure the perceived self-competence to execute complex, dual-purpose social-investigative tasks:

• Forensic Information Gathering and Fact Discrimination: Items such as “Gather info from collateral contacts” and “Think critically on facts vs. hypotheses” target the cognitive capacity to resist premature cognitive closure, distinguish verifiable evidence from uncorroborated allegations, and synthesize data from schools, law enforcement, and medical providers.
• Crisis Engagement and System Navigation: Items such as “Engage families,” “Identify family strengths,” and “Integrate compassion and investigative skill” operationalize the delicate clinical equilibrium between empathetic casework and rigorous child safety enforcement.
• Safety Intervention and Defensive De-escalation: Items including “Assess safety,” “Explain need for safety plan and/or protective custody,” and “Answer pointed questions from parents and caregivers” evaluate the investigator’s confidence in asserting state authority, handling parental anger, and establishing viable, legally sound protective safety plans.
• Specialized Adversarial Execution: Items such as “Address underlying conditions such as domestic violence, substance abuse, mental health, developmental disabilities,” “Testify in court,” and overall confidence to “Work as a DCFS investigator” measure preparedness for systemic, high-stakes institutional interactions under acute scrutiny.

2. Multisource Pedagogical Feedback Receptivity

The secondary quantitative dimension assesses the trainee’s subjective valuation of feedback provided across distinct educational roles. Grounded in adult learning theory, the DEST captures how trainees process feedback from classroom instructors, simulation trainers, and standardized actors playing familial roles (e.g., “Mother Figure,” “Paramour Figure,” and “Other Adult Caregiver”). This dimension captures the educational impact of receiving immediate, unscripted feedback directly from the perspective of simulated clients.

3. Reflective Metacognitive Log

The qualitative narrative dimension captures cognitive assimilation through a focused journaling prompt. By articulating the most meaningful concepts or skills acquired each day, trainees engage in self-regulation, transforming raw simulation stress into structured professional competence.

Theoretical Framework

The theoretical architecture of the Daily Experience of Simulation Training is deeply rooted in three converging frameworks of cognitive psychology, educational theory, and occupational competence: Bandura’s Social Cognitive Theory, Kolb’s Experiential Learning Model, and Ericsson’s Theory of Deliberate Practice.

1. Bandura’s Self-Efficacy Theory

Albert Bandura’s (1997) self-efficacy construct posits that an individual’s belief in their ability to succeed in specific situations governs how they think, behave, and feel. Self-efficacy is cultivated through four primary informational sources: mastery experiences, vicarious experiences, verbal persuasion, and physiological/affective states. Simulation training in child welfare provides direct mastery experiences through high-fidelity role-play, vicarious experiences through peer observation, verbal persuasion via targeted expert debriefing, and emotional desensitization to acute workplace arousal. The DEST measures the daily calibration of this self-efficacy construct, capturing how confidence shifts from baseline anxiety to grounded professional agency as trainees encounter realistic family resistance, domestic violence scenarios, and judicial interrogation.

2. Kolb’s Experiential Learning Cycle

David Kolb’s (1984) experiential learning theory delineates a four-stage cyclical model: Concrete Experience, Reflective Observation, Abstract Conceptualization, and Active Experimentation. The structure of the DEST parallels this cycle exactly. Trainees undergo a Concrete Experience in the simulation lab, engage in Reflective Observation during debriefings, formulate Abstract Conceptualizations aided by multi-trainer feedback, and engage in Active Experimentation the following training day. The DEST functions as an explicit metacognitive instrument that formalizes the Reflective Observation and Abstract Conceptualization phases via daily psychometric evaluation and narrative logging.

3. Ericsson’s Theory of Deliberate Practice

K. Anders Ericsson’s (1993) deliberate practice model establishes that true professional expertise is not accumulated merely through passive tenure, but through targeted, highly structured practice paired with immediate informational feedback, error detection, and repetitive refinement. Simulation training operationalizes deliberate practice in social welfare. The DEST captures the educational mechanisms of deliberate practice by holding trainees accountable to multi-perspective feedback (from trainers and actors alike), isolating concrete operational competencies, and systematically measuring self-efficacy growth day over day.

Validity

The psychometric validation of the Daily Experience of Simulation Training (DEST) was executed within a rigorous statewide evaluation framework involving cohorts of newly hired child protection investigators entering the Illinois Department of Children and Family Services (DCFS) through the Child Protection Training Academy (Chiu et al., 2023).

1. Content and Face Validity

Content validity was established through collaborative, expert-driven curricular design. The 13 confidence items were derived directly from statutory mandates, Illinois DCFS core practice models, forensic interviewing protocols, and extensive job analysis interviews conducted with veteran investigative supervisors, administrative judges, and clinical child welfare educators. Each item maps directly to a high-frequency, high-liability competency required during the initial 60-day child abuse/neglect investigation window. Face validity was confirmed via cognitive walkthroughs with pilot cohorts of novice caseworkers who confirmed that the DEST items accurately reflected their daily cognitive challenges, emotional hurdles, and procedural duties.

2. Construct and Structural Validity

Construct validity is evidenced through the tool’s exceptional structural stability and factorial invariance across multiple assessment points. In longitudinal evaluation designs, repeated-measures instruments frequently suffer from construct drift or temporal instability. As detailed in the psychometric analyses conducted by Chiu and colleagues (2023), principal component analyses across all six daily measurement waves (Day 1 baseline morning through Day 5 post-simulation) demonstrated that the 13 items consistently load on a singular, dominant self-efficacy dimension. The factor structure retained strong conceptual coherence across time, confirming that the tool reliably measures the same latent professional confidence construct across the full duration of training.

3. Longitudinal Sensitivity to Change (Criterion Validity)

A crucial validity requirement for any training evaluation tool is sensitivity to experiential intervention (responsiveness). The DEST demonstrated significant, progressive upward trajectories in mean scores across training days. Novice investigators exhibited moderate-to-low baseline confidence on Day 1 morning (particularly on complex items such as testifying in court, framing protective custody, and identifying subtle caregiver deception). As simulation scenarios progressed, mean item scores systematically increased, accompanied by standard deviation compression, reflecting converging group competence and successful instructional mastery.

Reliability

The reliability of the Daily Experience of Simulation Training – Confidence Subscale (DEST-CS) has been demonstrated through internal consistency analyses across six distinct longitudinal assessment waves. In high-stakes human service measurement, an internal consistency coefficient exceeding .80 is considered strong, while values above .90 reflect gold-standard precision suitable for individual-level diagnostic and evaluative applications.

Across every single administration point, the Cronbach’s alpha coefficients of the multi-item confidence scale consistently exceeded .95, showing exceptional item homogeneity without problematic redundancy. Item-total correlations remained consistently high across all items, indicating that each measured skill contributes directly to the underlying self-efficacy continuum.

Factor Analysis

To evaluate the dimensionality and latent structural stability of the 13-item confidence subscale, Chiu et al. (2023) subjected the assessment data to exploratory Principal Component Analysis (PCA) with varimax rotation independently across each of the six measurement time points. This rigorous analytical design allowed researchers to observe whether the factor structure remained unified or fragmented as trainees gained experience.

1. Unifactorial Solution and Variance Explained

Across all six time points, the mathematical criteria (Kaiser-Guttman eigenvalue-greater-than-one rule and scree plot inspections) consistently identified a robust one-factor solution. The 13 items coalesce cleanly around a singular latent construct representing child protection investigative self-efficacy. The statistical parameters confirmed remarkable strength:

• Eigenvalues: The first extracted factor accounted for dominant eigenvalues ranging between 9.28 and 9.94 across the training cycle.
• Percentage of Variance: The primary factor explained between 71.4% and 76.5% of the total variance across all six administrations.
• Absence of Secondary Factors: Secondary components consistently exhibited eigenvalues substantially below 1.0, failing to capture meaningful residual variance and confirming the psychometric unidimensionality of the confidence scale.

2. Factor Loadings

All 13 items demonstrated salient, robust factor loadings on the primary component across time points. Loadings consistently surpassed .75 to .88 across items, reflecting high convergent validity. The consistency of these loadings over time illustrates structural invariance, confirming that the DEST-CS functions as an exceptionally stable tracking instrument across longitudinal training trajectories.

Instrument / Measurement Tool

The Daily Experience of Simulation Training is a multi-part, electronic assessment tool designed for daily web-based administration during intensive child welfare training academies. Below is the operational specification of the instrument:

• Test Type: Standardized self-report inventory, pedagogical feedback rating, and reflective log.
• Administration Method: Electronic delivery (responsive web survey accessible via laptops, desktop computers, tablets, or smartphones).
• Target Population: Adult learners (18+), specifically new child protection investigators, child welfare caseworkers, and protective service trainees.
• Total Number of Items: 20 items total, comprised of:
  • Administrative Site Identification: 1 item recording training campus/site.
  • Investigative Confidence Subscale (DEST-CS): 13 core items measuring specific competencies.
  • Pedagogical Feedback Appraisal: 5 items rating feedback helpfulness across instructional roles.
  • Metacognitive Narrative Log: 1 qualitative open-ended reflection item.
• Response Scales:
  • Confidence Subscale (Part I, Section A): 7-point Likert scale anchored from (1) Lowest to (7) Highest level of confidence today.
  • Feedback Subscale (Part I, Section B): 4-point Likert scale (very helpful, helpful, not helpful, very unhelpful) plus a Not Applicable (N/A) option.
  • Reflective Log (Part I, Section C): Narrative open-text field.
• Scoring and Indexing:
  • Confidence Composite Score: Calculated as the unweighted arithmetic mean or sum of Items 1 through 13. Summed scores range from 13 to 91; mean composite scores range from 1.00 to 7.00. Higher scores reflect greater task-specific self-efficacy.
  • Item-Level Diagnostic Profiling: Items are tracked individually over time (Days 1–5) to identify specific procedural gaps (e.g., lower average scores on “Testify in court” indicating a need for judicial debriefing reinforcement).
  • Feedback Utility Metrics: Percentages and categorical distributions are compiled per trainer role to evaluate instructional delivery.
• Implementation Guidelines (Part II): Five standardized operational protocols governing baseline delivery, daily afternoon timing, respondent privacy, email identifier sanitization, and browser cache troubleshooting.

Permissions & Fee and Test Year

• Test Year: 2023 (Initial publication and empirical validation by Chiu et al.).
• Intellectual Property & Permissions: The DEST was developed through the Children and Family Research Center (CFRC) at the University of Illinois at Urbana-Champaign in conjunction with the Child Protection Training Academy (CPTA) at the University of Illinois at Springfield, funded by the Illinois Department of Children and Family Services (DCFS).
• Commercial Status: Non-commercial instrument.
• Fee: Free for non-commercial educational, academic research, and public agency evaluation purposes.
• Permissions Inquiries: Educational institutions, human service agencies, and psychometric researchers seeking to adapt or deploy the scale should contact the corresponding author, Dr. Yu-Ling Chiu ([email protected]), or the publisher of the original validation study (Taylor & Francis / Journal of Public Child Welfare).

References

The theoretical, empirical, and psychometric foundations of the DEST are documented in the following scholarly references:

• Bandura, A. (1997). Self-efficacy: The exercise of control. W. H. Freeman and Company.
• Chiu, Y.-L., Cross, T. P., Wheeler, A. B., Evans, S. M., & Goulet, B. P. (2023). Development and application of a self-report measure for measuring change during simulation training in child protection. Journal of Public Child Welfare, 17(2), 239–257. https://doi.org/10.1080/15548732.2021.2016546
• Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406. https://doi.org/10.1037/0033-295X.100.3.363
• Goulet, B. P., Evans, S. M., Wheeler, A. B., Cross, T. P., & Chiu, Y.-L. (2020). Developing a simulation training academy in child welfare: The Child Protection Training Academy. Journal of Public Child Welfare, 14(4), 438–457. https://doi.org/10.1080/15548732.2020.1772422
• Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Prentice-Hall.

13. Items of the Scale (Questionnaire)