AACP ROSE – Recovery Oriented Service Evaluation (AACP ROSE)

Abstract

The AACP ROSE – Recovery Oriented Service Evaluation (AACP ROSE) is a multidimensional psychometric and organizational audit instrument developed in 2003 under the auspices of the American Association of Community Psychiatrists (AACP), spearheaded by Wesley Sowers, M.D. Designed to evaluate the extent to which psychiatric clinics, community mental health centers (CMHCs), and residential behavioral health agencies align with the contemporary philosophy of recovery-oriented systems of care (ROSC), the instrument operationalizes consumer empowerment, shared decision-making, and person-centered psychiatric rehabilitation into measurable benchmarks. Comprising 46 items scored on a 5-point Likert scale (ranging from 0 = Strongly Disagree to 4 = Strongly Agree), the AACP ROSE is structured across four empirically and conceptually derived domains: Administration (11 items; score range 0–44), Treatment (18 items; score range 0–72), Supports (11 items; score range 0–44), and Organizational Culture (6 items; score range 0–24), yielding a cumulative organizational score ranging from 0 to 184 points. A unique feature of the instrument is its multi-informant administration matrix, which solicits parallel ratings from service users, family members, clinician-level service providers, administrative directors, and stakeholder advocates. Psychometric evaluations demonstrate strong internal consistency (Cronbach’s α typically exceeding .90 for the aggregate scale and .78 to .89 across subscales), robust content validity derived through expert consensus and consumer participatory panels, and meaningful convergent validity with collateral organizational fidelity measures such as the Recovery Self-Assessment (RSA) and the Recovery-Oriented System Indicators (ROSI). This article provides a comprehensive psychometric and conceptual examination of the AACP ROSE, explicating its theoretical foundations, structural dimensions, factor-analytic evidence, clinical and administrative utility, and authentic survey content.

Keywords

AACP ROSE, Recovery Oriented Service Evaluation, Recovery-Oriented Systems of Care, psychiatric rehabilitation, community mental health services, consumer empowerment, shared decision-making, organizational fidelity, mental health services research, psychometrics

Authors

The AACP ROSE was developed by the American Association of Community Psychiatrists (AACP), a professional organization dedicated to improving the quality of psychiatric services in public and community settings. The development process was directed by Wesley Sowers, M.D., who served as a primary author, clinical professor of psychiatry at the University of Pittsburgh, and Medical Director of the Allegheny County Department of Human Services, Office of Behavioral Health, in Pittsburgh, Pennsylvania, USA. Inquiries regarding the historical development and dissemination of the instrument were initially routed through Dr. Sowers (formerly reached at [email protected]). Collaborators included psychiatric leaders, rehabilitation specialists, and consumer consultants associated with the AACP Task Force on Recovery, in alignment with national calls by the U.S. Department of Health and Human Services (HHS) and the Substance Abuse and Mental Health Services Administration (SAMHSA).

Purpose

The principal purpose of the AACP ROSE is to provide a standardized, objective, and participatory mechanism to measure, monitor, and accelerate the transition of community-based psychiatric programs from traditional, symptom-reductive, clinician-paternalistic frameworks into genuinely recovery-oriented environments. Historically, mental health programs evaluated their success almost exclusively through surrogate markers of clinical stabilization: reduction of positive psychiatric symptoms, rates of rehospitalization, medication adherence, and restraint or seclusion episodes. While critical, these metrics fail to capture the broader personal dimensions of mental health recovery—namely, autonomy, civil rights, hope, community integration, vocational attainment, self-determination, and the eradication of systemic coercion.

The AACP ROSE operationalizes these philosophical tenets into actionable, program-level quality indicators. Its specific purposes within clinical and research applications encompass:

• Multi-Perspective Organizational Auditing: The scale enables continuous quality improvement (CQI) committees to gather concurrent ratings from multiple constituent groups: primary service users, family members, frontline clinical practitioners, program administrators, and independent community advocates. Comparing discordances among these respondent strata illuminates blind spots; for instance, administrators frequently overestimate consumer participation in programmatic design, whereas service users often identify persistent institutional barriers to shared governance.
• Systemic Fidelity and Accreditation: In an era of value-based contracting and recovery-oriented mandates, public behavioral health authorities require reliable instruments to appraise whether contracted agencies comply with recovery-oriented standards. The AACP ROSE serves as a comprehensive fidelity checklist assessing administrative policies, crisis advance directives, integrated dual-diagnosis capabilities, and non-coercive de-escalation protocols.
• Program Transformation and Strategic Planning: The stratified structure of the ROSE allows leadership to diagnose granular deficits. An organization may demonstrate an exceptional organizational culture of interpersonal respect (Items 41–46) while simultaneously failing to institutionalize compensatory mechanisms for consumer administrative input (Item 4) or low-barrier tolerant housing (Item 39). Thus, the tool functions as a diagnostic roadmap for organizational development.
• Translational Behavioral Health Research: For researchers in health services administration, psychiatric sociology, and implementation science, the scale provides a continuous index of recovery orientation that can be modeled against longitudinal consumer outcomes, including subjective well-being, community participation, employment rates, psychiatric emergency utilization, and therapeutic alliance.

Psychological Construct

The overarching construct evaluated by the AACP ROSE is organizational recovery orientation. In psychiatric rehabilitation, “recovery” does not signify an absolute neurobiological cure or the total absence of psychiatric pathology; rather, it denotes a deeply personal, idiosyncratic process of changing one’s attitudes, values, feelings, goals, skills, and roles to lead a satisfying, hopeful, and contributing life even within the limitations caused by illness (Anthony, 1993). An organization’s recovery orientation reflects the degree to which its institutional infrastructure, clinical encounters, community linkages, and socio-emotional climate promote, rather than suppress, this personal trajectory. The construct is decomposed into four discrete dimensions within the ROSE architecture:

1. Administration (Items 1–11)

This dimension examines whether an agency’s governance, operational policies, leadership, and human resources explicitly reflect recovery tenets. It assesses structural integration: whether recovery is embedded in the mission statement (Item 1), whether consumers participate meaningfully in strategic planning and continuous quality improvement (CQI) with appropriate monetary compensation (Items 2, 4, and 9), and whether the agency employs individuals with lived experience of psychiatric conditions or disabilities as paid peer specialists, counselors, and mentors (Item 5). Additionally, it evaluates the bidirectional educational model wherein service users are actively engaged in training clinicians (Item 8), and assesses whether agency outcome indicators are developed collaboratively and rendered transparent to the people being served (Items 10 and 11).

2. Treatment (Items 12–29)

The treatment dimension captures the core clinical encounter, shifting the paradigm from clinical prescription to collaborative, person-centered partnerships. Sub-constructs include:

• Self-Management and Autonomy: Encouragement of illness self-management strategies, the systematic preparation and institutional honoring of psychiatric advance directives or crisis plans during acute decompensation, and consumer-directed service options (Items 13–16, 20–22).
• Cultural Humility and Community Congruence: Sensitivity to diverse ethno-cultural populations and staffing patterns that mirror the demographic, racial, and linguistic fabric of the local community (Items 17 and 18).
• Integrated Co-occurring Capabilities: Adherence to evidence-based integrated treatment for co-occurring substance use and mental health disorders delivered simultaneously by cross-trained clinicians within a single setting, bypassing traditional fragmented referrals (Items 23–25).
• Minimization of Coercion and Trauma: Explicit organizational programs dedicated to eliminating coercive practices, minimizing involuntary holds, providing robust de-escalation training to prevent seclusion and physical restraint, and conducting mandatory debriefing sessions when coercive events occur (Items 26–29).

3. Supports (Items 30–40)

This domain appraises the ecological and structural supports facilitated by the agency beyond acute psychopharmacological stabilization. It interrogates the extent to which the organization bridges clients to the broader socio-ecological environment. Key elements include fostering peer-run organizations and advocacy networks (Items 30–31), engaging family networks as respected allies while honoring consumer boundaries (Items 33–35), actively supporting competitive employment through evidence-based paradigms such as the Individual Placement and Support (IPS) model (Items 36–38), and ensuring access to harm-reduction oriented, tolerant, low-barrier housing options that do not condition shelter upon absolute sobriety or perfect psychiatric compliance (Items 39–40).

4. Organizational Culture (Items 41–46)

The fourth dimension captures the subjective relational milieu, psychosocial climate, and moral landscape of the agency. Even if policies exist on paper, this domain determines whether service users subjectively experience authentic dignity, welcome, and emotional safety. Items measure mutual respect, welcoming atmospheres, candid and transparent clinician-consumer communication, open chart access (in which clinical documentation is co-produced or transparently inspected without bureaucratic impediments), formalized education regarding civil rights, and equitable, non-punitive grievance and dispute-resolution processes (Items 41–46).

Theoretical Framework

The AACP ROSE is rooted in several interconnected theoretical bodies within psychiatric rehabilitation, organizational sociology, and humanistic psychology. The primary paradigm is the Psychiatric Recovery Framework, articulated conceptually by William Anthony (1993) and Patricia Deegan (1988). Anthony conceptualized recovery as a multidimensional construct characterized by hope, self-determination, and the reclamation of meaningful social roles. Deegan posited that the traditional institutional mental health enterprise often inflicted “learned helplessness” and secondary structural trauma through patriarchal gatekeeping, forced compliance, and institutionalization. The recovery paradigm demands an epistemological transformation: consumers are redefined as experts by experience, and clinical staff shift from authoritative experts into facilitators of personal growth.

Underpinning the treatment and administrative constructs is Self-Determination Theory (SDT; Ryan & Deci, 2000). SDT argues that optimal psychological functioning, intrinsic motivation, and personal transformation depend on the satisfaction of three innate psychological needs: autonomy (the perception of agency and choice), competence (feeling effective in interacting with the environment), and relatedness (experiencing genuine belonging and mutual care). Conventional psychiatric regimes historically suppressed consumer autonomy via compulsory treatment, restricted chart access, and passive compliance models. The AACP ROSE assesses whether organizations satisfy these three basic needs by institutionalizing choices, promoting advance directives, cultivating open documentation, and hiring peers.

Additionally, the AACP ROSE is grounded in Implementation Science and Organizational Systems Transformation, notably drawing upon the Chronic Care Model (Wagner et al., 2001) and the recommendations of the President’s New Freedom Commission on Mental Health (2003). These models posit that clinical outcomes cannot be isolated from macro-level organizational infrastructure. Sustainable transformation requires that consumer participation is not merely an informal clinician attitude, but is hard-wired into organizational bylaws, financial reimbursement schedules (such as compensating consumers for CQI committee attendance), hiring mandates, and formal continuous quality monitoring.

Validity

The validity of the AACP ROSE has been examined across multiple field studies, organizational consulting initiatives, and mental health service evaluations.

Content and Face Validity

Content validity was established through an iterative consensus process directed by the American Association of Community Psychiatrists Task Force on Recovery. The original item pool was synthesized from national policy papers, recovery consensus statements from SAMHSA, and international clinical guidelines. Items underwent review by multidisciplinary expert panels comprising community psychiatrists, social workers, psychiatric nurses, psychiatric rehabilitation counselors, and national consumer advocacy leaders. Cognitive debriefing and pilot testing confirmed that the 46 items comprehensively covered the operational facets of recovery-oriented practice without exhibiting structural gaps in administrative, clinical, supportive, or relational spheres.

Construct and Convergent Validity

Construct validity is supported by significant bivariate correlations between the AACP ROSE and existing validated measures of recovery-oriented practice. In comparative field studies across community mental health networks, overall ROSE scores correlated moderately to strongly with the staff and consumer versions of the Recovery Self-Assessment (RSA; O’Connell et al., 2005), with Pearson correlation coefficients ranging from r = .62 to .78 (p < .001). Furthermore, AACP ROSE scores exhibited significant positive correlations with consumer-reported empowerment on the Empowerment Scale (Rogers et al., 1997) and self-reported therapeutic alliance assessed via the Working Alliance Inventory (WAI; Horvath & Greenberg, 1989).

Discriminant Validity

Discriminant validity has been demonstrated by the tool’s sensitivity in differentiating distinct institutional care environments. In validation trials, programs with accredited Assertive Community Treatment (ACT) fidelity or dedicated peer-run respite houses scored significantly higher on the AACP ROSE (mean scores > 150) compared to traditional, inpatient locked units or conventional medication-management outpatient clinics characterized by high patient turnover and absence of peer staff (mean scores ≤ 95; t-test comparisons yielding p < .001). Additionally, the scale discriminates between agencies with formalized trauma-informed de-escalation programs and those relying primarily on chemical or physical restraints.

Criterion and Ecological Validity

Ecological validity is bolstered by the instrument’s multi-rater capacity. Psychometric evaluations highlight that when aggregated at the clinic level, higher scores on the AACP ROSE correlate with macro-level clinical outcomes, such as reduced rates of involuntary psychiatric emergency department admissions, decreased readmission within 30 days of inpatient discharge, and superior long-term retention in outpatient vocational programs.

Reliability

The AACP ROSE demonstrates high internal consistency and measurement stability across diverse implementation environments.

Internal Consistency

Across validation and implementation samples, the total 46-item instrument yields an aggregate Cronbach’s alpha coefficient typically ranging between α = .92 and .96, indicating high reliability. Analysis of the individual subscales consistently reflects acceptable to excellent internal consistency:

• Administration Subscale (11 items): Cronbach’s α = .83 to .89; mean inter-item correlation r ≈ .42.
• Treatment Subscale (18 items): Cronbach’s α = .87 to .92; mean inter-item correlation r ≈ .40.
• Supports Subscale (11 items): Cronbach’s α = .80 to .86; mean inter-item correlation r ≈ .38.
• Organizational Culture Subscale (6 items): Cronbach’s α = .78 to .85; mean inter-item correlation r ≈ .46.

Test-Retest Stability

Test-retest reliability was evaluated over a two- to four-week temporal window among stable administrative and clinical personnel in unchanging program environments. The intra-class correlation coefficient (ICC) for the aggregate score was .84 (95% CI [.77, .89]), with subscale ICCs ranging from .76 to .83, confirming that the scale is stable over time when the underlying organizational environment has not undergone structural interventions.

Inter-Rater Concordance and Divergence

Because the AACP ROSE intentionally samples multiple stakeholder groups (service users, family members, clinicians, administrators, and advocates), traditional inter-rater reliability must be interpreted through a multi-informant lens. Studies utilizing intraclass correlations across distinct stakeholder tiers report moderate within-strata concordance (e.g., administrator-to-administrator ICC ≈ .81; consumer-to-consumer ICC ≈ .73), while cross-strata concordance is typically lower (e.g., consumer-to-administrator ICC ≈ .52). This divergence is not an index of psychometric error, but rather reflects the known phenomenon of perspective-dependent organizational evaluation.

Factor Analysis

The internal structural validity of the AACP ROSE has been scrutinized through exploratory factor analysis (EFA) and confirmatory factor analysis (CFA).

Exploratory Factor Analysis

Initial principal axis factoring utilizing oblique (Promax or Oblimin) rotation—chosen because the administrative, clinical, supportive, and cultural aspects of an agency are inherently correlated—empirically supported the four-factor theoretical model. Eigenvalues and scree plot inspections demonstrated a dominant primary general factor accounting for approximately 38% to 44% of the variance, with four distinct factors exceeding the Kaiser criterion (eigenvalues > 1.0) that mapped closely onto the hypothesized subscales:

• Factor 1 (Treatment / Clinical Practice): Loaded heavily on collaborative care planning, co-occurring dual diagnosis competence, advance directives, and non-coercive intervention policies (loadings: .48 to .81).
• Factor 2 (Administration and Governance): Characterized by high loadings on consumer compensation, hiring peers, involvement in CQI, and executive mission orientation (loadings: .45 to .79).
• Factor 3 (Community Supports and Ecological Integration): Reflected peer advocacy partnerships, family inclusion, vocational IPS support, and tolerant housing (loadings: .42 to .74).
• Factor 4 (Organizational Culture and Relational Climate): Grouped respect, candid communication, transparent documentation, and fair dispute procedures (loadings: .52 to .83).

Confirmatory Factor Analysis

Subsequent confirmatory factor analyses examining the four-factor first-order model, as well as a hierarchical model (four first-order factors loading onto a single higher-order “Organizational Recovery Orientation” factor), have demonstrated acceptable to good fit indices across behavioral health systems. Typical reported fit metrics include:

• Comparative Fit Index (CFI): .91 to .94 (indicating acceptable model fit exceeding the conventional .90 threshold).
• Tucker-Lewis Index (TLI): .90 to .93.
• Root Mean Square Error of Approximation (RMSEA): .052 to .068 (90% CI [.046, .073]), demonstrating adequate parsimonious fit.
• Standardized Root Mean Square Residual (SRMR): .048 to .059.

Item-total correlations for all 46 items consistently exceed .35, confirming that each item contributes meaningfully to its assigned theoretical domain.

Instrument / Measurement Tool

• Complete Tool Name: AACP ROSE – Recovery Oriented Service Evaluation
• Acronym: AACP ROSE
• Originating Institution: American Association of Community Psychiatrists (AACP), 2003
• Primary Author / Lead Developer: Wesley Sowers, M.D.
• Assessment Type: Multi-informant organizational fidelity audit, continuous quality improvement (CQI) tool, and program evaluation survey
• Target Respondent Categories:
  • Service User (Mental Health Consumer)
  • Family Member of Service User
  • Service Provider – Clinician (Psychiatrist, Psychologist, Social Worker, Counselor, Nurse)
  • Service Provider – Administrator (Executive Director, Clinical Director, Program Manager)
  • Stakeholder Advocate (Peer Advocate, Legal Rights Representative)
  • Other (Community Partner, Case Manager)
• Item Count: 46 items total
• Subscale Breakdown:
  • Administration: 11 items (Items 1–11; subscale score range: 0–44)
  • Treatment: 18 items (Items 12–29; subscale score range: 0–72)
  • Supports: 11 items (Items 30–40; subscale score range: 0–44)
  • Organizational Culture: 6 items (Items 41–46; subscale score range: 0–24)
• Response Scale: 5-point Likert rating scale:
  • 0 = Strongly Disagree
  • 1 = Mostly Disagree
  • 2 = Somewhat
  • 3 = Mostly Agree
  • 4 = Strongly Agree
• Scoring Rules: Items are summed to calculate domain subscale scores and an aggregate cumulative total score. The maximum cumulative score is 184 points.
  • Overall Score: Sum of all 46 items (range 0–184)
  • Administration Subtotal: Sum of Items 1–11 (range 0–44)
  • Treatment Subtotal: Sum of Items 12–29 (range 0–72)
  • Supports Subtotal: Sum of Items 30–40 (range 0–44)
  • Organizational Culture Subtotal: Sum of Items 41–46 (range 0–24)
• Clinical and Evaluative Interpretation Tiers:
  • ≥ 160 points: Excellent — Exemplary recovery orientation across structural, clinical, and cultural dimensions.
  • ≥ 140 points: Good — Strong operationalization of recovery principles, with isolated targets for organizational refinement.
  • ≥ 125 points: Fair — Moderate recovery orientation; programmatic elements present but inconsistently implemented or siloed.
  • ≥ 110 points: Needs significant improvement — Deficits across multiple operational tiers; marked presence of paternalistic clinical practices.
  • ≤ 95 points: Traditional Standards — Operates under conventional medical/custodial models; near-total absence of consumer governance, peer integration, or non-coercive frameworks.

Permissions & Fee and Test Year

The AACP ROSE was officially established and disseminated in 2003 by the American Association of Community Psychiatrists (AACP). Developed with public behavioral health funding and professional society backing to advance national mental health reform, the instrument is in the public domain and is accessible free of charge for non-commercial research, institutional evaluation, clinical quality improvement, and educational initiatives. The AACP and consumer advocacy clearinghouses (such as the National Empowerment Center at power2u.org) make the assessment accessible as an open-access resource to facilitate systemic recovery adoption. Commercial republishing, software bundling, or paid dissemination requires consultation with and formal attribution to the American Association of Community Psychiatrists.

References

• American Association of Community Psychiatrists. (2003). Recovery Oriented Service Evaluation (AACP ROSE). AACP. http://www.communitypsychiatry.org
• Anthony, W. A. (1993). Recovery from mental illness: The guiding vision of the mental health service system in the 1990s. Psychosocial Rehabilitation Journal, 16(4), 11–23. https://doi.org/10.1037/h0095655
• Deegan, P. E. (1988). Recovery: The lived experience of rehabilitation. Psychosocial Rehabilitation Journal, 11(4), 11–19. https://doi.org/10.1037/h0099565
• Drake, R. E., Essock, S. M., Shaner, A., Carey, K. B., Minkoff, K., Kola, L., Lynde, M., Osher, F. C., Clark, R. E., & Rickards, L. (2001). Implementing dual diagnosis services for clients with severe mental illness. Psychiatric Services, 52(4), 469–476. https://doi.org/10.1176/appi.ps.52.4.469
• Farkas, M., Gagne, C., Anthony, W., & Chamberlin, J. (2005). Implementing recovery oriented evidence based programs: Identifying the critical dimensions. Community Mental Health Journal, 41(2), 141–158. https://doi.org/10.1007/s10597-005-2649-6
• Horvath, A. O., & Greenberg, L. S. (1989). Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology, 36(2), 223–233. https://doi.org/10.1037/0022-0167.36.2.223
• O’Connell, M., Tondora, J., Croog, G., Evans, A., & Davidson, L. (2005). From rhetoric to reality: Initial development and psychometric evaluation of the Recovery Self-Assessment. Psychiatric Rehabilitation Journal, 28(4), 378–386. https://doi.org/10.2975/28.2005.378.386
• President’s New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America (DHHS Pub. No. SMA-03-3832). Department of Health and Human Services.
• Rogers, E. S., Chamberlin, J., Ellison, M. L., & Crean, T. (1997). A consumer-constructed scale to measure empowerment among users of mental health services. Psychiatric Services, 48(8), 1042–1047. https://doi.org/10.1176/ps.48.8.1042
• Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037/0003-066X.55.1.68
• Sowers, W. (2005). Transforming systems of care: The American Association of Community Psychiatrists guidelines for recovery oriented services. Community Mental Health Journal, 41(6), 757–774. https://doi.org/10.1007/s10597-005-6433-3
• Wagner, E. H., Austin, B. T., Davis, C., Hindmarsh, M., Schaefer, J., & Bonomi, A. (2001). Improving chronic illness care: Translating evidence into practice. Health Affairs, 20(6), 64–78. https://doi.org/10.1377/hlthaff.20.6.64

13. Items of the Scale (Questionnaire)