Formative Evaluation of a Technology-Based Behavioral Health Program for Victims of Crime, North Carolina, 2019-2021 (ICPSR 38261)

Time Period(s) View help for Time Period(s)

Date of Collection View help for Date of Collection

Data Collection Notes View help for Data Collection Notes

Study Purpose View help for Study Purpose

This Phase 1 study sought to systematically assess, understand, and address several challenges around access to services faced by victims of crime (VOCs) from marginalized populations, including Latinos and those in rural communities and urban areas, through a formative evaluation of a behavioral health model of telehealth services for VOCs implemented by El Futuro. The study included three overall goals:

1. Formative Evaluation: Evaluability and Logic Model Development. The goal of the evaluability assessment (EA) was to examine whether El Futuro's current approach and design could answer overarching questions about its program design, implementation needs with the current design, and questions related to data being captured for adequate and informed evaluation.
2. Implementation: Guide Development and Pilot. For this goal, the focus was to assess discrepancies between the implementation plan and execution, exploring issues of implementation of interventions for a variety of VOCs. This included what implementation components looked like, as well as considerations for fidelity, intensity, and exposure.
3. Post-Implementation: Interpretation. For this goal, the research team assessed the usefulness and value of telehealth from El Futuro stakeholder perspectives by eliciting stakeholder recommendations for further telehealth intervention refinements. They also assessed victim/client satisfaction with the intervention and implementation process and identified barriers and facilitators.

Pilot study guiding questions were focused on issues around implementation, including:

1. What are the barriers to and facilitators of engaging the targeted population in TeleFuturo, including each method of service delivery? Are the VOCs who need TeleFuturo accessing it? Why or why not?
2. During implementation of each TeleFuturo component, what are the barriers and facilitators experienced by providers?
3. How does the use of technology-based methods to implement evidence-based practice affect implementation in comparison to traditional service implementation for VOCs?
4. What are the hours and burden on providers associated with implementing TeleFuturo for VOCs? Is it sustainable without outside funding?
5. Were potential harms of using technology-based services with VOCs (e.g., breaches in confidentiality, suicidality, and lethality concerns) mitigated?

Study Design View help for Study Design

The research team conducted a rigorous formative evaluation of El Futuro's technology-based telemental health program for victims of crime (VOCs) in vulnerable populations, including Latinos and those in rural settings. El Futuro named this menu of technology-based programming TeleFuturo. TeleFuturo was formatively evaluated and the research team developed a logic model and implementation guide as part of Goals 1 and 2 of this study phase. As part of the evaluability assessment (EA), the research team assessed staff capacity to implement TeleFuturo services. They gathered information related to staffing availability and staff capacity, commitment and support from staff and other important stakeholders, organizational culture, and practice.

In-depth interviews were conducted via Zoom with El Futuro TeleFuturo staff, including the Executive Director, Clinical Program Director (oversees service providers and implementation of services), Director of Grants and Strategic Development (oversees business development and fundraising), and Director of Operations (oversees data management). The Executive Director and Clinical Program Director completed a 1-hour interview relating to organizational infrastructure, implementation approach, financial resources, program monitoring, and data collection procedures. They were asked questions about regular data collection procedures, data management, program design and implementation, populations served, available resources, and evaluation capacity of all staff. The research team proposed several data collection efforts to El Futuro leadership to determine the evaluation and training and technical assistance needs for TeleFuturo based on their procedures already in place for face-to-face programming.

A video interview was conducted with the Director of Grants and Strategic Development to capture information about determining the needs of El Futuro in areas related to business development, grant writing, marketing, and logistics. The research team also interviewed the Director of Operations to understand the organization and capacity of the data collection systems within El Futuro including electronic health records (EHRs), behavioral health treatment data, and diagnostic tools and measures.

Information from the initial leadership interviews was used to develop the telehealth service provider interview protocols. El Futuro's Clinical Program Director provided a list of potential respondents to the research team, including the Clinical Supervisor and Clinic Manager and other service providers that represented a variety of perspectives regarding evaluation, program implementation, and data collection and the different intervention types: psychiatry, psychotherapy, case management, youth-focused, and adult-focused. Service providers completed a 30-minute to 1-hour interview about their role in the organization, responsibilities around data collection and documentation, treatment services provided, and evidence-based practices delivered.

Using data from El Futuro's larger client population, the research team developed the Depression Anxiety Stress Scales (DASS) face-to-face and telehealth using graded response item response theory that was designed to maintain maximum reliability across the clinical range of mental health treatment seekers. Results indicated minimal loss in reliability in reducing the DASS to six items. The DASS-21 and DASS-6 were used for pilot results with the intention of using the DASS for telehealth (six-item version) in Phase 2 and beyond.

For the pilot study evaluation questions for Phase 1, El Futuro providers drew on clinical encounters of 2,623 clients since October 2019, and 1,017 of those clients were VOCs who received telehealth services. After the sites closed because of the COVID-19 pandemic, all clients including VOCs in the rural sites and the main El Futuro site began receiving therapy using telehealth, including clients who had previously participated in face-to-face services. The overall pilot included 12,413 clinical encounters, the majority of which were teletherapy.

Time Method View help for Time Method

Universe View help for Universe

Victims of crime over the age of 18 years presenting to a mental health clinic for mental health services.

Unit(s) of Observation View help for Unit(s) of Observation

Data Type(s) View help for Data Type(s)

Mode of Data Collection View help for Mode of Data Collection

Presence of Common Scales View help for Presence of Common Scales

A shortened version of the Depression Anxiety Stress Scales (DASS)-21, the DASS-6 was used to examine improvement after telehealth sessions. The DASS-6 is a tool for assessing depression, anxiety, and stress in both English and Spanish in telehealth settings.

• DASS-21: Henry JD, Crawford JR. The short-form version of the Depression Anxiety Stress Scales (DASS-21): construct validity and normative data in a large non-clinical sample. Br J Clin Psychol. 2005 Jun;44(Pt 2):227-39. doi: 10.1348/014466505X29657. PMID: 16004657.

Original Release Date View help for Original Release Date

2024-08-28

Version History View help for Version History