A Process and Outcome Evaluation of the 4-H Mentoring/Youth and Families with Promise (YFP) Program [Utah, 2005-2010] (ICPSR 33781)

Principal Investigator(s): Orchowsky, Stan, Justice Research and Statistics Association; Poulin, Mary, Justice Research and Statistics Association

Summary:

This study was an evaluation of Utah's 4-H Mentoring/Youth and Families with Promise (YFP) program, a statewide 4-H mentoring program designed for at-risk youths. Both the degree to which the YFP program was implemented and operated as planned and the long term outcome of program participation and sustainability of program effects were assessed.

The process evaluation addressed (1) characteristics of the population served (risk factors, protective factors, socio-demographic characteristics); (2) type and dosage of program activities; (3) mentor-mentee relationships; (4) satisfaction with the program; (5) involvement of youths' family members in the program; (6) characteristics of mentors (e.g., type of mentor, sex, race); and (7) issues related to the youths' length of stay.

The outcome evaluation consisted of an experimental group of at-risk youth 10-14 years old who were paired with a mentor, participated in 4-H activities, and also attended Family Night Out (FNO) activities to strengthen family bonds. A comparison group was constructed of students of the same age attending the same schools but who did not participate in the YFP program. The outcome evaluation measured whether the YFP program increased youths' interpersonal competence, improved youths' academic performance, strengthened youths' family relationships, and prevented delinquency. Outcome indicators were measured at the start of, during, and up to three years after program involvement. In addition to self-reporting pre/post tests, official school and court records were recorded. Additionally, youths' perceptions of mentor relationship quality were measured with the Youth-Mentor Relationship Questionnaire.

Access Notes

  • One or more files in this data collection have special restrictions ; consult the restrictions note to learn more. You can apply online for access to the restricted-use data. A login is required to apply.

    Access to these data is restricted. Users interested in obtaining these data must complete a Restricted Data Use Agreement, specify the reasons for the request, and obtain IRB approval or notice of exemption for their research

    Any public-use data files in this collection are available for access by the general public. Access does not require affiliation with an ICPSR member institution.

Dataset(s)

DS0:  Study-Level Files
Documentation:
DS1:  Youth File
Documentation:
Download:
No downloadable data files available.
DS2:  Mentor Surveys
Documentation:
Download:
No downloadable data files available.

Study Description

Citation

Orchowsky, Stan, and Mary Poulin. A Process and Outcome Evaluation of the 4-H Mentoring/Youth and Families with Promise (YFP) Program [Utah, 2005-2010]. ICPSR33781-v1. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2015-12-23. https://doi.org/10.3886/ICPSR33781.v1

Persistent URL: https://doi.org/10.3886/ICPSR33781.v1

Export Citation:

  • RIS (generic format for RefWorks, EndNote, etc.)
  • EndNote XML (EndNote X4.0.1 or higher)

Funding

This study was funded by:

  • United States Department of Justice. Office of Justice Programs. National Institute of Justice (2005-IJ-CX-0046)

Scope of Study

Subject Terms:    program evaluation, youthful offenders, youths at risk

Smallest Geographic Unit:    program site

Geographic Coverage:    United States, Utah

Time Period:   

  • 2005--2010

Date of Collection:   

  • 2005--2010

Unit of Observation:    Individual

Universe:    Youths served by the 4-H Mentoring/Youth and Families with Promise program who attended various school districts in Utah from 2005-2010, and a comparison group of youths similarly aged to program youths was measured by the outcome evaluation. Additionally, the YFP programs themselves served as a the focus of process evaluation.

Data Type(s):    administrative records data, experimental data, observational data, survey data

Methodology

Study Purpose:   

The purpose of this study was to evaluate the success and effectiveness of the 4-H Mentoring/Youth and Families with Promise (YFP) Program. The degree to which the program was implemented and operated as planned was assessed as well as measuring its impact on at-risk youth, including the long-term outcomes of program participation and the sustainability of those effects.

Study Design:   

The process evaluation of Utah's 4-H Mentoring/Youth and Families with Promise (YFP) program assessed the degree to which the program was implemented and operated as planned both as a whole as well as measuring variances across program site locations. The assessment addressed characteristics of the population, mentor-mentee relationships, characteristics of mentors, issues related to length of youth stay, type of program activities, and familial involvement of youth in the program. These were measured using the following tools:

  • Site coordinator interviews and dosage data
  • Observations of program activities
  • Mentor surveys
  • Interviews and surveys of program staff
  • Youths interviews
  • Socio-demographic information
  • Program documentation were used to assess program implementation (manuals).

The longitudinal outcome evaluation of the YFP program collected data over three waves:

Wave 1: Pretest -- administered at the start of the program, usually at the beginning of the school year. Collected for YFP youths, parents, and the comparison group (similarly aged youths who had no involvement with the program).

Wave 2: Post-test -- administered when the youth completed the program, or at the end of the school year. If the youth stayed in the program longer than one year, then subsequent post-tests were administered until the youth was no longer in the program (typically 8-32 months from program start). Survey data for the youths in the control group was collected at the end of the school year.

Wave 3: Follow-Up -- telephone interviews were conducted with YFP and comparison youths annually up to three years after program discharge/date of last post-test for treatment group and after last post-test for the comparison group. They were used to assess current academic performance, interpersonal competence, family relationships, and delinquency/risky behavior as well as perceptions of YFP program involvement (in YFP youths only).

The outcome indicators were measured using:

  • The Behavioral and Emotional Rating Scale Version 2 (BERS-2)
  • Official school and court records
  • Goals that youth and parents had about participation in the program
  • Socio-demographic and background information
  • Youth self-reports of behavior and perceptions of the program

The at-risk youth surveys were typically administered at YFP or FNO events, mailed, or taken during a home visit by staff. The comparison group students took the tests at school.

Sample:    The treatment group included youths and their parents served by the YFP program from 2005-2010. The comparison group was a purposive sample of similarly aged school students.

Time Method:    Longitudinal: Cohort/ Event-based , Longitudinal: Panel

Mode of Data Collection:    record abstracts, coded on-site observation, face-to-face interview, mail questionnaire, on-site questionnaire, telephone interview

Data Source:

Outcome evaluation: Pre- and post-program self-reported surveys of delinquency behavior were administered to the experimental and comparison youths and parents of experimental youths.

Outcome evaluation: Follow-up surveys up to three years after the YFP program.

Outcome evaluation: YFP mentor surveys

Outcome evaluation: Utah Administrative Office of the Courts records

Outcome evaluation: Official school records

Process evaluation: Site coordinator interviews, surveys and reports about the implementation of YFP at their site

Process evaluation: Observations of program activities

Process evaluation: Interviews with youth

Process evaluation: YFP program guide and mentor manual

Description of Variables:   

There are 741 cases and 1,184 variables in data set 1 including information from multiple time periods (pre, post, and follow-up surveys) and cohorts. There are various categories of assessment including:

  1. The Behavioral and Emotional Rating Scale Version 2 (BERS-2), a 52 item scale, used to create a multitude of variables that measured the behavioral and emotional strengths of children. It was administered during Pre and Post tests and is comprised of five subscales:

    • Interpersonal Strength -- ability to control emotions or behaviors in social situations -- 15 items
    • Intrapersonal Strength -- outlook on competence and accomplishments -- 11 items
    • Affective Strength -- ability to accept affection from and express feelings toward others -- 7 items
    • Family Involvement -- participation in and the relationship with family -- 10 items
    • School Functioning -- competence in school and classroom tasks -- 9 items
  2. Career Strength Scale -- This measured interest and aptitude for career and vocational development -- 5 items

  3. Youth Mentor Relationship Questionnaire -- 20 item scale with four subscales "not dissatisfied", "helped to cope", "not unhappy" "trust not broken". This was measured on a scale of 1-4 with higher scores corresponding to more successful relationships.

  4. Several variables identify delinquent behavior. The Utah Administrative Office of the Courts provided information on each incident which came to the attention of the court for each youth, including the incident date, charges, intake decision, and conviction. Self-reported delinquent behavior was collected in the pre- and post-tests on the number of times the behavior occurred.

  5. Socioeconomic and family status questions ranging from whether the youth qualifies for free lunch, who is living with the youth, government services that are being received are also present in the data. Responses are reported through scales and direct wording used by the youth.

  6. Date variables -- Date variables were collected for pre- and post-test completion, discharge date, and study discharge date.

  7. School Data -- School absences and GPA records.

  8. Court Data recording delinquent incidences.

  9. Analysis comparing pre- and post-test score changes for BERS-2 scale.

  10. Variables considering the matching of youth to mentor, including the reason the match occurred, the date the match occurred, and whether the youth unmatched to the mentor and why.

  11. Variables concerning the date the youth and mentor manuals were issued, and the date the parent was given their guide.

  12. Variables considering the type of mentor that the youth was given, and whether the mentor attended orientation.

Data Set 2 consists of 91 cases and 104 variables representing survey data from the YFP program mentors. Questions focused on the mentor's length of service in YFP, the matching that took place with the mentee, interactions with their mentee, and their evaluation of mentee success.

Response Rates:   

The response rate varied by data collection point. There were 401 youths in the treatment group and 327 in the comparison group.

Experimental group

  • Pre-test: 392
  • Post-Test 1 Only (no Pre-Test): 9
  • Pre-Test and Post-Test 1: 270
  • Pre-Test and Post-Test 2: 98
  • Pre-Test, Post-Test 1 and Post-Test 2: 93
  • Post-Test 3: 28
  • Follow-up interview: year one: 40 percent, year 2: 31 percent, year 3: 30 percent
  • School data: 39 percent
  • Court data: 82 percent

Comparison Group

  • Pre-test: 327
  • Pre-Test and Post-Test 1: 280
  • Pre-Test and Post-Test 2: 144
  • Pre-Test, Post-Test 1 and Post-Test 2: 131
  • Follow-up interview: year one: 37 percent, year 2: 49 percent, year 3: 36 percent
  • School data: 73 percent
  • Court data: 98 percent

Presence of Common Scales:    Behavioral and Emotional Rating Scale (BERS) - 2nd edition Youth-Mentor Relationship Questionnaire (YMRQ)

Extent of Processing:   ICPSR data undergo a confidentiality review and are altered when necessary to limit the risk of disclosure. ICPSR also routinely creates ready-to-go data files along with setups in the major statistical software formats as well as standard codebooks to accompany the data. In addition to these procedures, ICPSR performed the following processing steps for this data collection:

  • Created variable labels and/or value labels.
  • Standardized missing values.
  • Performed recodes and/or calculated derived variables.
  • Checked for undocumented or out-of-range codes.

Version(s)

Original ICPSR Release:   2015-12-23

Version History:

  • 2016-01-13 Updated User Guide.

Related Publications

Utilities

Metadata Exports

If you're looking for collection-level metadata rather than an individual metadata record, please visit our Metadata Records page.

Download Statistics