Which Program Characteristics Are Linked to Program Impacts? Lessons from the HPOG 1.0 Evaluation

In 2010, the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services awarded the first round of five-year grants from the Health Profession Opportunity Grants (HPOG 1.0) Program to 32 organizations in 23 states; five were tribal organizations. The purpose of the HPOG Program is to provide education and training to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals for occupations in the healthcare field that pay well and are expected to either experience labor shortages or be in high demand. HPOG 1.0 grantees designed and implemented programs to provide eligible participants with education, occupational training, support, and employment services to help them train for and find jobs in a variety of healthcare professions.

The impact evaluation of HPOG 1.0 randomized 13,717 individuals into treatment or control groups across 42 HPOG programs operated by 23 non-tribal grantees. Members of the treatment group could access the HPOG Program; members of the control group could not. The difference in outcomes between the treatment and control groups is HPOG’s impact. Each program developed and implemented its own HPOG model based on ACF guidelines and its own decisions about which program features and supports would be most effective to help its participants complete training.

This paper uses variation in program characteristics—including program components, implementation features, local context, and participant traits—to explore which characteristics are associated with the size of HPOG’s short-term impact on participant outcomes. We examine the relationship between program characteristics and impacts on four key HPOG outcomes—educational progress, employment, employment in healthcare, and earnings.

Many stakeholders—including practitioners, policymakers, funders, and researchers—may be interested in which combination of program components, implementation strategies, participant characteristics, and local context make important contributions to a program’s impact on individuals. Insights from this kind of research can help inform future program design and implementation.

Primary Research Questions
1. What characteristics of HPOG programs associate with any impacts on four key outcomes?
2. How do program characteristics associate with impacts for various outcomes?

The purpose of this paper is to identify which HPOG program characteristics associate with short-term impact magnitude. It aims to contribute to a better understanding of the HPOG Program by assessing how variation in program impacts associates with variation in various program characteristics, including design, implementation, and context.

From a methodological perspective, this work also provides a model for how to analyze cross-site data from multi-site experiments. It is often not practical, feasible, or even desirable to randomize all the program characteristics that one might want to learn about. Controlling for as many characteristics as possible enables researchers to isolate at least some individual program characteristics that might enhance or suppress overall impact. These observations can prove important for program practice and also generate causal hypotheses that can be tested in future evaluations.