Home --> Health Equity Lens --> Data, Research and Evaluation for Health Equity --> Evaluation for Health Equity
|
Evaluation for Health Equity
|
|
IN THIS SECTION
- The Utility of Evaluation In Health Equity - Evaluating Complex Community Initiatives - Evaluation Challenges - Strategies to Overcome Evaluation Challenges |
Evaluation essentially involves a value judgment about how well something worked and whether it should be continued. In the interim, evaluation is critical for informing the ongoing work of any initiative and helps to ensure that stakeholders are focused on activities thought to be the most effective. Effective program evaluation is a systematic method of improving and accounting for public health actions (CDC, 1999).
Evaluation is one of the Ten Essential Public Health Services outlined by the CDC and a framework for evaluating public health efforts, developed by public health leaders at the CDC, is widely used within the field. An illustration of the key elements of the framework is presented in Figure 1. Click here to read more on the evaluation framework. |
The evaluation of health equity initiatives is particularly important, because it can reveal the effects of initiatives on different groups, ensure that stakeholders do not lose sight of the intention to reduce gaps in health status, and ensure equity in the distribution of resources needed for optimal health. The importance of understanding what works, for whom, under what conditions, and whether health inequities have decreased, increased, or remained the same requires a deliberate focus on equity in evaluation efforts (CDC, 2013).
Recommended Resource
In their Practitioner Guide for Advancing Health Equity, the CDC offers a number of questions for practitioners in government and community-based organizations to reflect upon when working to incorporate health equity into evaluation efforts (CDC, 2013. p. 33)
Practitioner Guide for Advancing Health Equity - Evaluation Efforts
Where are we now?
- How are we currently assessing the effect(s) of our efforts to address health equity?
- Do we have the expertise to develop, implement, and assess an equity-oriented evaluation plan?
- What process can we establish to routinely engage community stakeholders, including those experiencing health inequities, in all aspects of our evaluation efforts?
- What are our current health equity strategies, activities and goals?
- How can our logic model be modified to reflect our health equity activities and goals?
- How can we reframe or create new evaluation questions to better understand our effect on health inequities?
- What are the key variables we should use to track the influence of our efforts on populations experiencing health inequities?
- How can our sampling plan be designed or modified to answer our health equity-oriented evaluation question(s)?
- What processes do we have in place to determine when culturally appropriate tools or methodologies are needed?
- If modifications are needed, how can we ensure our evaluation tools meet the needs of populations experiencing health inequities (e.g., language and literacy needs)?
- Are the data we are collecting reflective of the real experience of the populations experiencing inequities? Are other approaches needed?
- Does our performance monitoring system allow us to track and identify needs that may arise when implementing efforts in underserved communities?
- How can we structure our evaluation processes to understand the long-term effects of our efforts on health inequities?
- Does our analysis plan allow us to answer the following: What worked? For whom? Under what conditions? Is there any differential impact? Have inequities decreased, increased, or remained the same?
- If not, how can we modify the analysis plan to answer these questions?
- Does our outcome evaluation allow us to determine differential effects across population groups?
- Does our process evaluation allow us to understand the key factors that influenced the outcomes of our efforts in underserved communities?
- What actions do we need to take to improve or enhance our evaluation plan to understand our effects on health equity (e.g., have inequities decreased, increased, or remained the same)?
- How and where do we typically disseminate our evaluation findings?
- What commitment can we develop to ensure we share findings, even if negative?
- How can we ensure we share our findings in plain and clear language that can be understood by stakeholders, partners, and community members?
- How can our findings be used to support more action in communities of greatest need?
- How can we revise the ways in which we share lessons learned to help others concerned with addressing health inequities?
- What can we do differently to improve or enhance our ability to conduct health equity-oriented evaluations?
- What is our plan of action to implement improvements in our evaluation efforts?
The Utility of Evaluation in Health Equity
Evaluation is a critical component to achieving health equity and should be as multi-faceted, responsive, and flexible as the initiatives themselves (Preskill, Parkhurst, & Splansky-Juster, 2014). Evaluation efforts are most useful when considered at the outset of the initiative. In fact, thinking about evaluation during the planning phase of a health equity initiative can contribute to the likelihood of its success. It encourages participants to think in depth about the goals of the initiative and what success looks like in the short-term as well as the long-term. This requires participants to clearly articulate their theory of change, or the rationale for their approach, and helps them communicate this effectively with other stakeholders. Similarly, incorporating evaluation at the outset of a program provides an opportunity to identify intermediate measures of progress towards the ultimate goal of achieving health equity, which is particularly important for place-based initiatives or community-based efforts that target upstream SDOH. In such cases, stakeholders must consider the relation between the targets of the intervention (e.g. affordable, quality housing) and longer-term outcomes related to health status and health equity (lower rates of asthma among low income children). Finally, incorporating evaluation into early planning phases encourages stakeholders to consider important questions related to needed resources and the capacity available for evaluation.
Sample Strategy 1
A good example of using evaluation during the planning stages of an intervention is provided by an organization called the Children and Families Commission of Orange County (CFCOC). CFCOC was created as a result of Proposition 10 in California, where funding from a tax on tobacco products is used to support early childhood development for children ages 0-5. CFCOC’s vision is that all children (in the county) are healthy and ready to learn. Through an extensive planning process, CFCOC identified the following goals to reach their vision:
|
Sample Strategy 2
In Delaware, the Nurse-Family Partnership (NFP) provides a similar example of a coordinated strategy that incorporates evaluation from the outset for meeting multiple objectives. Implemented by Children and Families First, the NFP is an evidence-based community health program that links newly pregnant, first-time mothers with a registered nurse. The nurse provides home visits throughout the woman’s pregnancy and during the first two years of the baby’s life. NFP has three distinct, but complementary, goals:
The program was launched in Delaware in 2010. Evaluation data revealed early success in terms of positive health outcomes for babies. As of 2014, 90 percent of babies served by the program were born full-term; and 88 percent were born at a healthy weight. Furthermore, 82 percent of mothers in the program initiated breastfeeding, and 21 percent were still breastfeeding at their six month follow-up (Wallace, 2014). These evaluation data contributed to a recent increase of $1.3 million in state appropriations, which doubles the number of women and babies served. |
Evaluating Complex Community Initiatives
The evaluation of comprehensive place-based initiatives, is particularly challenging due to the complexity of this type of initiative as well as the uniqueness of communities.
“Most comprehensive place-based initiatives consist of multiple interventions over a number of years at individual, group, institutional, social and political levels. Any one of these interventions could be an evaluation in and of itself, but with [this approach] you want to capture what matters” (Kelly, 2010, p. 19).
Based on his experience working with the evaluation of the Casey Foundation’s Making Connections initiative, Kelly developed Five Simple Rules for Evaluating Complex Community Initiatives. These include the following:
- Evaluations of complex, major initiatives are not experiments but part of the community change process.
- Evaluations need a strong focus on the processes of community change.
- Evaluations need to measure ongoing progress toward achieving outcomes and results to help a community guide its change process and hold itself accountable.
- Evaluations need to understand, document, and explain the multiple theories of change at work over time.
- Evaluations need to prioritize real-time learning and the community’s capacity to understand and use data from evaluations. (Kelly, 2010)
Additional Resources
- Resources to support evaluation of community-based health equity initiatives can be found in the Community Toolbox referenced in previous sections of this guide.
- The CDC also provides links to a number of valuable resources for evaluation, accessible here.
Evaluation Challenges
The process of evaluation can be tedious and confusing and is not free of barriers. Because a successful evaluation relies on the use of data collection and analysis, the barriers inherent in those processes are also applicable to evaluation (see Limitations of Data Collection and Analysis for Health Equity on page 149 of this guide). In addition, the evaluation process is subject to other barriers, which are more likely due to the substantial partnering that is necessary during the evaluation process. Regarding health equity efforts, this process is also subject to barriers that arise due to the complexities involved in assessing long-term outcomes related to SDOH.
|
“We must learn not only whether an intervention can work, but how, why and for whom, and how we can do better.” |
Ideally, the evaluation process begins during the planning phase of developing an intervention. Because this phase is often highly collaborative and involves input from numerous stakeholders and community members, the evaluation process is often subject to much debate. Therefore, public health practitioners and partners will benefit from clearly defining their desired outcomes and deciding on the appropriate measures for assessing these outcomes early in the planning phase. Similarly, by garnering buy-in for the evaluation process early on, stakeholders will be more likely understand the importance of the evaluation, and then later make changes (such as resource allocation and alterations in program design) when the evaluation results indicate room for improvement.
Specifically in regard to the evaluation of health equity-focused interventions, it is important to understand the limitations of assessing interventions that target SDOH. Typically, grants are awarded for short-term interventions (i.e. three to five years), which stunts the ability to assess impacts on the SDOH, such as income level and educational attainment. Therefore, the evaluation process for such interventions should include measures that can be used to indirectly assess the likely impact on SDOH.
Specifically in regard to the evaluation of health equity-focused interventions, it is important to understand the limitations of assessing interventions that target SDOH. Typically, grants are awarded for short-term interventions (i.e. three to five years), which stunts the ability to assess impacts on the SDOH, such as income level and educational attainment. Therefore, the evaluation process for such interventions should include measures that can be used to indirectly assess the likely impact on SDOH.
Strategies to Overcome Evaluation Challenges
To effectively address the evaluation challenges listed here, practitioners may need to build upon traditional evaluation methods and consider alternate approaches. For instance, when logic models are used to guide program implementation and evaluation, they must incorporate equity-related activities and outcomes. Because changes in living conditions (such as an increase in the number of affordable housing units or an increase in average wages) may be the target of the intervention, they should also be the focus of the evaluation. Evaluators must recognize that changes in health outcomes related to changes in the physical and social environment may take several years, if not generations, to manifest. Although tools like logic models can be useful in articulating the expected long-term changes, evaluators may need to consider intermediate outcomes and unique measures as indicators of impact. Case studies and other qualitative evaluation methods, for instance, can be used to help demonstrate impact. Finally, since health equity-focused interventions typically target culturally diverse groups, culturally appropriate tools and methodologies are essential to effective evaluation of health equity interventions.
Additional Resources
Information regarding culturally appropriate measures that can be incorporated into evaluation can be found at the San Diego Prevention Research Center’s website.
Because of the growing attention to health equity nationally, and the limited availability of evidence-based strategies for achieving health equity, federal agencies and national funders may be a resource for financial support and/or technical assistance.
Readers are encouraged to subscribe to electronic mailing lists that provide this information and conduct regular scans.
Because of the growing attention to health equity nationally, and the limited availability of evidence-based strategies for achieving health equity, federal agencies and national funders may be a resource for financial support and/or technical assistance.
- The U.S. Department of Health and Human Services’ Office of Minority Health offers support for identifying funding sources in response to specific organizational needs (Click here).
- Other organizations maintain lists of available funding opportunities that are regularly updated. For instance, the Association of American Medical Colleges posts weekly updates of funding opportunities to support research on health disparities and health equity (Click here).
Readers are encouraged to subscribe to electronic mailing lists that provide this information and conduct regular scans.