Evaluation Road Map
Knowing your Linkages program is working requires a clear definition of success, a way to measure that success and a feedback loop to improve Linkages as a result of what the evaluation shows. It starts with agreement on the client-centered results Linkages will help families achieve including a theory of change for getting from point A to point B. To evaluate your Linkages program follow the steps below using the planning tips and links to resources to help guide you along the way.
Select Outcomes
Customize Your Linkages Logic Model
Identify the Measures For Your Outcomes
Collect Data
1Select Outcomes
Before evaluation begins, follow the Program Design Road Map to create a clear definition of your Linkages program, who it will serve and what interventions will be used to meet program goals. Evaluation of your Linkages program starts with knowing what success looks like for your initiative. This means determining which results for families (outcomes) you want Linkages to achieve.
What changes in the lives of Linkages families would you want to see to help you know whether the program has been successful? Refer to How To Select Linkages Outcomes button to guide the selection of outcomes that best demonstrate the positive impact you expect your Linkages program to accomplish.
Experience Tells Us…
- Select a small number of outcomes that are critical to the type of Linkages program you are implementing.
- Be sure to select outcomes that are beneficial for both CalWORKs and Child Welfare.
2Customize Your Linkages Logic Model
The next step in evaluation is to refine your hypothesis for why you believe Linkages will work. To do this, review the Linkages Logic Model and customize it to match your specific Linkages initiative. Use your customized logic model to convey a concise summary of your Linkages program to various audiences. It will not only guide evaluation, but can serve as a training and communications tool as well.
Experience Tells Us…
- Make sure your Logic Model is succinct–with enough detail to be useful, but not overly complex.
- Refer to your Logic Model during implementation to monitor that your Linkages program is on course.
- The Logic Model can be a living document which can be adjusted as your program evolves.
Make your own logic model diagram using the systems perspective version as a template. Review and adjust the Inputs/Resources and Outputs/Activities which will lead to the Outcomes selected as relevant to your Linkages initiative. Also, refine the External Factors that may affect the success of Linkages in your location. Next, work with your planning team to ensure a logical progression from each column of the diagram to the next. Specifically, consider these questions:
- RESOURCES/INPUTS: What are the key ingredients necessary for implementing Linkages? Consider which leadership and staff need to be involved, what training and new procedures are needed, and how large your client base needs to be to make implementation worthwhile. Also consider the data collection and evaluation procedures you need to have in place to evaluate your initiative.
- OUTPUTS: Describe the work that is accomplished through the Linkages program including what gets done and who is reached. What are the activities and processes that define your Linkages initiative? These are the set of actions, interventions, or services that are unique to Linkages. Who is served by Linkages? These are the clients or customers who benefit from Linkages.
- OUTCOMES: Represent the difference Linkages will make. How will your Linkages initiative benefit families? For most programs, the outcomes answer the question, “Why are we implementing Linkages?” Outcomes are often longer-term results than Outputs and reflect the goals of the initiative.
- EXTERNAL FACTORS: These are contextual factors that can affect the success of Linkages but are outside the boundaries of the Linkages initiative. Consider how these factors can influence some of the initiative’s inputs or activities. If an external factor is likely to pose a barrier to your program, you can start early on planning around it to the extent possible.
3Identify the Measures For Your Outcomes
With your customized logic model in hand, the next step is to determine how to measure the outcomes you want to evaluate. This involves knowing who your Linkages families are, translating outcomes into specific data elements and deciding when measurement will occur. Use the Worksheet Resource to engage your planning team and county data analysts to answer the following questions: • What time period do you want to establish from which to draw the data?
- What time period do you want to establish from which to draw the data?
- What identifying information is shared commonly between both data systems?
- Which clients are your Linkages clients?
- What client information do you need to measure each outcome?
Experience Tells Us…
- Make sure the data you’ve identified to calculate your outcomes is consistently captured and accessible.
- Train staff on how to enter the critical data that will help track the success of Linkages.
- Encourage supervisors to reinforce the value of evaluation with their staff.
4Collect Data
Data collection involves gathering both quantitative data about your Linkages clients and collecting qualitative information directly from staff and clients. Together, this tells you whether Linkages service coordination is effective in changing families’ lives for the better and how staff and clients are experiencing the program.
Quantitative data comes primarily from the Child Welfare (CWS/CMS) data system and the CalWORKs/TANF data system. To identify Linkages clients and link each client’s information from both of these systems requires, be sure to label or flag your Linkages clients with a special label, code or common identifier in your database(s).
Qualitative data can be drawn from surveys, interviews or focus groups to reveal the level of satisfaction staff and clients have about Linkages. See the resource links below for sample satisfaction surveys that can be adapted for use in your location. The quality of service coordination occurring on Linkages cases can be gleened from case file reviews through analysis of case plans, contact notes or other narrative case documentation.
Experience Tells Us…
- When an outside evaluator is involved in data collection, protecting client identities is essential
- Create a unique ID number for each of your Linkages clients. This provide the evaluator with the information they need on each client without providing any personally identifying information like names or social security numbers.
- A unique ID number should not contain the client’s initials, date of birth, or social security number.
- Collect and enter data in a consistent way – a process called data validation. For example, if you have a database field called “Linkages start date,” make sure that means the same thing to all of the people who collect and enter the information.
5Analyze Your Results
This is where you make meaning out of your evaluation data. Invite your Implementation Team, including line staff and supervisors, to help interpret what the results show. As a point of comparison, use outcomes data to examine changes over time. To do this, measure your outcomes for (at least) two different time periods. Compare each outcome to see if there was a change from the earlier to the later time period, and whether that change was positive or negative.
As your Linkages program continues, you can look at outcomes data from one year to the next to see whether the numbers are improving, getting worse, or staying the same. You may also be able to compare a group of Linkages clients with a group of similar clients that are not in your Linkages initiative to see if Linkages makes a difference.
Experience Tells Us…
- Changes in the evaluation findings for better or worse may not always be due to Linkages–other factors like economic downturns can affect outcomes too.
- Engage those closest to Linkages day-to-day practice to help interpret the evaluation findings.
- Communicate evaluation results to staff, leadership, stakeholders and other interested parties.
6Adjust Linkages Based on Results
Conclusions drawn from evaluation can suggest changes and refinements necessary to improve or expand your Linkages program. This may include revisions to policies, procedures, staff development or other aspects of Linkages. Gather your implementation team to interpret the findings and identify program improvements; include both line staff and managers in the dialogue.
Once the implications of your evaluation results are known, revisit the Program Design Road Map and plan for the various components of Linkages that may need to be adjusted. Be sure to clearly communicate the shifts in your Linkages program to all involved.
As your Linkages program continues, you can look at outcomes data from one year to the next to see how its faring. You may also be able to compare a group of Linkages clients with a group of similar clients that are not in your Linkages initiative to see if Linkages makes a difference. This information is useful in helping you understand areas of success and areas of challenge for your initiative.
Experience Tells Us…
- Sustained leadership is necessary to ensure consistent evaluation of Linkages programming.
- Keep key champions informed about program improvements based on evaluation findings. They can help explain to staff why Linkages processes & procedures are being modified.