CONCEPTUAL FRAMEWORK

A program evaluation should always be based on a sound, logical conceptual framework. Clearly defining the conceptual framework is the first step in developing an evaluation plan.

Bennett's Hierarchy of Evidence

One framework historically used within Cooperative Extension is Bennett’s Hierarchy of Evidence. In brief, Bennett says "We expend resources to conduct activities intended to obtain participation among targeted audiences, participant’s reactions to program activities affect their learning –- knowledge, opinions, skills, and aspirations. Through learning, people take action which helps achieve impact –- social, economic, and environmental changes."

In order for activities to improve social, economic, civic, or environmental condition(s), individuals must adopt behaviors that improve these conditions. Therefore, activities must be planned and must focus on the specific behavior(s) that is necessary to achieve the targeted social, economic, civic, and environmental condition(s).

To do this, resources that support the implementation of planned activities must be secured. Secondly, participant reactions to educational programs affect the extent of their participation in the planned activities. If participants have positive reactions, it is believed that they are more likely to acquire the knowledge, attitudes, skills, and aspirations (KASAs) that are precursors to adopting the targeted behaviors. It is believed that as participants acquire new KASAs, they are more likely to adopt targeted behaviors. As participants adopt behaviors, they have the potential to change the social, economic, civic, and environmental conditions that were identified as the reason to conduct the program. Changing social, economic, civic, and/or environmental conditions should always be the goal of an educational program.

It is important to note that Bennett’s hierarchy oversimplifies the reality of educational programs. For example, participant reactions may occur prior to and during activities. Also, practices might be adopted before attitude or knowledge change occurs. However, the simplification of this hierarchy is necessary to give one an understanding of how to focus a program evaluation.

The seven levels of Bennett’s hierarchy of evidence.

1. Resources –- staff and volunteer time; salaries; resources used; equipment, travel

2. Activities –- events; educational methods used; subject matter taught; media work; promotional activities.

3. Participation –- number of people reached; characteristics/diversity of people; frequency and intensity of contact/participation.

4. Reactions –- degree of interest; feelings toward the program; positive or negative interest in topics addressed; rating of educator; rating of program activities.

5. Learning –- knowledge (awareness, understanding); opinions (outlooks, perspective, viewpoints); skills (verbal or physical abilities;); aspirations (ambitions, hopes).

6. Actions –- behaviors, such as decisions taken, recommendations adopted, practices implemented, actions taken, technologies used, policies enacted.

7. Impact –- social, economic, civic, and environmental conditions intended as end results, impacts or benefits of programs.

Evidence of program outcomes becomes stronger as one goes up the hierarchy. Unfortunately, the difficulty and cost of getting quality data increases as ones moves up the hierarchy. This is often the reason that educators tend to rely on assessing Resources, Activities, Participation, and Reactions to demonstrate program value. To be able to present more defensible results about the value of one's program, an educator must focus their program evaluations on Learning and Actions. It is very difficult for educators to assess impact and so they should not be expected to assess their programs at this level. Impact assessment should be conducted by individuals who have received formal training in evaluation.

Logic Model

The logic model is another model widely used within the Cooperative Extension Service. It is an expansion of Bennett’s Hierarchy of Evidence. The logic model displays the sequence of actions that describe what the program is and what it will do –- how inputs result in outputs that lead to outcomes. The five core components of the logic model are:

1. Inputs –- staff and volunteer time, salaries, resources used, equipment, travel

2. Outputs -- activities, services, events and products that reach people who participate or who are targeted

3. Outcomes -- results or changes for individuals, groups, communities, organizations, communities, or systems

4. Assumptions -- the beliefs one has about the program, the people involved, and the context and the way one thinks the program will work

5. External Factors -- external factors that interact with and influence the program action

A detailed description of the logic model is available on the University of Wisconsin Extension website at: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

In the logic model, outcomes are at three levels –- short-term, medium-term, and long-term. Short-term outcomes focus on learning, such as changes in awareness, knowledge, attitudes, skills, aspirations, opinions, and motivation. Medium-term outcomes focus on action and so behaviors of program participants are assessed. Long-term outcomes assess social, economic, civic, and/or environmental conditions.

Short-term and medium-term outcomes can be easily measured by educators and so provide the foundation of county-based program evaluation. Evaluating long-term outcomes (social, economic, civic, and/or environmental conditions) is much more difficult because it requires a higher level of expertise on the part of the educator and requires more resources. Therefore, it is best that assessment at this level be led by University Faculty rather than by local educators.

Test Your Knowledge

1. What are the seven levels of Bennett's Hiearchy of Evidence?

2. What level of Bennett's Hierarchy of Evidence provides the best evidence of program value?

3. What are the five components of the logic model?

4. What are the three levels of outcomes that are a part of the logic model?

5. What do long-term outcomes typically assess?

ANSWER KEY