Summative evaluation is not only about measuring impact. Evaluation conducted at the conclusion of a programme could focus on a range of different interests such as cost effectiveness, appropriate educational materials, and efficient implementation strategies.
In general, however, people want to know what difference a programme has made. What impact has there been on the institution, the learner, the target audience, and the programme beneficiary. Even cost-benefit analysis presupposes some understanding of the impact that the programme has had and the weighting of that against the cost.
But measuring impact of a social intervention is not easy:
- The targets move.
- Indicators are hard to determine.
- Controlling the inputs and processes is difficult.
- It is possible for the programme to be interrupted.
A fairground metaphor is useful. In a test of strength (and as a means of parting patrons from their hard-earned money in the hope of impressing friends), fairgrounds often have a "ring the bell and win" game. It consists of a target to be hit, a hammer, and a long column with a bell at the top that rings if the impact of the hammer on the target is hard enough.
In this exercize, there are a range of indicators (normally using words that start off derogatory of the person swinging the hammer, encourage the person to try again, and then ring the bell for a prize); there is a clear target; there is a very simple input (person plus hammer); a range of potential processes (how does the person swing the hammer); and an impact that can be measured (assuming that the whole game is not rigged).
Would that social processes, of which education is one, were so simple.
The Moving Target
In the first place, discussions of impact make some presumptions about the target. It is assumed that the target is known, that the programme is directed at that particular target, and that the target is subject to no other significant alternative programmes or learning processes.
But, people are constantly moving and changing as a result of a range of different social activities and opportunities. It is unlikely, in educational programmes such as voter and civic education, that everything will be known about the public for whom the programme has been prepared. And, because programmes are conducted over time and space, it is likely that they will not only be received by a particular specified target group but by others, and the target group may also shift in size or scope as more becomes known during the implementation of the programme.
Despite these difficulties, the more that is known about the learner constituency, including the development of a programme baseline (see Baseline Studies), the more the evaluation will take into account all of the programmes to which it was exposed. If the evaluation is open to change, the estimation of the programme’s impact will be more reliable.
The Indicators of Impact
In the fairground, ringing the bell is an indicator of success and at the same time the objective of the exercize. There is a very close correlation between programme objectives (described by some as results statements) and programme indicators. If a programme is to be evaluated summatively, then a set of indicators should be established at the same time as the objectives are framed.
The questions that planners will ask are:
- How can we know that our objective has been achieved?
- How will we measure the achievement of the objective?
There may be a range of indicators for each objective, or alternatively one central indicator that validates the whole set of objectives. Planners look for this silver bullet, but it is a chimera.
Spoiled and Invalidated Ballots and Voter Turnout as Indicators
Voter educators have been tempted to use the indicator of spoiled and invalidated ballots as this silver bullet. If the objective of a voter education programme is to ensure that voters are able to vote, then it appears to make sense that a low percentage of spoiled and invalidated ballots means that the educational programme was successful.
But the percentage of spoiled and invalidated ballots can be affected by many other things:
- There may be a very good ballot design.
- Voters may be allowed assistance at the point of voting.
- The definition of a valid ballot may be very broad and vote counters may be generous in determining acceptable votes.
- Ballot box stuffing or other forms of cheating can result in prepared ballots entering the box rather than those of the uneducated or ill-prepared voter.
Or it may be that the voter education programme has had an objective of motivating people to vote. An obvious indicator would be the percentage turnout.
As with the spoiled ballot, there could be a range of alternative explanations:
- The sun could be shining.
- The political contestants could have succeeded in developing a high interest in the issues or outcome.
- The social significance of the election might be considerable.
- Material incentives or threats of retaliation may have driven up turn-out.
- Turn-out statistics may have been manipuated by election officials.
Evaluators should interrogate indicators carefully to ensure that they do indeed have significance only for the measurement of the impact of the programme. Or they may have to, at substantial cost and time, investigate whether these alternative explanations have validity and by a route of exclusion, attempt to show the impact of the programme itself.
Indicators that focus only on the personal do not come to terms with the overall impact of the programme on the society. Measuring the impact of the programme on the larger society requires assumptions about cause and effect that can hardly be described let alone measured.
The Inputs and Processes
Establishing indicators is difficult. Controlling the inputs and processes that might lead to an impact is even more tricky. Some of these can be invisible, or unintended. The plan may specify certain inputs and describe certain ways in which the programme will be implemented (the process). But successful completion may come about because of some unanticipated and unreported variations.
Evaluators who are involved with programmes from the beginning or who use methods that encourage participant observation and data collecting may notice these. An astute investigator may notice anomalies that can be explained only by looking for the unrecorded, and may then be able to find it.
But the bigger the programme, the more difficult it is to keep track of all the inputs and processes and, through evaluation, ascribe impact to one or the other, or ascribe lack of impact to one or the other, except in gross or obvious terms.
A particular set of training events can be very successful in turning out effective trainers:
- This may be a result of the training programme developed and used.
- It may equally hinge on the personality and skills of the trainer.
- It may be that participants were all effective trainers in their own right.
- Participants may have been recruited from an organization that gave them continuing support and ongoing assessment of progress.
Possible Interruptions
In the "ring the bell" game, the bell may not ring not because of the impact of the hammer but because of the game owner's hidden anti-success contraption.
There are many possible reasons why a programme may not achieve its targets even though it does everything according to the book. It is not always possible to understand these reasons or identify them fully.
It is not the fault of the voter education programme if an election is stolen. Citizen apathy, despite an extended civic education programme, may be caused by a political system which disempowers rather than through a poorly developed or implemented programme.
In Summary
Establishing the impact of a programme requires basic knowledge about the target audience, carefully considered and humbly stated indicators, the ability to make visible and reflect on alternative reasons for the apparent success or otherwise of the programme, and extensive knowledge not only about the plan but also about the actual implementation process.
And all of these are difficult to achieve even in a small and limited programme intervention. In a large national programme, it therefore becomes extremely difficult to measure impact with any degree of certainty.
Evaluators should, therefore, consider summative evaluations with some caution and develop a range of ways of communicating the value of a programme rather than making glib statements about the extent to which it has changed the world.
