Monitoring and Evaluation

Educational programmes have intended outcomes. They have plans that are being followed in order to achieve these outcomes. These plans consist of a range of components working together to ensure their successful implementation.

This section deals with ways in which monitoring and evaluation take place and some of the dilemmas faced by those who conduct evaluations. It looks at methodological and organizational questions, and at the key question faced by all evaluators: the extent to which programme implementers take advantage of the evaluation recommendations. The section following offers a general overview of evaluation and its relationship to educational testing.

By implementing monitoring plans and evaluating their outcome once completed, educators seek to ensure that they are being accountable to their stakeholders, true to their intentions, and that they themselves will learn from disciplined reflection on the experience of the programme for further work that they might do.

Terminology and Meaning

While it is common to use the twin terminology of monitoring and evaluation, this section merges these terms into the single generic term evaluation, and talks about formative evaluation (that conducted with a view to changing a programme while it is in progress) and summative evaluation (that conducted to sum up the programme). But neither is entirely independent of the other.

The meaning of the word evaluation is a slippery one. Those requesting evaluations may be trying to determine the value of a programme by measuring its impact. Evaluators may already have a view of the programme and may be merely seeking external verification of that view. They may be trying to build support for a particular course of action either to defray responsibility or, more generously, to obtain a second opinion before acting.

What Can be Evaluated

Everything about a programme is worth evaluating, and there is no restriction on the locus of an evaluation but the feasibility of doing it. There may be an assessment of the contextual analysis that is done, the relevance of purpose, aims and objectives, the cost effectiveness of the programme, the extent to which it met its objectives, the process by which it did this, the competence of staff, the relationship between materials and objectives, and so on. At its heart, people expect evaluation to tell them something about the difference the programme made to the participants or intended beneficiaries.

Such evaluation inevitably takes place, whether planned or unplanned. People develop opinions about aspects of the programme or its success. These opinions may be well-informed, or they may be informed by prejudice, a particular limited experience, or a restricted perspective on the programme. Good evaluations are designed to ensure that the most reliable picture is obtained of a programme. Educators should ensure a professional rather than merely intuitive evaluation for all programmes that are complex in their number of components, cost, length, or long-term implications.

Educational and Programme Evaluation

Educational evaluation has often, therefore, consisted of evaluating the learners and their achievements or progress over the period of the educational programme. Depending upon the type of intervention, this can be done in a number of ways. First, this can be done by testing competence or knowledge through written and oral presentations or examinations. Second, information might be achieved by collecting baseline data and conducting of tracking polls throughout the course of the intervention to identify changes in awareness levels, attitudes, and even behaviours. This qualitative data might be further supplemented by qualitative feedback collected through focus groups. Some election statistics might also provide useful information. With the case of some polling data and election statistics, however, educators need to keep in mind that it may not be possible to isolate their intervention as the sole reason for a particular result.

Programme evaluation goes beyond the assessment of individual learning. But educational programmes should not escape the need to ensure that this is assessed. In non-formal educational programmes this may not be best done by examinations, although tests of competency are essential for training of educators.

Indicators

Evaluators establish a range of indicators by which they will be able to judge whether or not the necessary learning has taken place.

Similar indicators are established for other components of the programme. In each case, the achievement of these is deemed (unless there is evidence to the contrary) an indication that the programme has been successful. Sample indicators might include: A significant increase in the number of registered voters, in the number of first time voters, or of voters using a special voting service might all be indicators that an education programme has affected awareness levels and behavior. Similarly, decreases in the number of spoiled or invalidated ballots might also be used as indicators of voters' better understanding of the voting process.

Establishing these indicators is a difficult task and has to be done at the same time as the setting of objectives. Failure to do this leads to the development of programmes whose objectives are not able to be evaluated, or the development of indicators that are alternatively insignificant or established after the fact.

Evaluation Controversy

Because evaluation is seen as the important component in validating a programme to outsiders, and because it contains these technically difficult activities (see Measuring Impact and Importance Of Prior Planning ) and such a range of stakeholder expectations and fears, it can become a highly charged and political activity, especially if the programme has a high stakeholder investment. This could be the case if it is especially expensive, or if the programme has been endowed with societal significance (for example a founding election or the tryout of new technology).

Educators should retain control over the evaluative process and its planning, implementation and costs. The best way to do so is to ensure that evaluation is built into the programme from the beginning and is not an activity imposed on the programme at a later date. When it forms part of the standard operating routine, evaluation becomes a valuable tool. On the other hand, when it is an afterthought or used for crisis management, evaluation becomes a burden and a controversial and difficult activity.

Who Should Evaluate

Educational programmes will be evaluated, whether professionally or intuitively. The question is, who will be responsible for planning and conducting the evaluation.

Monitoring is a responsibility of those who manage the programme. They may use a range of tools to ensure that it is done accurately and adequately. But evaluation implies a distance from the programme that, it is assumed, cannot be achieved by the staff themselves.

Nevertheless, there are aspects of an evaluation that can appropriately be done by staff, and there are evaluation designs that benefit from staff participation.

Independent Evaluators

External evaluators provide independence and disinterest in the evaluation outcome necessary to ensure the study produces reliable results. Such results are unencumbered with the bias, shortsightedness or defensiveness that comes from those who are too close to the programme.

Outsiders can find it easier to obtain objective comment from participants and access to those who have been excluded from the programme. They also can provide the additional prestige that an evaluation requires, to be taken seriously and in order to ensure that the recommendations of the study are implemented.

They will have the time to conduct the evaluation that is unlikely to be available to staff who are involved in ongoing programmes.

And finally, commissioning external evaluators may be the only way of obtaining the necessary expertize for the evaluation of a large and complex programme.

Self-Evaluation

Those who are close to the programme bring their own intimate knowledge of the programme and of the intended and actual outcomes to an evaluation. They have an interest in the outcome and,in most cases, also a professional commitment to improvement of their work, so that implementation of evaluation findings is more likely.

Internal evaluations increase the awareness of staff of the programme issues which have to be considered, such as the cost-effective use of resources, the need for clear objectives, the importance of collecting appropriate information throughout the programme.

Weaknesses of Both Approaches

• External Evaluators. External evaluators can take a considerable amount of time to understand the context within which the programme is running, to be able to enter the world of the programme stakeholders in order to interpret their responses to questions and their reaction to evaluator reports, and to follow the programme plan and its implementation.

They are costly, although the cost of employing outsiders can be offset by their ability to conduct the evaluation in a shorter period of time, assuming their familiarity with the context and the concepts.

They do not have to live with their recommendations, and they do not have to live with the consequences of their reports. At the worst, this can result in unrealistic recommendations which do not adequately reflect an understanding of the environment within which a programme has to be delivered.

• Insiders. Insiders, on the other hand, can get too close to the programme. They have relationships that have to be protected and personal careers that can be jeopardised. They may have a vested interest in a particular outcome or recommendation.

As mentioned, they often do not have sufficient time, although it is possible to give the task to a specific team within the organization. It may be difficult for them to get reliable and objective information, either from their colleagues or from programme participants.

Combining Insiders and Outsiders

Effective evaluation requires a combination of insiders and outsiders. It may be that the final report is prepared by the outsiders, while the insiders act as liaison and facilitation staff. Alternatively, both insiders and outsiders can declare their interests in the report.

An effective evaluation design encourages stakeholder participation, concentrates on skill development, and establishes the usefulness of the report and its recommendations. It uses a variety of data collection and analysis tools that encourage participation and ownership.

In such an evaluation, the roles of evaluators and of all programme stakeholders is carefully described, and the determination how to make use of staff insiders or independent outsiders is based on what is most effective for the particular evaluation study.

Monitoring Or Formative Evaluation

Evaluators refer to formative evaluation, which is similar to monitoring. It is activity that runs parallel to programme implementation and which seeks to refine and improveme the programme during the implementation phase.

Advertizing agencies refer to tracking research: keeping the programme on track, or keeping track of the programme.

The tools used for such evaluation can be the same as those used in more traditional end of project or summative evaluation (see Measuring Impact). They also include more traditional managerial tools and those developed for dealing with individual educational events.

The difficulties faced by formative evaluation are not those of data collection. They are primarily those of establishing mechanisms for turning evaluation insights into programmatic innovation and amendment. This section deals with conflicts that might emerge, with the importance of the programme plan (see also Obtaining and Maintaining Commitment To the Plan) and with some monitoring tools.

Monitoring Must Keep Pace

Education programmes run on a tight schedule. Events and activities are planned in advance and decisions are made about production of materials, preparation of educators, and broadcast of information that are not easily altered. Face-to-face activities, in particular, require substantial lead times.

Evaluators must be closely linked to those managing the programme and must be able to analyse data received very rapidly if it is to make a difference. In addition, they need to have good knowledge of the programme objectives and purpose so as not to suggest courses of action that take the programme away from these objectives. All suggested changes need to enhance achievement of objectives, with one exception.

Extended programmes can build in mid-term assessments that are more thorough and can include an assessment of strategy and objectives. Any changes recommended as a result of feedback received have to be carefully considered, especially if there are contractual obligations based on the original programme objectives and outline. While it is relatively easy to make changes to how a programme is implemented, it is much more difficult to make changes to what is intended.

In some cases, this is necessary if the information being received suggests major problems with the programme design. Formative evaluation usually is concerned with refinement.

Conflict between Evaluators and Programme Managers

Even if there is a clear distinction between the evaluation and management teams, and even if the roles are carefully clarified, conflict can emerge between those who obtain and analyse evaluation data and those who are responsible for programme implementation.

Regular meetings, and a formal agreement about how to resolve such conflicts, are of assistance. Evaluators can be forced to note for the summative record recommendations that were not followed. At the same time, having had their recommendations followed can also cause conflict if the changes are not successful. Evaluators can suddenly find themselves evaluating programme innovations that they motivated, but which are not working.

The role of evaluators therefore becomes quite problematic unless they act as technical assistants providing unanalysed data directly to programme managers and assisting in the joint analysis of such data. This relationship is less open to conflict.

Working the Plan

The preliminary work done on the education plan is the most essential tool available for formative evaluation or programme monitoring. On a regular basis it is possible to check the extent to which the programme is being implemented and the extent to which changes in the context or amongst the target constituency require adaptations to the programme.

Having a plan documented also enables such changes to be recorded and noted for later general evaluation.

Tools for Monitoring

In addition to the general data collection tools discussed in this topic area, the client response or post-event reaction form is perhaps the most important.

Such response forms have been refined by commercial concerns and it is possible to use these as models of layout and brevity. When the audience is literate, it is possible to collect individual responses from events or particular services, process these, and use the results to fine-tune the programme. When the audience is illiterate, small group discussion and recorded feedback can serve a similar purpose.

Additional tools can include telephone complaints lines, peer assessment of educators, short surveys, and occasional gatherings of stake holders in focus-group-like discussions.

Role of Staff

However, the best source for information is a reflective and aware staff. Orientation to evaluation and monitoring concerns should be part of every training event, and programme meetings should include opportunities for staff to communicate what they have learned and what they have discovered from their interactions in the field.

The purpose of formative evaluation is not to be able to say: "I told you so," but to make sure that the programme is appropriate and effective. Everything should be geared toward this end and the educators should create a learning environment both internally and externally.

Measuring Impact

Summative evaluation is not only about measuring impact. Evaluation conducted at the conclusion of a programme could focus on a range of different interests such as cost effectiveness, appropriate educational materials, and efficient implementation strategies.

In general, however, people want to know what difference a programme has made. What impact has there been on the institution, the learner, the target audience, and the programme beneficiary. Even cost-benefit analysis presupposes some understanding of the impact that the programme has had and the weighting of that against the cost.

But measuring impact of a social intervention is not easy:

• The targets move. 
• Indicators are hard to determine. 
• Controlling the inputs and processes is difficult. 
• It is possible for the programme to be interrupted.

A fairground metaphor is useful. In a test of strength (and as a means of parting patrons from their hard-earned money in the hope of impressing friends), fairgrounds often have a "ring the bell and win" game. It consists of a target to be hit, a hammer, and a long column with a bell at the top that rings if the impact of the hammer on the target is hard enough.

In this exercize, there are a range of indicators (normally using words that start off derogatory of the person swinging the hammer, encourage the person to try again, and then ring the bell for a prize); there is a clear target; there is a very simple input (person plus hammer); a range of potential processes (how does the person swing the hammer); and an impact that can be measured (assuming that the whole game is not rigged).

Would that social processes, of which education is one, were so simple.

The Moving Target

In the first place, discussions of impact make some presumptions about the target. It is assumed that the target is known, that the programme is directed at that particular target, and that the target is subject to no other significant alternative programmes or learning processes.

But, people are constantly moving and changing as a result of a range of different social activities and opportunities. It is unlikely, in educational programmes such as voter and civic education, that everything will be known about the public for whom the programme has been prepared. And, because programmes are conducted over time and space, it is likely that they will not only be received by a particular specified target group but by others, and the target group may also shift in size or scope as more becomes known during the implementation of the programme.

Despite these difficulties, the more that is known about the learner constituency, including the development of a programme baseline (see Baseline Studies), the more the evaluation will take into account all of the programmes to which it was exposed. If the evaluation is open to change, the estimation of the programme’s impact will be more reliable.

The Indicators of Impact

In the fairground, ringing the bell is an indicator of success and at the same time the objective of the exercize. There is a very close correlation between programme objectives (described by some as results statements) and programme indicators. If a programme is to be evaluated summatively, then a set of indicators should be established at the same time as the objectives are framed.

The questions that planners will ask are:

• How can we know that our objective has been achieved? 
• How will we measure the achievement of the objective?

There may be a range of indicators for each objective, or alternatively one central indicator that validates the whole set of objectives. Planners look for this silver bullet, but it is a chimera.

Spoiled and Invalidated Ballots and Voter Turnout as Indicators
Voter educators have been tempted to use the indicator of spoiled and invalidated ballots as this silver bullet. If the objective of a voter education programme is to ensure that voters are able to vote, then it appears to make sense that a low percentage of spoiled and invalidated ballots means that the educational programme was successful.

But the percentage of spoiled and invalidated ballots can be affected by many other things:

• There may be a very good ballot design. 
• Voters may be allowed assistance at the point of voting. 
• The definition of a valid ballot may be very broad and vote counters may be generous in determining acceptable votes. 
• Ballot box stuffing or other forms of cheating can result in prepared ballots entering the box rather than those of the uneducated or ill-prepared voter.

Or it may be that the voter education programme has had an objective of motivating people to vote. An obvious indicator would be the percentage turnout.

As with the spoiled ballot, there could be a range of alternative explanations:

• The sun could be shining. 
• The political contestants could have succeeded in developing a high interest in the issues or outcome. 
• The social significance of the election might be considerable. 
• Material incentives or threats of retaliation may have driven up turn-out. 
• Turn-out statistics may have been manipuated by election officials.

Evaluators should interrogate indicators carefully to ensure that they do indeed have significance only for the measurement of the impact of the programme. Or they may have to, at substantial cost and time, investigate whether these alternative explanations have validity and by a route of exclusion, attempt to show the impact of the programme itself.

Indicators that focus only on the personal do not come to terms with the overall impact of the programme on the society. Measuring the impact of the programme on the larger society requires assumptions about cause and effect that can hardly be described let alone measured.

The Inputs and Processes

Establishing indicators is difficult. Controlling the inputs and processes that might lead to an impact is even more tricky. Some of these can be invisible, or unintended. The plan may specify certain inputs and describe certain ways in which the programme will be implemented (the process). But successful completion may come about because of some unanticipated and unreported variations.

Evaluators who are involved with programmes from the beginning or who use methods that encourage participant observation and data collecting may notice these. An astute investigator may notice anomalies that can be explained only by looking for the unrecorded, and may then be able to find it.

But the bigger the programme, the more difficult it is to keep track of all the inputs and processes and, through evaluation, ascribe impact to one or the other, or ascribe lack of impact to one or the other, except in gross or obvious terms.

A particular set of training events can be very successful in turning out effective trainers:

• This may be a result of the training programme developed and used. 
• It may equally hinge on the personality and skills of the trainer. 
• It may be that participants were all effective trainers in their own right. 
• Participants may have been recruited from an organization that gave them continuing support and ongoing assessment of progress.

 
Possible Interruptions

In the "ring the bell" game, the bell may not ring not because of the impact of the hammer but because of the game owner's hidden anti-success contraption.

There are many possible reasons why a programme may not achieve its targets even though it does everything according to the book. It is not always possible to understand these reasons or identify them fully.

It is not the fault of the voter education programme if an election is stolen. Citizen apathy, despite an extended civic education programme, may be caused by a political system which disempowers rather than through a poorly developed or implemented programme.

In Summary

Establishing the impact of a programme requires basic knowledge about the target audience, carefully considered and humbly stated indicators, the ability to make visible and reflect on alternative reasons for the apparent success or otherwise of the programme, and extensive knowledge not only about the plan but also about the actual implementation process.

And all of these are difficult to achieve even in a small and limited programme intervention. In a large national programme, it therefore becomes extremely difficult to measure impact with any degree of certainty.

Evaluators should, therefore, consider summative evaluations with some caution and develop a range of ways of communicating the value of a programme rather than making glib statements about the extent to which it has changed the world.

Importance Of Prior Planning

It is possible to evaluate an educational programme after it has been completed. But it is a much more difficult task. Because of this, evaluation should be built into the programme from the start.

It may otherwise prove impossible to do more than merely report on the completion of the programme in a manner that could as easily be done by a concluding report.

To properly evaluate a programme, there must be measurable objectives, meaningful indicators, concrete evaluation criteria, a commitment to document the programme, and preparation of the necessary information that an evaluation requires at the time of the events.

All these pre-suppose that the decision to evaluate is taken at the time the programme is planned, and that the evaluation design is built into that planning. Very few programme planning exercizes achieve this simultaneity.

The best that can be hoped for is that the commitment to evaluation is made at the inception, that money is set aside for this activity, and that dates are scheduled for designing the evaluation, bringing in any external evaluators and conducting the necessary evaluation activities that happen separately from the programme activities.

The earlier this happens, the easier it is to obtain a satisfactory summative evaluation.

Evaluation Methodology

Evaluation is a specialized form of social research conducted either by one or more independent consultants, or by the staff of a programme. As such, it makes use of any social research methodologies (see Surveys) that enable the answer to a set of evaluation concerns or questions.

These questions will be established together with any limitations on methodology or other obligations in a Terms Of Reference or evaluation contract document.

When the terms of reference are established, the evaluation team is able to consider a range of Evaluation Design Options and, on the basis of the design, use a range of different Data Collection Options.

Because elections involve social policy and can be highly politicised, evaluators have to consider the related concepts of Legitimacy and Reliability.

Finally, evaluation design is significantly determined by the time available and the size of the budget (see Time and Cost of Evaluations).

Terms Of Reference

Whether an evaluation is done by an outside team or the staff of the programme, it is necessary to write a comprehensive terms of reference (TOR) document.

Such a document enables all stakeholders (and a number of them are likely to become involved in the evaluation as recipients of the report, as interviewees, or as providers of core information) to understand and agree to the areas under investigation, the manner in which the enquiry will take place, the time schedule and, when the TOR constitutes a contract with external evaluators, the costs.

A TOR document typically covers the following areas:

• The Background. A short section setting out the reason for the evaluation and the background to the programme.
• Primary Purpose of the Evaluation. A statement descriptive of the overall intention of the evaluation so that it can also be judged according to the extent to which it achieves the goals of those who request the investigation.
• The Methodology to be Adopted. The most extensive section, this identifies whether the evaluation is to be primarily qualitative and descriptive, whether it is going to be quantitative and survey based, and the manner in which the evaluators should go about their job. This section can also include decisons about meetings to be convened of stakeholders, whether interviews are to be face to face, group, or telephonic, structured or unstructured.
• The Time Line. An evaluation typically must be conducted in a very tight time frame. To alert all participants to this, a detailed time frame may be included with the TOR. If this is not possible, then starting and report delivery dated will be specified.
• Additional Information. Each evaluation is different. It is likely there will be other conditions that have to be specified. These might include restrictions on who does the evaluation, whether the team includes or excludes staff, and other evaluator qualifications. There may also be instructions about the accessibility of certain stakeholders to the evaluators, and existing data that should be taken into account.
• Contractual Obligations. If the TOR is to be used as a tender document or as a contract between an external evaluator and the education programme, then the TOR will include information about cost, the manner in which the report will be submitted, the obligations of the evaluator to submit preliminary reports, confidentiality and disclosure of information and so on.

A TOR document is an essential tool in conducting an evaluation and it is likely to go through a series of drafts before being accepted by those whose acceptance is required.

Discussion on this matter is also facilitated by the development of a TOR. When an evaluation is determined upon, it might be assumed that the appropriate stakeholders are the educator team, their organization, and possibly the programme financer. As discussion ensues, other stakeholders can become apparent and the draft can be shown to them.

Care needs to be taken over the development of a TOR document, because it is the document to which evaluators refer during the evaluation. Many evaluations and evaluators are attacked (see Legitimacy and Reliability ) and they must inevitably refer to the TOR in their defence.

The more explicit the document, the better for all concerned.

Evaluation Design Options

A good evaluation will meld qualitative insights with quantitative analysis to establish a strong case for the recommendations being made. It will consider the utility value of its recommendations and will be conducted in such a way that the recommendations are likely to be adopted and implemented.

There are different ways of achieving these goals. Designing an evaluation requires some decisions about approach:

• who should be involved 
• the design to be chosen 
• the objects of the evaluation

Then, some micro choices must be made in relation to data collection instruments and methods (see Data Collection Options).

Independent Outsiders or Knowledgeable Insiders?

What relationship will there be between external evaluators (if any) and programme staff? These questions are addressed in Who Should Evaluate.

Typical Evaluation Designs

Many evaluations could more easily be considered as professional opinions. A single person (or possibly a small team) is given carte blanche to speak to stakeholders using a semi-structured interview, which might be conducted one to one or in a group. Evaluators will also have access to documents of the programme.

On the basis of these, and even using the series of interviews to test their developing opinion, evaluators prepare a report. The report can be submitted as is or, if there is time, tested with a representative group of stakeholders before submission.

Such an evaluation stands or falls by the reputation of the evaluator. It can be done rapidly and at limited cost. Because of the reputation of the evaluator, it can be done on the basis of a very generalized terms of reference (TOR) document. The evaluators are chosen because they know the field and the background to the programme, and because they are able to enter the world of the stakeholders with ease.

An Audit

A related design provides the evaluation team with access to all documentary material from the programme and conducts the evaluation entirely as a paper assessment. No interviews are conducted other than with those who commission the report, and all information necessary to the evaluator is considered to be available.

Such an evaluation can be extended by conducting general surveys based on preliminary indications of areas of interest so that there is additional data gathering. But the primary sources are documentary.

While such an investigation (perhaps of the voter education materials) can be useful, it can never replace an evaluation of a programme in action.

A Disciplined Conversation

The most complex, and most participatory evaluation, is that which can most adequately be described as a continuing discussion.

In such a design, the discussion begins with the development of the TOR. It may include the establishment of one or more standing committees of stakeholders to assess the progress of the evaluation, to discuss data and findings, and to dictate further research.

Evaluators typically play the role of group facilitators and technical assistants. They may also manage the collection of information, but there can even be data collection by individual stakeholders.

In such an evaluation, the final report is negotiated and can consist of a set of meetings at which recommendations or proposals are not only assessed but put into action by the responsible bodies or individuals.

How Close Should Evaluators Get?

Between these three typical designs lie many nuances, and each evaluation is approached by the evaluation team in the manner most likely to yield reliable results. For example, by its very nature, participatory evaluation becomes closely entwined with general programme implementation, which enables it to become increasingly self-monitoring rather than summative.

In such participatory exercises, the role of evaluators can become a contested one. They are outsiders with insiders' influence. Confusion can develop between evaluation and programme implementation: evaluation or reflection on experience becomes primary.

In an ongoing adult education group, this can be appropriate, but in a national education programme, it can become cumbersome and undermining of the general programme design.

Objects of Evaluation

Typically, an evaluation begins with a set of questions to which answers are sought. Such a list can become more extensive as the evaluation proceeds, or it may be discovered that a smaller and more concise list is sufficient.

These questions need to be generated in consultation with the organization that sponsored the evaluation. The various stakeholders can frame their questions differently or ask different questions, but the final list establishes the parameters of the evaluation and its objects.

By setting out a list of questions, the evaluation provides the first step toward utilization of the results. Relevance is predetermined, and ownership is partially guaranteed. The evaluators could discover that, as a result of ignorance or intentional misdirection, stakeholders have not asked a crucial question that they then add to the list [1].

Evaluators do so at their peril. They must provide their motivation for including such questions, and may be accused of going beyond their brief. It will be up to evaluators to establish the importance of the question for the outcome of the evaluation.

Notes:

[1] Typical examples of such questions might be those relating to the continuation of a programme, the role of the director or governing structure of the programme, or the outcome of a particular pet project of the staff.

Data Collection Options

As a form of social enquiry, all appropriate data collection options are available to the evaluator. As such, methods used to collect information about Assessing the Context for the education programme will be as appropriate for evaluation of the programme.

Methods include open-ended, structured group or individual interviews, focus groups, surveys, contact with interlocutors and intermediaries, consultative meetings with stakeholders, and the use of existing information (see Voter Background and Using Existing Data ).

Amongst this existing information are the client survey or post-meeting reaction forms, which should be used in all face-to-face and service-related educational programme elements. In addition, evaluators should have access to other documents prepared by the programme and investigations done by others.

The relationship between planning and evaluation should now become transparent. Much of the data required for programme development is collected during an extensive evaluation of the previous programme cycle. Linking programme development with previous evaluations is essential if efficient use is to be made of resources. Indeed, an evaluation might have amongst its terms of reference (see Terms Of Reference) the requirement that an assessment is made in preparation for a future programme.

Legitimacy and Reliability

There are two remarks that evaluators come to dread. The first is: "Who are you?" The second is: "It's not that I disagree with you, but did you really speak to the right people/ask the right question?"

The first remark deals with the legitimacy of the evaluators and hence their findings. The second deals with the reliability of the findings and is generally couched not as an attack on the results or outcome of the evaluation but rather on the methodology.

It is hard not to repeat the fault of the person making the remarks and become equally defensive, but there are things that an evaluation team and the educator can do to establish legitimacy and make sure the outcome is reliable.

Establishing Legitimacy

Evaluators can be selected as a result of a full discussion by all of the stakeholders who, having developed an agreed terms of reference (TOR) document, and having established a set of criteria for the evaluator, then select the person from a list of preferred people.

They may just as easily get appointed by the programme financer or the governing body of the programme, or even the director of the programme.

Even in the first case, there will be some who are surprised, for whatever reason, that the evaluation is happening, that they are being evaluated, and that the evaluator is the person they are.

First Meetings

It is essential for evaluators early on to establish who the key stakeholders are, and armed with a TOR document, interview them or meet with them to discuss the evaluation and seek their expectations of its outcome. It may be that these interviews are part of the evaluation design. In such a case, stakeholders who may possibly be resistant to the evaluation should be first on the list of those interviewed.

After establishing a basic agreement that the evaluation is going to happen, that its outcome will be prejudiced by noncooperation, and that cooperation and participation by all increases the likelihood of the results being more reliable, evaluators should maintain their relationship with the stakeholders throughout the evaluation.

Regular Reporting

Keeping in touch is done by adequate communication through meetings or the submission of interim reports.

It is also done by canvassing any alterations to the TOR or any recommendations that are likely to be a surprise or likely to have political import with stakeholders or at least with a set of evaluation guarantors.

Evaluation Guarantors

These guarantors will be a set of stakeholders who are highly committed to the evaluation and who have sufficient organizational power or a high enough reputation that their commitment will ensure the ongoing commitment of others. To people of this nature, regular reports should be made, together with a commitment from them that they will accept the outcome of the evaluation even if it comments on their role.

Evaluators have to ensure that the outcome of the evaluation expresses truths about the programme without fear or favour: but they have to do this in a way that enables the truths to be heard.

Ensuring a Reliable Outcome

Evaluators are presenting a set of recommendations to a group of people who have responsibility for programme development. The recommendations could affect the staff, financial base, operational design, and the personal aspirations of individuals within the programme or governing structures. A high-level evaluation can even have an impact on the outcome of an election or the renomination prospects of a member of the electoral authority.

It is essential that those making use of the results can believe in them and can trust them. In order to do that, evaluation reports need to consider the following:

• Ensuring a Transparent Process. Recipients of the recommendations need to understand the process that produced the recommendation. This includes the methodology for the collection of information, the manner in which the information was analysed, and the manner in which the evaluator came to the recommendation, including any testing and draft versions that may have been prepared. Even if there has been good communication up to the presentation of the report, evaluators will have thought harder about the recommendation than those listening to the report, unless the evaluation was entirely participatory. Evaluations conducted behind closed doors stand more risk of being misunderstood unless people are taken through the same thought processes as the evaluator.
• Testing Recommendations Against the TOR. The TOR document contains the overall purpose of the evaluation. It spells out how the recommendations will be used. So, the extent to which the report meets these explicit expectations is the extent to which the recipients will accept its results. When there are divergences from the TOR, these will need to be explained.
• Creating Support in Advance of the Final Report. Before delivery of a final report, evaluators should conduct a series of activities to ensure that the recommendations and the report are reliable. Amongst the possibilities open to evaluators, and depending on the extent to which the evaluation team is independent of the programme, are the following:
• an external audit of the draft report by a recognized expert in the field
• consultative meetings over the whole draft or aspects of the draft with stakeholders
• discussions with the likely implementer of a recommendation regarding its feasibility and validity
• the collection of additional information to test the recommendations in more detail

By the time a final report is prepared, support should be widespread, if not universal.

• Layering the Recommendations. It is likely that an evaluation will result in a range of recommendations: the essential, the obvious, the innocuous, and the controversial. Reports are more likely to be well-received if the recommendations are layered so that people can concentrate on the contentious or the difficult and accept the straightforward. Acceptance of the straightforward validates the report and ensures that discussion moves from the questions of reliability and methodology to questions of feasibility and implementation.

Time and Cost of Evaluations

Evaluations are expensive. Even if they are conducted by the staff, they take valuable time. If they are conducted by outsiders, this normally includes international or national teams requiring fees, accommodation, and air travel.

Once the team is in place, there are substantial costs for data collection: national or regional surveys, field work, stakeholder interviews, consultative meetings.

In general, evaluation studies should receive a budget of between one percent and three percent of the total programme budget. In some cases, additional money may be needed. It is unlikely that it will be possible to conduct an evaluation, even one internally conducted, for a very small amount of money. Programmes that are not expensive to conduct, and perhaps that do not intend to continue, should consider whether a traditional evaluation is worth doing.

In such cases, it may be sufficient to use staff meetings to assess the programme through a series of basic questions that address best practices and lessons learned:

• What went well?
• What could have been improved?
• What would we do differently if we did this again?

When a small percentage of an educational programme amounts to a substantial budget, educators should consider how best to spend the money in order to have an evaluation that is useful.

Quick and Dirty, or Deep and Thoughtful?

Researchers can extend evaluations over long periods of time. Usefulness requires good information as soon as possible.

Cost and time must be weighed carefully. It may appear to be true that spending more and taking longer will produce more reliable results. This may not be true, and whether those results will come in time for policy development and future planning must be assessed.

In general, evaluations err, unsurprisingly, on the quick and dirty. When there is a great deal at stake, and where there is more money available and a larger programme to evaluate, educators may consider setting in motion the deep study.

Both have their advantages. But there are many things that need to be done, and only so much time and money. Which will it be?

Baseline Studies

Because of the importance of measuring impact, evaluators and education planners have to resort to baseline studies. These are studies which, using a range of methods to establish a starting point for the programme, make it possible to see if anything has changed as a result of the programme.

This section suggests ways of obtaining baseline information, and how to manage if this is not possible.

This base or starting point is best established before a programme is implemented, and certainly before an evaluation is undertaken. It may be possible to recall how things were, but memory is closely bound with present perceptions and future aspirations.

With luck, it might be discovered that a study done by someone else is contemporaneous with the programme starting point. But, leaving it to chance or memory is not the best policy.

Difficulties of Measuring the Base

Studies operate on the assumption that they can determine what information is relevant and what is likely to change as a result of the programme, and on the assumption that information can be collected and analysed with sufficient speed, and that it will not change before the programme is implemented.

These are major assumptions. When the information is gross, it may be possible to do. A school building programme can count the number of schools in existence, implement the programme, and count again. A voter education programme being conducted in a society where people have not voted before can be fairly confident about that fact and can measure voting performance during the first election.

When there are a large number of factors affecting baseline information, establishing the baseline is much more difficult and requires considerable research and analytical skills.

Nevertheless, developing even a ragged baseline as a working model is preferable to having no idea where the programme is starting and how to tell whether it has made any impact at all.

Gathering Baseline Information

There are two ways in which baseline information can be obtained on a regular basis. When neither of these ways is available, educators have to weigh the need for a baseline study when they conduct their first programme.

It may be more cost effective to go into the field without a full understanding of the context and without the ability to evaluate impact fully so that a second cycle can use the data obtained in the first.

Periodic Elections

When elections are happening regularly, and where voter and civic education programmes have been institutionalized, it is possible to look back on an annual or periodic basis. This becomes the baseline information for each successive programme, and as the amount of information accumulates, it is possible to identify trends, to compare results of programmes, and to transfer lessons learned in one programme to another.

This can be done only if there is good record keeping from one election to another and if there is continuity in the organization (whether the electoral authority or a nongovernmental organization) conducting the educational programme.

Context Assessment Data

While an educational programme can be developed without undertaking a context analysis, this will surely mean it is limited in scope or less effective in outcome. Conducting a context assessment ensures that a great deal of the information considered necessary for a baseline study is in place.

The programme will have at its disposal basic information about voters or citizens, survey or anecdotal information about their educational needs, and certain basic geopolitical information that, while primarily intended to facilitate the programme, can also be used by evaluators.

Indeed, the relationship between evaluation and context assessment is a symbiotic one. The ultimate goal of a major educational programme is the cyclical continuity that enables much of the setup work to become regular.

In this way, there is the possibility of constant programme improvement, as well as a research cycle that maintains certain basic information about programme participants or target constituencies.

Unfortunately, many national education programmes do not repeat themselves at intervals regular enough to achieve this. In countries that do not institutionalize civic and voter education, large-scale programmes must constantly redo work and pay for it on each occasion.

If No Baseline Can be Established

It may not be possible to establish a reliable base. Evaluation can still happen, and even impact studies can happen.

An evaluation can be designed with a series of cycles so that the same area or question is revisited at periods during a programme. Or, evaluators can draw comparisons between similar areas, one that has experienced the programme and one that has not.

There are a number of social study techniques that may work, and most evaluations operate in this way. However, those who want to study the impact of an education programme over time should work at establishing a significant baseline.

Developing Organisational Capacity

Skilled evaluation is an essential aspect of adult education and social programming. Yet, there are few skilled evaluators available in many countries, particularly those with emerging democracies. This section suggests ways to change this unfortunate state of affairs and to use evaluation as a way of developing organizational and educational capacity.

The Responsibility of Those Commissioning Evaluations

Those commissioning evaluations should be considerate of the need to develop capacity in this regard. When education programmes have been substantially funded from or provided with technical support from the international community, there is a tendency to also commission international evaluators.

While this may result in a slightly less painful but certainly more expensive evaluation, it leaves countries continually dependent on external support. Evaluations should seek, in their conduct and choice of agents, to develop domestic organizational and personal capacity.

Evaluation is a form of enquiry that brings together organizational insights, social research, educational theory, and group skills. It enables evaluators to get very close to a programme without having had to implement it. These factors make an evaluation an ideal opportunity for learning and skills development for those who will have to conduct not only evaluations but also programmes of their own.

Those commissioning evaluations should consider the following:

• Are there existing indigenous independent institutions that could conduct the evaluation?
• Should electoral authority develop its own evaluation capacity?
• Can evaluation programmes be implemented without international leadership, if not without international participation?

Indigenous Independent Organizations

While few countries have the privilege of having organizations that specialize in educational or programme evaluation, many have social research institutes based at tertiary institutions, organizational development consultancies, or adult education centres and associations.

Because programme evaluation is an essential component of programme design, organizations that specialize in grant making may also have evaluation capacity.

In addition to the formal organizations from which an evaluator group might be drawn, there will also be individuals situated in academic institutions and civil society organizations who may have evaluation expertize.

Make Good Use of Limited Resources

If the programme is operating in an environment where such individuals or institutions have only limited experience, it may be possible to segment a larger evaluation and commission aspects of it. It may also be possible to establish a team of evaluators comprising an international and domestic component.

It can be possible to include in the terms of reference (TOR) some training and skills development responsibility, making a certain number of places on the team available to local people, making sure that there is participation in management and planning of the evaluation, and so on.

In each of these cases, the management of the evaluation is more complicated, but the developmental benefits are considerable.

Electoral Authorities

It is not appropriate for the election authority to entirely evaluate its own programme. It inevitably should draw in outside evaluators, even if they act primarily as facilitators of a participatory process. The election authority is likely to be involved with a broad range of educational initiatives. When it decides to strengthen civil society by outsourcing much of its work to independent organizations, its own ability to evaluate programme plans, to assess educational tenders (competitive bids), and to evaluate implemented programmes is a real advantage.

Specialized evaluators employed by an electoral authority can not only operate in the voter education area, but can also be useful in the evaluation of training programmes conducted for election officials, party agents, and others.

For these reasons, electoral authorities should consider building their own evaluation capacity, whether by starting with technical assistance from the international community or by deploying already qualified evaluators.

These people should participate in any programme evaluations if they are in the process of being established, and they should develop a library of evaluation studies. Because such studies are seldom published, this requires contact with other electoral authorities and educational institutions in order to obtain them.

Managing the Implementation of Evaluation Studies

Programme evaluation is an international endeavour. It is increasingly the domain of organizational development consultancies from the private sector and, as a result, there is considerable competition for work and the costs of commissioning individuals, companies or organizations is increasing.

Countries commissioning evaluation studies of their voter and civic education programmes should have the capacity to manage the teams of evaluators that get created. They need sufficient expertize to ensure that they are able to manage any external consultants and also to develop indigenous leadership capable of establishing and leading such teams.

In order to achieve this, they should look for opportunities for their electoral staff and others to obtain international experience.

Managing Evaluation Recommendations

Both formative and summative evaluations make recommendations. But those prepared at the conclusion of a summative report have a double burden. First, they must receive general acceptance. Then, they must await implementation until a future programme.

The formulation and communication of recommendations is crucial to the success of an evaluation. Unless the recommendations are taken seriously and have an impact on future work, the evaluation study will have been somewhat in vain. This is the case despite terms of reference (TOR) that ask primarily for a judgement of whether the programme has been effective. Surprisingly, a report that merely describes the impact of the programme is not always well-received. Those who commission evaluations want recognition and affirmation, but they also want suggestions for the way forward.

While the evaluation may discover a range of interesting things, it is constrained in its development of recommendations by the TOR or by any negotiated alterations to that document.

Testing Recommendations

Before conveying the final report, evaluators need to test the recommendations they plan to make with a representative stakeholder group. In the case of recommendations for future programmes, evaluators need to interact with those who can test the validity and feasibility of the proposal.

If the proposal has implications for budgeting, staff, or organization, it may be necessary to test validity and feasibility with an expert outsider. When this is not possible, the recommendation may need to include suggestions about how additional work can be done before implementation.

Directing the Recommendations

Recommendations not directed to an agency capable of implementation may be of general interest, but have the virtue of sermons rather than proposals for policy. If it is not clear at the time that the recommendations are drawn up precisely who will be required to implement them, then this should be ascertained during the testing or final reporting phase.

If necessary, a recommendation can be couched as a double proposal. Such a proposal might state: "This body should investigate and steer the setting up of a standing committee on programme evaluation."

Layering Recommendations

The evaluation team should organise the set of recommendations that emerge from the evaluation so they are easy to understand and also capable of prioritisation over time.

Some recommendations are simple to implement and arouse no controversy. These should be identified as such. For the evaluator, they have the advantage of establishing commitment to the report without requiring major investment from the recipients of the report.

Recommendations that are essential should be separated from those that are optional. Those that will make a fundamental impact on future programmes may be separated from those that might be tried if there is an interest.

Timing Recommendations and the Entire Report

Evaluations are time bound. Individual recommendations should have time schedules attached to them, particularly if the evaluation includes preliminary programme assessment responsibilities.

But the whole report should also set itself a shelf life. Recommendations that are initially passed over but are warmed up long after the report is presented, become less relevant because time has passed. This is particularly true of recommendations related to staffing or organizational change.

Evaluators should set a time limit on the efficacy of the recommendations and also suggest a way in which future evaluations may be considered if necessary.

Planning for Implementation

Evaluators move on. Much of what they suggest or write is not implemented. By increasing the ownership, legitimacy, and reliability of the process as the evaluation unfolds, it is possible to increase the chance for implementation of the recommendations.

There is also the face-saving implementation that takes place apparently without regard for the evaluation study, and evaluators should be prepared for this.

Nevertheless, there are moments when evaluators should consider working with their client to establish terms of reference that enable some planning for implementation, and some participation by the evaluation team in that implementation.

Teams that include members of staff of the client organization may have more success in this, as may evaluations that take time to communicate their findings and have these discussed in a planning and stakeholder forum.