There are two remarks that evaluators come to dread. The first is: "Who are you?" The second is: "It's not that I disagree with you, but did you really speak to the right people/ask the right question?"
The first remark deals with the legitimacy of the evaluators and hence their findings. The second deals with the reliability of the findings and is generally couched not as an attack on the results or outcome of the evaluation but rather on the methodology.
It is hard not to repeat the fault of the person making the remarks and become equally defensive, but there are things that an evaluation team and the educator can do to establish legitimacy and make sure the outcome is reliable.
Establishing Legitimacy
Evaluators can be selected as a result of a full discussion by all of the stakeholders who, having developed an agreed terms of reference (TOR) document, and having established a set of criteria for the evaluator, then select the person from a list of preferred people.
They may just as easily get appointed by the programme financer or the governing body of the programme, or even the director of the programme.
Even in the first case, there will be some who are surprised, for whatever reason, that the evaluation is happening, that they are being evaluated, and that the evaluator is the person they are.
First Meetings
It is essential for evaluators early on to establish who the key stakeholders are, and armed with a TOR document, interview them or meet with them to discuss the evaluation and seek their expectations of its outcome. It may be that these interviews are part of the evaluation design. In such a case, stakeholders who may possibly be resistant to the evaluation should be first on the list of those interviewed.
After establishing a basic agreement that the evaluation is going to happen, that its outcome will be prejudiced by noncooperation, and that cooperation and participation by all increases the likelihood of the results being more reliable, evaluators should maintain their relationship with the stakeholders throughout the evaluation.
Regular Reporting
Keeping in touch is done by adequate communication through meetings or the submission of interim reports.
It is also done by canvassing any alterations to the TOR or any recommendations that are likely to be a surprise or likely to have political import with stakeholders or at least with a set of evaluation guarantors.
Evaluation Guarantors
These guarantors will be a set of stakeholders who are highly committed to the evaluation and who have sufficient organizational power or a high enough reputation that their commitment will ensure the ongoing commitment of others. To people of this nature, regular reports should be made, together with a commitment from them that they will accept the outcome of the evaluation even if
it comments on their role.
Evaluators have to ensure that the outcome of the evaluation expresses truths about the programme without fear or favour: but they have to do this in a way that enables the truths to be heard.
Ensuring a Reliable Outcome
Evaluators are presenting a set of recommendations to a group of people who have responsibility for programme development. The recommendations could affect the staff, financial base, operational design, and the personal aspirations of individuals within the programme or governing structures. A high-level evaluation can even have an impact on the outcome of an election or the renomination prospects of a member of the electoral authority.
It is essential that those making use of the results can believe in them and can trust them. In order to do that, evaluation reports need to consider the following:
- Ensuring a Transparent Process. Recipients of the recommendations need to understand the process that produced the recommendation. This includes the methodology for the collection of information, the manner in which the information was analysed, and the manner in which the evaluator came to the recommendation, including any testing and draft versions that may have
been prepared. Even if there has been good communication up to the presentation of the report, evaluators will have thought harder about the recommendation than those listening to the report, unless the evaluation was entirely participatory. Evaluations conducted behind closed doors stand more risk of being misunderstood unless people are taken through the same thought processes as the evaluator.
- Testing Recommendations Against the TOR. The TOR document contains the overall purpose of the evaluation. It spells out how the recommendations will be used. So, the extent to which the report meets these explicit expectations is the extent to which the recipients will accept its results. When there are divergences from the TOR, these will need to be explained.
- Creating Support in Advance of the Final Report. Before delivery of a final report, evaluators should conduct a series of activities to ensure that the recommendations and the report are reliable. Amongst the possibilities open to evaluators, and depending on the extent to which the evaluation team is independent of the programme, are the following:
- an external audit of the draft report by a recognized expert in the field
- consultative meetings over the whole draft or aspects of the draft with stakeholders
- discussions with the likely implementer of a recommendation regarding its feasibility and validity
- the collection of additional information to test the recommendations in more detail
By the time a final report is prepared, support should be widespread, if not universal.
- Layering the Recommendations. It is likely that an evaluation will result in a range of recommendations: the essential, the obvious, the innocuous, and the controversial. Reports are more likely to be well-received if the recommendations are layered so that people can concentrate on the contentious or the difficult and accept the straightforward. Acceptance of the straightforward validates the report and ensures that discussion moves from the questions of reliability and methodology to questions of feasibility and implementation.
