There are two remarks that evaluators come to dread. The first is, 'Who are you?' The second is,
'It's not that I disagree with you, but did you really speak to the right people/ask the right
question?'
The first remark deals with the legitimacy of the evaluators and hence their findings. The second
deals with the reliability of the findings and is generally couched not as an attack on the results or
outcome of the evaluation but rather on the methodology.
It is hard not to repeat the fault of the person making the remarks and become equally defensive,
but there are things that an evaluation team and the educator can do to establish legitimacy and
make sure the outcome is reliable.
Establishing Legitimacy
Evaluators can be selected as a result of a full discussion by all of the stakeholders who, having
developed an agreed terms of reference (TOR), and having established a set of criteria for the
evaluator then select the person from a list of preferred people.
They may just as easily get appointed by the programme financer or the governing body of the
programme, or even the director of the programme.
Even in the first case, there will be some who are surprised, for whatever reason, that the
evaluation is happening, that they are being evaluated, and that the evaluator is the person he is.
First Meetings
It is essential for evaluators early on to establish who the key stakeholders are, and armed with a
TOR, interview them or meet with them to discuss the evaluation and seek their expectations of its
outcome. It may be that these interviews are part of the evaluation design. In such a case,
stakeholders who may possibly be resistant to the evaluation should be first on the list of those
interviewed.
After establishing a basic agreement that the evaluation is going to happen, that its outcome will
be prejudiced by noncooperation, and that cooperation and participation by all increases the
likelihood of the results being more reliable, evaluators should maintain their relationship with the
stakeholders throughout the evaluation.
Regular Reporting
Keeping in touch is done by adequate communication through meetings or the submission of
interim reports.
It is also done by canvassing any alterations to the TOR or any recommendations that are likely to
be a surprise or likely to have political import with stakeholders or at least with a set of evaluation
guarantors.
Evaluation Guarantors
These guarantors will be a set of stakeholders who are highly committed to the evaluation and who
have sufficient organisational power or a high enough reputation that their commitment will ensure
the ongoing commitment of others. To people of this nature, regular reports should be made,
together with a commitment from them that they will accept the outcome of the evaluation even if
it comments on their role.
Evaluators have to ensure that the outcome of the evaluation expresses truths about the programme
without fear or favour; but they have to do this in a way that enables the truths to be heard.
Ensuring a Reliable Outcome
Evaluators are presenting a set of recommendations to a group of people who have responsibility
for programme development. The recommendations could affect the staff, financial base,
operational design, and the personal aspirations of individuals within the programme or governing
structures. A high-level evaluation can even have an impact on the outcome of an election or the
renomination prospects of a member of the electoral authority.
It is essential that those making use of the results can believe in them and can trust them. In order
to do that, evaluation reports need to consider the following:
- Ensuring a Transparent Process. Recipients of the recommendations need to understand
the process that produced the recommendation. This includes the methodology for the collection
of information, the manner in which the information was analysed, and the manner in which the
evaluator came to the recommendation, including any testing and draft versions that may have
been prepared.
Even if there has been good communication up to the presentation of the report, evaluators will
have thought harder about the recommendation than those listening to the report, unless the
evaluation was entirely participatory. Evaluations conducted behind closed doors stand more risk
of being misunderstood unless people are taken through the same thought processes as the
evaluator.
- Testing Recommendations Against the TOR. The TOR contains the overall purpose of
the evaluation. It spells out how the recommendations will be used. So, the extent to which the
report meets these explicit expectations is the extent to which the recipients will accept its results.
When there are divergences from the TOR, these will need to be explained.
- Creating Support in Advance of the Final Report. Before delivery of a final report,
evaluators should conduct a series of activities to ensure that the recommendations and the report
are reliable. Amongst the possibilities open to evaluators, and depending on the extent to which
the evaluation team is independent of the programme, are the following:
- an external audit of the draft report by a recognised expert in the field
- consultative meetings over the whole draft or aspects of the draft with stakeholders
- discussions with the likely implementer of a recommendation regarding its feasibility and
validity
- the collection of additional information to test the recommendations in more detail
By the time a final report is prepared, support should be widespread, if not universal.
- Layering the Recommendations. It is likely that an evaluation will result in a range of
recommendations, the essential, the obvious, the innocuous, and the controversial.
Reports are more likely to be well-received if the recommendations are layered so that people can
concentrate on the contentious or the difficult and accept the straightforward. Acceptance of the
straightforward validates the report and ensures that discussion moves from the questions of
reliability and methodology to questions of feasibility and implementation.
