FMR 14 : published July 2002

UNHCR

Real-Time Evaluations

A real-time evaluation (RTE) is a timely, rapid and interactive review of a fast evolving humanitarian operation undertaken at an early phase. Its broad objectives are to gauge the effectiveness and impact of a given UNHCR response and to ensure that its findings are used as an immediate catalyst for organisational and operational change.

Criticism of UNHCR's response to humanitarian needs during the 1999 Kosovo operation prompted the agency to introduce RTEs in order to be able to undertake rapid, analytical evaluations of ongoing emergencies and provide suggestions for improvement at a time when they can still make a difference. In June 2000, the Evaluation and Policy Analysis Unit (EPAU) issued a provisional RTE framework. To date RTE missions have assessed programmes in Sudan/Eritrea, Angola, Pakistan, Iran and Afghanistan. Recent UNHCR experience has demonstrated that RTEs have many advantages in terms of speed, impact and partnerships. Within the organisation the RTE concept has quickly become accepted by managers and staff at all levels. Among UNHCR's key external stakeholders (donor states, NGOs and other UN agencies) the introduction of RTEs has been welcomed as an indication of the organisation's commitment to transparency and enhanced operational effectiveness. EPAU is working to develop the RTE methodology and share UNHCR's experience with partner organisations.

An RTE is not simply a cut-down version of a conventional humanitarian evaluation; rather it is interactive and intended to provide immediate inputs into an ongoing operation. An RTE should be able to analyse an emergency response on the basis of a number of benchmarks, including UNHCR's protection mandate, policy statements, sectoral and thematic guidelines, the standards set out in the organisation's Emergency Handbook and the specific objectives of the operation in question. An RTE will seek to analyse difficulties and find effective responses and to capture examples of good practice and incorporate them in planning, programming and training activities.

An RTE should be launched as soon as a new emergency has occurred or appears imminent. Initially, the evaluators will be active participants in the crisis cell established for the emergency, collecting and reviewing relevant documentation on a systematic basis. Evaluation teams interview senior managers and establish contact with key offices and personnel in the field. The evaluators' role is not simply a passive or analytical one as they use their skills and experience to provide advice to emergency managers and alert them to impending problems. Before they leave a field location an RTE team holds an interactive debriefing with UNHCR staff and, if possible, with representatives of partner organisations. Back at HQ they quickly provide debriefings to the High Commissioner and other members of senior management, to Executive Committee members and to NGOs.

A real-time evaluator should be familiar with emergency operations and evaluation methodologies.While external participation in a real-time evaluation should not be excluded, the team leader will ideally be a UNHCR staff member but not one normally attached to EPAU.

UNHCR has found that the advantages of an RTE can be summarised in three words: timeliness, perspective and interactivity. Findings are made available quickly to a wide range of stakeholders in time to make a difference to an unfolding operation. Real-time evaluators have been directly involved in the emergency planning process and in facilitating and sustaining dialogue between field and HQ staff. Skilled real-time evaluators are able to approach an emergency from an informed perspective, bringing knowledge and lessons from past emergency evaluations into the evaluation process and its transparent outputs.

RTEs are not risk free. Time for consultation with beneficiary populations is limited. A narrow focus on UNHCR's own operations can detract from a system-wide perspective. There is a risk of placing controversial findings in the public domain at a time when UNHCR is engaged in delicate negotiations with states and other actors. Despite these limitations, UNHCR has found RTEs to be a valuable new tool which we will continue to develop.