‘it’s important to understand how…funds are being used, what outcomes are resulting from these programmes, and the impacts of the programmes for stakeholders.’
According to the Global Humanitarian Assistance Report 2019, published by Development Initiatives, international assistance from governments and private donors equalled around US$28.9 billion, representing a growth of almost one third since 2014. The report also stated that in 2018, ‘the amount requested through UN-coordinated appeals rose for the third consecutive year’.
With significant amounts of money – often from direct donations or from taxpayers – being directed either towards helping vulnerable populations or towards solving some of the world’s most pressing problems, it’s important to understand how those funds are being used, what outcomes are resulting from these programmes, and the impacts of the programmes for stakeholders.
This is why monitoring and evaluation (M&E) is important for Non-Governmental Organisations (NGOs).
Although the terms are typically closely linked and referred to together as ‘M&E’, monitoring and evaluation are two very different activities. Monitoring is normally done within the organisation, or via a third-party monitoring contract. Evaluation should never be done internally and always requires use of an external evaluator. Let’s take a closer look at the distinction between them.
An Introduction to Monitoring and Evaluation
Any project undertaken – whether in the non-profit realm of humanitarian aid or development work, or in the for-profit corporate arena – will need to be properly managed. Project Cycle Management (PCM) entails all of the management activities and processes for decision-making over the course of a project’s life cycle. Project monitoring and project evaluation occur at differing points in time along this life cycle.
There are typically four phases of a project life cycle: Project Initiation, Project Planning, Project Implementation, and Project Closure. Although monitoring is done during the Project Implementation phase, and evaluation is done during the Project Implementation and Closure phase, both have to be planned for in the Project Planning phase.
The monitoring of a project involves tracking the progress of a project to see how it is performing in relation to the outcomes and impact desired. Monitoring looks at the inputs (the resources used to conduct the activities for the project), the outputs (the direct, immediate results achieved by the project’s activities), the outcomes (the longer-term results attributable to the project), and the impact (the ultimate goal that is supported by this project, among others).
The evaluation of a project looks at whether the use of the inputs, as well as the outputs achieved, actually produced the desired outcome. Evaluation also considers whether the costs associated with the project were effectively used in achieving the desired outcomes, or whether other projects should be considered in the future as means for the desired outcomes.
The Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development (OECD) established a set of Evaluation Criteria to be used to determine ‘the merit, worth, or significance of an intervention’:
- Relevance looks at whether the intervention is doing the right things
- Coherence checks the compatibility of an intervention with other interventions
- Effectiveness determines whether the intervention is achieving its objectives
- Efficiency tracks the extent to which the intervention delivers, or is likely to deliver, results in an economic and timely way
- Impact assesses the difference the intervention will make
- Sustainability verifies whether the benefits of the intervention will last
For shorter-term projects, the evaluation is done at the end of the project. However, for longer projects, you should plan to have a mid-term evaluation, which can check to see if your project is on the right path. If there are any challenges, it’s important to correct them before it’s too late.
Monitoring and Evaluation Tools and Techniques
There are several tools and techniques used for M&E activities. The following are some of the most important that you should know about.
Logical Frameworks
Logical frameworks (also called results frameworks, depending on the donor) are the foundation of all M&E activities, as these provide the actual framework and benchmarks for a project.
When using logical frameworks for monitoring, in a best-case scenario, project planners will have defined an impact that they want to achieve – such as ‘Have drinkable water in X country’. They then go down the results chain, defining outcomes that would support this impact. One outcome could be a certain number of water filtration sites, while another could be working with the places that are polluters in order to decrease the amount of pollutants going into the water. From these outcomes, the project planners then develop outputs, which are basically ‘sub-goals’ that help to achieve the outcomes. Then, from the outputs, the activities are created.
Once this entire results chain is done – and only at this point – do the planners start to develop indicators that are adequate for measuring what the project aims to accomplish. If the project is related to the environment, indicators might include the level of pollutants in the water supply of a vulnerable village. If the project is health-related, indicators might include the percentage of children who have been immunised against measles.
Finally, the project planners will set the goals for the project, which should be decided upon based on their budget. For the environmental project, the goal could be a reduction of pollutants in the water by 80%. For the health project, the goal could be an increase in the percentage of immunised children to 90%.
The indicators, outputs, outcomes, and impact are then put into a ‘logical framework’ (also known as a logframe), which is a structured, formatted chart that allows for easy monitoring of the progress of a project. (We’ll go into more detail about logical frameworks below.) Because everything is measurable either through numbers or short qualitative sentences, one simply needs to track the data as the project proceeds. At the end of the project, you can measure the desired outcomes, and determine if you achieved your goals as laid out in the chosen indicators.
Tools for Collection, Management, and Visualisation of Data
Any development project will need to utilise tools to collect data – and to manage that data and put it into an easy format for analysis – as the project proceeds. Of course, while it is possible to collect data with pen and paper, now – in 2020 – there are much better ways to collect data.
From an efficiency and data protection point of view, digital data collection is the key here.
In recent years, Open Data Kit (ODK) has become the standard for the international development sector. While there are a myriad of other options, ODK is an open–source option that is continuously being further developed and does not bind you to one provider. There are several web applications that use the ODK standard and have additional features, such as case management. You can utilise free versions – such as ODK’s own ODK Aggregate and ODK Central (which you need to host yourself), and KoBoToolbox, which is free and supported globally by OCHA – or you can opt for other Software as a Service (SaaS) solutions, such as ONA, CommCare, SurveyCTO, and others.
After the data collection, you will need to store the data and use it for monitoring. Many organisations use Excel to store their data, but there are an increasing number of efforts to visualise data, to make sure it is used more frequently. Here, the most common visualisation approaches are the use of PowerBI or Tableau.
It should be mentioned that, currently, most people have shifted away from pure progress monitoring – such as counting the number meetings that people had – and more towards results-based monitoring (RBM), with a threefold goal:
- Reporting (against the logframe),
- Steering (using the data to determine if the project is on track and what is working or not working), and
- Learning (what worked well in the long-term, what can be scaled up).
‘It is vital for project planners to consider the monitoring and evaluation of a project early on – in the Project Planning phase. Many NGOs attempt to do the M&E themselves, but later end up realising they need external help.’
Evaluation Tools
Once the project is completed – or at a mid-term point for long projects – there are tools used by evaluators in order to assess the impact and effectiveness of the project in relation to its goals. Normally, you could have quantitative methods of evaluation, such as surveys and household- or individual-level questionnaires that allow you to say something more general about a whole group of people by interviewing a representative sample of it.
Then there are qualitative methods that go more in-depth. Here, some commonly used approaches are interviews of key informants, focus group discussions, case studies, outcome harvesting, or the Most Significant Change approach.
Within the context of the humanitarian community, many M&E consultants use the above-mentioned Evaluation Criteria developed by the OECD DAC, in order to ‘enable the determination of the merit, worth, or significance’ of a project.
It is vital for project planners to consider the monitoring and evaluation of a project early on – in the Project Planning phase. Many NGOs attempt to do the M&E themselves, but later end up realising they need external help.
Honest, impartial feedback is part of any evaluation, which is why it is vital that the project’s evaluation be conducted by a consultant who was not involved in its monitoring. This will help keep an evaluation free from bias, which is crucial to ensuring that development projects remain focused on truly improving the lives of project stakeholders.
This is a quick overview, and there is much more to say about M&E, but this will give a good idea of what all the fuss is about. If you or your company require monitoring and evaluation services in the humanitarian sector, or if you have any other questions regarding the services, please do not hesitate to contact us about a no-obligation consultation.