Friday, 15 August 2008

Performance Measurement as a Critical Leverage Point for Improving International Development Impact

Individuals and organizations need good performance feedback in order to be effective and meet their objectives. In the private sector there is the luxury of sales, profit and other clear measures that indicate output and outcome performance. These measures are used to identify and reinforce successful business patterns, such as effective strategies or high performing individuals, and to disrupt and correct unsuccessful patterns. They are critical to learning, development and long-term success. To survive, private sector organizations have to meet the performance expectations for these measures that are set by their funders. This mechanism helps to ensure that high-performing organizations and individuals are able to succeed and expand their impact while low performing organizations and individuals are not.

Unfortunately, measuring output, and particularly outcome performance is very difficult and generally poorly done in the international development sector. Outputs are often complex and highly variable, there are rarely any clear measures of outcome performance and outcomes are usually affected by a vast array of factors outside the control of any single organization or even the international development sector as a whole.

In the absence of high quality output and outcome feedback information, people naturally focus on things that are easy to measure, such as activity and costs. Consider this example. You have 100 euro to donate to charity. Care says that 95% of your money will go directly to projects for people in need. Save the children says that 50% of your money go directly to projects for people in need. Given this choice, most people would immediately say "that's a no-brainer, I'll give my money to Care!".

However, without knowing the track record of the two organizations for meeting outcome objectives, how do we know that a 5% investment in organizational capability will benefit those in need more than a 50% investment? How much should be invested in understanding the dynamics of the problems affecting the people in need? How much should be invested in training staff, in supporting technology, in measuring performance, etc? Without reliable performance feedback, noone knows. So, the overiding assumption that drives behaviour in the system is that more projects is good, overhead is bad. The consequence of this dynamic is that international development organizations almost invariably over extend themselves across too many projects and massively under invest in long-term strategic capabilities.

It is for this reason that I have been trying to help international organziations and the system as a whole improve the way that outputs and outcomes are measured, and how this information drives accountability, learning, improvement and long-term impact. I believe that this is a critical leverage point for achieving rapid and dramatic improvements in the perfromance of the international development system as a whole. In a future blog I'd like to discuss some of the areas of investment I think are required, but a few of the key ones are robust measurement methodologies; a conceptual framework for continuously improving the collective understanding of the systemic dynamics affecting development outcomes; widely adopted data standards; a culture and practice of data sharing within and across organizations; tools to facilitate data and knowledge generation and sharing; and a culture of accountability.

With regard to the tools, I have been working on some ideas for improving the efficiency and effectiveness of collecting the data required for output and outcome performance feedback. I have been working with a technology called xforms, which enables the creation of very user-friendly electronic forms that can be accessed through a web browser and used off-line. Using electronic forms makes data collection easier and faster, improves the quality of data and enables broader data sharing.

A number of experiments using electronic forms are being conducted but the forms are typically on a small hand-held device and the small screen is a major constraint. I have been thinking that the ideal interface for the forms would be an A4 or Letter sized touch screen tablet computer. It would only need enough power to run a cut-down version of Linux and a Firefox browser. It could have a relatively small flash memory card for storage and wouldn't need most of the bells and whistles that make computers complicated and expensive today.

One option I've been thinking about is the XOXO pc from the One Laptop Per Child project. It's their second generation kid-focused laptop that is under development. It has two touch screens that fold together. One of the screens acts as a keyboard when necessary. The attraction of this device is that it is cheap, has the specs required and enough screen real estate to make form completion fairly easy. It also is a device that hopefully will be very common and familiar in the settings where the data collection would take place.

Another option is an initiative launched by Michael Arrington of Tech Crunch to produce a "dead simple web tablet for $200". This is only at the dream stage but would be a fantastic device for field data collection. Check out the initiative here. Search for "Fraser" to see my comments on this blog where I describe how the device could be used by UN agencies. After a wise crack that the device could be developed with UN slush funds I had to come to the defence of the UN because there is such low awareness of what the UN does and the positive impact that it has.


Nadejda Loumbeva said...

Very interesting thoughts. Just a few comments:

I agree that ''more projects is good, overhead is bad'' does not contribute towards IOs and the UN being better organisations. Rather, it effectively cripples them to learn from what they do. Still, IOs and the UN are, at least in some ways, forced into the ''more projects is good, overhead is bad'' mode of thinking by donors, partners, other stakeholders and perhaps even beneficiaries. Why? Probably because of the perception on behalf of donors, partners, other stakeholders and beneficiaries of what is meant by effectiveness, and efficiency. In order to change this situation, it would be most important to change this perception. If this changes, then the whole system in which IOs and the UN operate would be in a better position to change for the better. No?

Perhaps a most direct way of changing this perception could be in the context of multi-stakeholder partnerships. Brokering a partnership is one thing, implementing it is very different. Implementing partnerships should create conditions for culture change in the participating organisations ... in this way a change of perceptions. Partnerships are opportunities for organisations that may not be used to their full potential.

Changing this perception, though, would not be sufficient. A next step would be to, as you say above, enable a conceptual framework for continuously improving the collective understanding of the systemic dynamics affecting development outcomes - I understand this as enabling these organisations to be learning organisations by cultivating a systems-thinking approach to development.
The rest of what you mention above:
-robust measurement methodologies;
-widely adopted data standards;
-a culture and practice of data sharing within and across organizations;
-tools to facilitate data and knowledge generation and sharing;
-and a culture of accountability ...
all of these would be at least part of the ingredients that would enable these organisations to cultivate a systems-thinking approach to development. These would also enable them to learn from experiences in a multi-stakeholder sort of a way, i.e., together with those who are their partners and those who are their beneficiaries.
In my mind, though, the ''conceptual framework for continuously improving the collective understanding of the systemic dynamics affecting development outcomes'' comes first ... if not, then we would have IOs and UN organisations that are just more efficient, not necessarily more effective. It seems to me that organisations tend to fall in this trap easily ... whilst this trap is getting bigger and bigger.

Finally, on the xforms, XOXO pc-s and touch-screen tablet computers - what wonderful ideas! But, there is the danger of not using these well. It would be important to ensure these are serving not only efficiency, but also effectiveness. In other words, what these ideas are plugged into is important.
A further way of leveraging the full potential of such devices could be by enabling not only IO and UN staff but also partners and beneficiaries to input data in the devices, or in some other way participate in the process of measurement. What sort of a system and dynamics one would create for this to work well would be a different story.

Thanks for your very interesting thoughts. !!

Alanna said...

One thing I love is using cell phones for the collection is distrubted data. Cheap, existing technology. You'd need to simplify your data needs to an SMS level, though, which I guess may be impossible.

nadejda loumbeva said...

Speaking of mobile phones, these are, apparently, the mobile technology of choice according to discussions during a recent joint conference of IAALD, AFITA and WCCA (collectively, these stand for agricultural information management and ITs/ICTs in agriculture – please see above link). Apparently, mobile phones are increasingly the mobile technology of choice when it comes to agriculture knowledge and information sharing among farmers, i.e., people on the ground. ... One might think this works well because the level of sophistication of the data and information farmers want to share is not very high and so easily fit into an sms and/or a digital photograph (could be predominantly the case, but not exclusively!). Still, even in societies where the level of data and information is very sophisticated, such as Japan, mobile phones have been on the rise for many years. ... In Japan, apparently almost everything people do on the internet (check and respond to email, download files, etc.) is done though personal mobile devices such as i-phones. ...
Some people go as far as saying (as I hear was said at the above conference) that laptops and computers are dead, whereas mobile phones/mobile technology is experiencing a boom most likely to stay on.
I thought this could add an interesting perspective to the above discussion on IO performance measurement ... the more flexible the means through which the data is gathered, the better. And, sometimes, perhaps, the closer the measurement device is to what is used/could be used by the people of the ground (beneficiaries), the better, in order to enable good communication. All depends on what the organisation does, what it ends up measuring itself against and the local context.
And so, mobile phones might be better than laptops and computers ...But I agree with Alanna that if this were to work well, mobile phones would have to be at the right level of sophistication in order to take in needed data.

Lavonne said...

People should read this.