National Office Failures


Where’s the Beef? Getting to work on impact measurement

james-haga-failure-2015By James Haga, VP Strategy & Investment

It’s said that history is written by those who can best tell the story. Yet in today’s context of increasing global transparency, stories alone don’t cut it.  People expect that what you say should be backed up by evidence—the harder, the better. This rings true, and points to an uneasy state of failure in EWB: despite a 15-year track record, we have too little hard evidence to illustrate our impact in pursuit of poverty reduction.

For many of us close to the organization, this can be anxiety-inducing. Just a few months ago, a new prospective donor rightly asked for historical impact data from our portfolio of venture investments and my face went purple as I grasped for partial answers. Having dispensed millions of dollars and endless people hours over the years, what can we categorically say has been accomplished? It’s uncomfortable just thinking about the question.

We can talk eloquently about our work, drawing on anecdotes and case studies, but this approach doesn’t hold up under scrutiny. Data is important because it gives us information about how our efforts, namely those of the ventures we support, are having an impact on the lives of other human beings. For intensely mission-driven people, knowing how we’re doing in pursuit of our mission is the ultimate bellwether—and right now, we still don’t have enough information to go on.

This past year, our attempt to measure a series of metrics with each of our investments fell flat, failing to ensure that the data we sought would first and foremost be useful to the people we work with, social entrepreneurs and their clients or partners.

Over the years, Engineers Without Borders’ investment strategy has evolved many times. This past year, we doubled down on supporting scalable and socially impactful business models in their earliest stages of development, introduced an Investment Committee to scrutinize our proposed investments and challenge our ideas of what has systemic or disruptive potential, and refined our suite of services designed to accelerate venture impact.

This evolving investment strategy has made static measures of impact less relevant. Still, this masks laziness on our part—we have been content to change the yardstick we use to measure ourselves without taking stock and aggregating the data available to us, imperfect as it may be.

In recent months, we’ve outlined a series of simple but meaningful impact metrics, such as the profiles of customers reached, that we are beginning to measure in collaboration with our ventures, taking cues from them on what data would enable them to better serve clients or partners. We’ve started investing more staff time – including outside expertise – towards sharpening our impact, data and learning approach. We still don’t have perfect clarity on what exactly are the right things to measure. That said, our guiding principles are to ensure that what we measure doesn’t become an impediment or burden for our ventures, and that the data collected serves our learning objectives, not just our donor reporting.

Given our approach to creating impact in the world, over the long-term our success will depend on having impact in the following three ways: over the long-term we will aim to make sense of our impact in three ways:

  1. Supporting a few of our ventures to scale and directly impact hundreds of thousands of lives;
  2. Learning from our investments about what works in delivering scalable social and how it can be leveraged to influence sectors and have systemic impact;
  3. Executing 1 and 2 above really well allows us to de-risk early-stage, systemic impact investing for a range of other actors.

All three of these success pathways depend on collecting quality data and using it to understand our impact.  While I’ll stop short of calling it a data revolution, we are soberly working to substantiate our compelling stories with hard data.


read more
RonanOWhere’s the Beef? Getting to work on impact measurement