How evaluating impact contributes to driving investment in adaptable and scalable strategies.
Several years ago, a mango farmer in Burkina Faso told us something that forever changed our approach to impact evaluation.
“I don’t want to answer any questions,” the farmer said as one of our impact surveyors approached him. “Here you come to ask us the same silly questions that you go sell to aid sponsors…. You’ll ask me my name, my family size, the kinds of goods I have.… Go take the answers from the [other organizations’ surveys] we provided last year.”
Related topics: Enhancing capacity for resilient agriculture – How to achieve large-scale impacts – Sustainability & Resilience in Rural America
As a mission-driven nonprofit, it’s imperative that we understand if our services are working—and how we can do better. But to that mango farmer, understandably, our approach to impact evaluation felt extractive. Since that day, we’ve continually evolved our impact evaluation efforts to generate the most value not just for us, but for our partners in rural communities. Here’s how.
Why evaluate impact?
In the first post of this series, we talked about why Root Capital measures impact: it helps us gauge our progress and figure out how to allocate limited resources toward the most positive change. We start by estimating the likely impact of prospective loans and borrowers through our social and environmental due diligence. (Note: Read more about our due diligence approach in part two of this series). As a complement to this, in 2011, we began conducting deep-dive impact evaluations. These evaluations measure the actual impact our services have on our client agricultural businesses, and the impact these businesses have on smallholder farmers. To date, we have conducted impact studies with more than 30 clients and have 10 additional studies underway.
Impact evaluation builds an evidence base that has both internal and external applications. Internally, our impact evaluation findings help us learn what’s working well and adapt our services accordingly. They allow us to test and validate our theory of change. Externally, these findings help us maintain accountability to our stakeholders: both the rural communities we serve and the donors and investors whose capital enables our work. For our client agricultural businesses, our evaluations help them define their own impact and receive feedback from their members. Also, because we share results of our impact studies publicly, they can inform the efforts of other rural service providers. By learning what works, we contribute to a body of knowledge that drives investment in adaptable and scalable strategies.
How do we evaluate impact?
To date, Root Capital has focused on measuring four buckets of impact:
1) The effectiveness of our lending approach (e.g., has the business grown and what role did Root Capital play in that growth?);
2) The impact that our lending clients have on their farmers;
3) How those impacts vary for different demographic groups, like women and youth; and
4) Farmer and employee satisfaction with our client businesses.
To evaluate these, we use a variety of methods—ranging from case studies to performance evaluations to quasi-experimental and experimental approaches. The exact method depends on what we’re evaluating, what our learning objectives are, and what resources we have available. In most cases, we do our evaluations in-house. Recognizing the vital importance of evaluation, over the years we’ve built an in-house impact team with robust experience and expertise. These staff members have a deep understanding of our work, our clients, and rural communities that enriches our evaluation efforts. When needed, we complement this with external expertise.
What have we learned?
Over the last decade and 30+ impact studies, we’ve learned a lot. Here are just a few of the lessons that have helped us evolve our approach to impact evaluation.
Lesson #1 – Utilize a participatory and client-centric approach
Going back to that mango farmer in Burkina Faso. What he told us, in 2011, was a wake-up call for Root Capital. Good intentions are not enough. We need to put our clients, their farmers, and employees at the very center of our evaluations. This means meeting them where they are: journeying to remote communities, working around their schedules, and—most importantly—understanding what will make our evaluations useful for them. We have learned to start each impact study by engaging clients directly in the survey design, and to share results back to them first, before disseminating them publicly. This participatory approach maximizes value not just for us, but for the communities we serve.
Lesson #2 – Pick the right evaluation approach for your resources
In 2016, Root Capital worked with Innovations for Poverty Action’s “Goldilocks Initiative” to review our approach to evaluation. They helped us understand that randomized evaluations aren’t generally feasible for us because of the size of our loans, the nature of our work, and the fact that we work with a limited number of businesses at a time. When developing an evaluation approach, we balance our learning objectives with our available resources to determine the best-fit methodology. As a nonprofit, Root Capital relies on philanthropic support to conduct impact evaluations—and we are fortunate to have philanthropic partners who recognize the importance of this work. But we still have to balance the cost of evaluation with the benefits that we, our clients, our funders, and other partners stand to gain. Our in-house evaluation capacity helps us keep costs far lower than the typical cost of external impact evaluations. By relying on existing monitoring data, hiring and training data collectors ourselves, and employing analytic methods that require fewer rounds of data collection, we are able to conduct evaluations in-house for a fraction of the cost of an external evaluation. If the scale or approach of the evaluation requires it, however, we work with external consultants to provide data collection, technical expertise, or even full evaluations.
Lessons #3 – Develop research and evaluation partnerships
When you’re bootstrapping your own impact evaluations, it’s essential to have partners. At Root Capital, we’ve learned to recognize our own limitations. We are not, after all, an impact evaluation firm; we are a nonprofit with a passion for evidence and accountability. So we work closely with external experts to complement our internal evaluation efforts. For example, as we implemented our latest quasi-experimental evaluation, we relied on technical advisory support from Dan Stein, Chief Economist at IDInsight, who worked with us as an independent consultant. We also work with in-country research partners, such as Participatory Development Associates, who provide invaluable support with data collection and independent analysis. External partners have also conducted rigorous independent studies of Root Capital that inform our global strategies—for example, economists Rocco Macchiovello and Arthur Blouin published a study highlighting Root Capital’s impact in a peer-reviewed journal last year. Through these partnerships, we maximize the quality of our evaluations, increase the breadth of our learning, and independently test our theory of change.
Lesson #4 – Use innovative approaches to increase rigor when needed
Though our resources are limited, we continually strive for increased rigor, in order to build even more confidence in our findings. In 2018, with support from the Wagner Foundation, Root Capital launched its first quasi-experimental studies. We set out to measure the income gains of farmers affiliated with our client businesses—a very important, but hard-to-measure, outcome. We didn’t have the funds to do multiple rounds of data collection; nor did we have baseline data from before we began lending to these businesses. With this in mind, we employed an approach called retrospective comparison group matching—a fancy name for the process of statistically matching treatment farmers (those affiliated with our client businesses) to comparison farmers (farmers who produce the same crops in the same regions, but who do not belong to a business financed by Root Capital) on certain baseline characteristics. This process reduces pre-intervention differences between the treatment and comparison groups, helping prevent bias in our evaluation results. (In part four of this series, we’ll present the findings from those incremental income studies.)
These lessons, and many others, guide our ever-growing efforts to understand the impact of Root Capital’s work. For example, in the last year, we added four studies evaluating the performance of our advisory services to complement those we’ve done on our lending. We’ve come a long way since interviewing that mango farmer, thanks to a commitment to continual learning and improvement—and we plan to continue evolving. By investing in impact evaluation, we ensure that “transforming rural communities” are more than just words in our mission statement.
About the author: Katie Naeve is the Director of Impact & Partnerships & Lead of the Women in Agriculture Initiative at Root Capital
EDITOR’S NOTE: The opinions expressed here by Impakter.com columnists are their own, not those of Impakter.com. Photo Credit: Root Capital
