The following blog introduces AIF’s new framework of Learning, Evaluation, and Impact by founding Director Dr. Varna Sri Raman. Dr. Raman has more than a decade of experience in social research including research design, management, implementation, and evaluation. Her career includes leadership roles at Kantar Public, BBC Media Action, and Sesame Street in India. She has worked with clients such as UNICEF, the Bill and Melinda Gates Foundation, CARE and Plan International, on many cross-sectoral research projects.
As the American India Foundation continues to expand its work and reach in India, we get asked very often, “So what does AIF do?” There are, in my view, two ways to answer this question: the first is to explain the intention behind AIF’s work in India, and the second is by demonstrating the impact of the work we do. Interestingly, both methods have to do with evidence, albeit with different parts of the research process. The first falls squarely in the bracket of monitoring, and the second in the domain of evaluation. Regardless, the important thing about impact, is that measuring it, is the recognized way to demonstrate the value that AIF is delivering to its beneficiaries as a whole. It is this recognition that has, first and foremost, driven the new investment and interest in Learning, Evaluation and Impact (LEI).
It is in this context that I am particularly pleased to share results from an impact assessment of our Digital Equalizer (DE) program in India. Undertaken in 2016 by a reputed third party research group, the study for the first time is a coherent and comprehensive assessment of the DE programmatic intervention in its third year as of 2016, in the western Indian state of Gujarat. The DE program aims to empower teachers to use technology to enhance the teaching-learning process and improve students’ learning outcomes in schools that are under-resourced and attended by students from marginalized communities. The study’s mixed methodology included quantitative and qualitative interviews with headmasters, teachers, students and program staff in both DE and non-DE schools.
The results of the study, while certainly not fool-proof, point towards positive directional improvement in DE schools when compared to non-DE schools insofar as attendance and teacher effectiveness is concerned. These two variables alone have time and again proven to be the chief determinants of children’s learning outcomes in a wealth of rigorous research evidence globally[1]. There is also some positive movement in the learning outcomes of children in DE schools, with a marginally better performance than children from non-DE schools. To read more about DE and access the results of the study, click here.
The DE study for AIF is a beginning towards a robust and consistent approach to evidence gathering and generation, in keeping with AIF’s commitment to evidence-based approaches for all stakeholders. I can think of several reasons for the move to evidence-based work, both from a programmatic perspective for internal stakeholders as well as from an advocacy-perspective for external stakeholders, both of which are important. The chief of these reasons, of course, is that impact is an increasingly important benchmark for any non-profit—i.e., what demonstrable, tangible outcomes are being achieved to benefit the population we are serving, and how is that demonstrable through evidence?
The second most important reason has to do with the twin objectives of monitoring and evaluation, both of which are forms of impact measurement. AIF is now looking at ways to embed impact measurement within the organization. The chief principle we hope to follow is to set up widely visible ways of judging our own success at delivering to beneficiaries. It is our belief that this transparency is a vital signal to both donors and beneficiaries that AIF is committed to a process of continuous internal improvement (in service delivery) and that we are open to be held accountable for our performance. I will confess that this visibility also has an additional motive, which is to make our progress more visible for ourselves. In my experience this is one of the best ways to stay invested in one’s work and to feel a sense of achievement, in a sector where success is very hard to come by if not at times unpredictable, despite the level of intellectual and research acumen that is required to successfully carry out the work.
The third and perhaps the least-discussed reason has to do with advocacy. Through our programs—such as the DE assessment demonstrates—AIF first and foremost advocates a programmatic approach to creating impact on the ground for beneficiaries. In the DE program, for example, while the intent is to improve children’s learning outcomes and technology is the tool, the fulcrum of impact for the schooling ecosystem is enabling the teacher to teach better, through access to engaging teaching-learning materials (TLMs) and adequate knowledge, to transact both the tech and teaching know-how in the classroom. It is the teacher who is the key to a transformative classroom experience for the child and everything else, including impact, flows from that. DE, therefore, through its experience advocates a teacher-centered approach to improving education.
Now, if we want to advocate the DE approach, we need to communicate the impact of DE effectively. Using empirical data as evidence of DE’s outcomes and benefits helps build the DE narrative, a big part of which is impact, not just output. This is the key to effective advocacy: the idea that data is essential in order to make the most of programmatic outcomes and that testimonials (read case studies) play a strong supporting role. Packaged together, this is the impact story and this is what is worth being disseminated and this is what constitutes social proof.
What is social proof you ask? Social proof is validation for AIF’s existence, it is the evidence of the need of our mission and the evidence of why AIF needs to continue to work—the evidence of our continuing purpose. Measuring impact can support both the mission and the continued purpose. Our vision for the LEI function of AIF is centered on measuring the right thing and measuring cost-effectively. As we move towards setting up LEI systems we will continue to share what we learn about impact measurement collectively—in terms of efficacy, efficiency and sustainability.
[1] See: John Hattie, Visible Learning, 2011. John Hattie developed a way of ranking various influences in different meta-analyses related to learning and achievement according to their effect sizes, wherein he ranked 195 influences that are related to learning outcomes from very positive effects to very negative effects. His study covered six areas that contribute to learning: the student, the home, the school, the curricula, the teacher, and teaching and learning approaches. His research is based on nearly 1200 meta-analyses. Available online at https://visible-learning.org/hattie-ranking-influences-effect-sizes-learning-achievement/
