Smart M&E

Back in 2020, during a casual conversation with my colleague and now friend, Manoj, at the William J. Clinton Foundation (WJCF), I shared the concept of automated, narrative-rich reports using R Markdown (RMD). I told him they are commonly used for teaching data science (if in the R language); and how we may want to use them for some of our static, but regular analyses, for the ongoing JEET project. There are many things (reports, manuscripts) written about the JEET project, but I have found this resource, by FIND, to be the most useful - for a broad overview. 

​

Manoj was more ambitious. He said -> but what if we create them for daily M&E activities? Is that possible. He was concerned about our ground operations --- we had 50+ field officers at the time, spread across 11 states and 22 districts, who were responsible for engaging private sector doctors within the idea of JEET.

​

The idea of JEET was simple - that (private sector) doctors notify any and all presumptive and diagnosed TB patients to the government of India. Additionally, they refer the presumptive patients for a CBNAAT test to confirm the diagnosis. If confirmed, they provide patients with the option of free drugs sponsored by the NTEP (National TB Elimination Program, GoI), among other things. However, this simple idea was difficult to achieve. Field officers did not always know how to optimize their visits - should they keep visiting the doctor who "does not notify", so as to convince him/her. Or should they ensure that those who do, should continue to - and hence visit them instead. Should their visits be based on how many patients this doctor receives? Or should it be based on if the tea-seller next to the doctor's clinic adds fresh cardamom to their tea- it could be a motivation factor, in all fairness!

Even as some districts and their operational managers had somewhat optimized these operations - the data carried a lot more answers, for which the right questions were yet to be asked. ​

​

what if we didn’t need to run a hundred different analyses or copy-paste outputs into Excel or/and PowerPoint every single time? What if the reports could be updated automatically, and actually tell the story of the data — clearly, visually, and accessibly?​

​

It took us several months from thereon. We had a ton of data - the lat/lons of private sector providers, number of times a particular field officer visited them, whether or not they notfiied their patients, and if so, did they utilize NTEP services such as free diagnostics or/and drugs. But everything, was disaggregated ---- in hundreds of CSV files. Our discussions varied - but included the following key themes: 

• which questions are to be answered in the report

• what statistical graphs are relevant and easy to read

• what color scheme is appealing to the eyes

• do we really need to show this detail

• is our commentary respectful....

​

Several months of labor led to a lightweight M&E solution designed to bridge data and decision-making. Using R Studio and R Markdown, we built a pipeline that:

• Automated data cleaning, merging, and analysis

• Produced detailed yet mobile-friendly HTML reports

• Enabled real-time updates, all from a single codebase

​​

These reports could be shared as bite-sized HTML files over WhatsApp, and hence could be easily on mobiles - under low-network settings. For the first time, the field officers evolved from being merely being data collectors to those who could use that data for planning their day, week and month! Things became transparent, data became a friend, and not just a task they had to complete at the end of the day!

​

The reports - were later customized to support overburdened treatment coordinators - who many a time were managing 200 patients at one point in time. The simple to read reports allowed for understanding differential burdens within the team, leading to an improvement in sharing of responsibilities. 

What began as an idea during a conversation turned into a system that supports real-time, evidence-based action across multiple points in TB care. ​

​

In March 2025, the solution was exhibited at the India Innovation Summit - demonstrating the utilization of solution for national level scale up. For easy distribution, we have hosted an anoymized (and shorter) version of the solution on Rpubs. If you are curious, I will be happy to share the underlying code with you - for recreation of the solution to your own use case. 

​​​

​