All posts by nick

12.09 2010

In the previous post, Nick shared the exciting news that data from health centers in Mozambique is now available for us to see. Living in an environment where information is easily available at our fingertips, we often take the availability of data like this for granted. In reality, obtaining data from a place where information systems and health records aren’t commonly used is quite challenging!

Yet, the information is crucial not only because it can strengthen the overall performance of the health system by enabling data-driven decision making by our partners on the ground, it is necessary in order to measure progress and impact. And as I’ve written previously, VillageReach is committed to monitoring and evaluating the progress and impact of the Dedicated Logistics System in Mozambique.

The key questions to ask when attempting to evaluate impact are whether or not the Dedicated Logistics System has met its objectives to improve health system performance and increase immunization coverage. In the baseline evaluation, we attempt to answer questions such as: how many children are immunized and how often do health centers run out of vaccines. Unfortunately, this type of information is just not available. There are no immunization registries to search. Health centers don’t keep patient records. Even if they did, we still wouldn’t be able to know how many children are not immunized because there are no vital record systems either. These things just aren’t tracked.

In order to get the information we believe is valuable to answer these questions, we went out and collected it. In order to estimate the immunization coverage, we conducted more than 800 household surveys in randomly selected villages across the two provinces of Cabo Delgado and Niassa. The sample size gives us enough statistical power to make an estimate of the true immunization coverage rate with 95% confidence and because the villages were randomly selected, the sample is representative of the population. In each household, we essentially asked whether or not the children living there had received certain vaccines. In addition, we conducted surveys in more than 60 health centers in those villages to get an idea of how the health centers were performing. We worked with a group in Mozambique who hired and trained local staff to conduct the surveys and complete the data entry. The process took about nine months from the time we first sought Ministry of Health approval for the study until we started seeing the data.

Despite a few challenges along the way including delays in schedules with field teams, traveling time to remote villages, and correcting for concerns with the data such as missing entries and incorrectly completed surveys completed, we have real numbers. We have real data from the very last mile. Finally, we can use this data to answer our questions regarding how many kids aren’t being reached by vaccination services and what we can do to improve this. Using this information, we can tailor our program activities in ways that can make the biggest impact and that is what we intend to do.

Before we can share the results publically, we need approval from the Ministry of Health. We’ve started the process and have already shared the results with provincial leaders in Cabo Delgado and Niassa. We will be presenting the results to the Ministry later this month or early next year. We are looking forward to sharing our results with you as soon as possible. Stay tuned.

Jessica Crawford
Program Associate

Read full story
12.03 2010

Five months ago as I sat here in Seattle, I found it a difficult exercise to imagine the performance of our health system strengthening programs operating worlds away from my desk chair. Even though geographically much closer, local health staff and management also face the same challenge caused by the extreme difficulty to pull routine data out of these rural, remote communities. Yet an understanding of the performance at this level is critical for health system improvement.

Now, five months after the implementation of our information system, local health staff and I are now starting to see this health system data.

In partnership with the Ministry of Health, my colleagues and I at VillageReach implemented a internet-enabled logistics management information system (LMIS) called vrMIS to provide information management to our health system strengthening program in the three northern provinces of Mozambique, Cabo Delgado, Niassa, and Nampula, serving a population of over 7 million people.

vrMIS, an application of the open source LMIS platform called openLMIS, collects routine data and then provides reporting analysis to health workers and their management in order to increase the effectiveness and efficiency of the supply chain from the provincial medical supply warehouse down to the health center or what we call the “last mile” of the health system.mapping_visualization3

Five months since vrMIS was implemented, I can see the data uploaded by the health staff. It’s incredible to see data from these health centers as well as powerfully insightful. I see medical supply inventory levels for these health centers. I see the number of tetanus vaccinations given last month. I see the refrigerator at a particular health center is currently not functioning correctly because of a leak in the gas line. I can see so many things. Upon further review, I start identifying performance trends and from this areas for improvement.

vrMIS shines a light on the last mile of the health system – all the way down to the health center level. This illumination provides new insights to the reasons why a health system is performing the way it is and enables management to take informed, appropriate action.

Still in Seattle, I can stop trying to imagine the performance of the health system. Now, I just open an Internet browser.

Nick Amland
Program Assistant

Read full story
10.14 2010

In Melinda Gates’ recent TEDX talk, she posed the question; how is it that coke can get its product to “far flung” places all over the developing world while governments and NGO’s seem to have so much difficulty delivering health commodities like vaccines?

Well this got us thinking a little about how this applies to us, especially because our work in Mozambique is specifically focused on this effort. Actually, it isn’t the first time we’ve thought about the coke model in our work. We used Mrs. Gates’ observations on Coca Cola as a framework to understand some of our own efforts. Pretty quickly, we see parallels in the way that Coca Cola and VillageReach do their work, at least in the way that it was presented in the TED talk!

Let’s start with the piece about real-time data. We’ve known that this is essential for our program in Mozambique to function even from the start. In fact, it is an integral part of our program design. We need to know when and at what health post there is a stock out of a particular vaccine, if there are problems at particular health centers with their refrigerators, or if immunization numbers drop suddenly from a region so that we can identify the problem and fix it. This is not as easy as it sounds. In our first attempt at collecting routine data, we used paper forms to collect the data and then sent these forms back to our Seattle office for input into our management information system. After the data was entered and reports were generated, we would send electronic versions of data analysis reports back to the field. This process took up to 3 months. We quickly found that the latency this process created significantly decreased the value of the data analysis reports. The reports were out of date and no longer as useful. However, this demonstrated to us how essential and potentially impactful real-time data could be.

Since this experience, we’ve released two versions of this management information system. Our latest version, vrMIS3 (built on the openLMIS platform), was developed on the principle of real-time data. We utilize several different data collection methods (paper forms, SMS, smartphones, computers) allowing us to extract real-time data in any environment. We now have vrMIS3 deployed that does just that – provides health workers with real-time data to enable more appropriate, more rapid decision making. In this way, we can help ensure that the vaccines are safely arriving to these “far flung” health centers and interruptions in vaccine services are avoided.

Mrs. Gates mentions the success of using local entrepreneurs as central to coke’s success. Again, we identified this opportunity and established it as part of our model from the beginning with the establishment of VidaGas. Local entrepreneurs made this happen. We’ve also been successful with “marketing” and by this we mean making people want to come to the health center for immunization services and through mass campaigns. While this hasn’t been a direct objective for us, it has been something we realize has been essential to our success. More to come on these two points in another blog!

We’ve seen success with this model in our demonstration project and are working on getting our “product” out there to the “far flung” places Mrs. Gates speaks about – just like coke. Now if we could just find a way to bring in those billions of dollars like coke does, we’d really be able reach a lot of children! Maybe we just need to add more sugar?

Jessica Crawford & Nick Amland

Read full story
09.24 2010

After spending nearly two weeks in an intensive course offered by the Institute for Health Metrics and Evaluation for its incoming fellows (and a couple of fortunate UW graduate students like me), I have an even greater appreciation for the role of evaluation in our work in global health and for the complexity and difficulty in doing it well. In her lecture on Evaluating Health Programs, Dr. Emmanula Gakidou, pointed to what is called “The Evaluation Gap” where billions of dollars from international donors and national governments are being channeled into health programs in low and middle income countries but we know relatively little about what programs are working and how well they are working. The reason being is that most of them are not rigorously evaluated.

Even while researchers continue to develop and test new interventions such as vaccines, diagnostic tools, and drugs through thorough clinical research, we don’t know how best to deliver them in countries with weak health systems. The result is an innovation pile-up where proven interventions to prevent and treat disease are available yet millions of people are dying because these interventions don’t reach them.

As a graduate student in public health, it seems to me that the field of global health is turning in this direction and placing a lot more value on measuring impact. As President Obama said in his speech at the MDG Summit; “let’s move beyond the old, narrow debate over how much money we’re spending and let’s instead focus on results-whether we’re actually making improvements in people’s lives.” We need to know what is working and what isn’t so we can better our efforts and get the interventions out to the people who need them.

Unfortunately, evaluation is difficult to do well. As I quickly learned in the IHME course, there are some serious limitations to deal with ranging from poor data quality and availability to the fact that the methodology of conducting a rigorous evaluation just sometimes isn’t possible or is really expensive. As expressed in a Lancet editorial: “Evaluation matters. Evaluation is science. And evaluation costs money. It’s time that the global health community embraced rather than evaded this message.”

VillageReach makes a sincere commitment to evaluation of its programs and has ever since its inception. For example, as we begin to scale-up the Dedicated Logistics System in Mozambique, we are engaged in operations research to inform our program decisions. In addition to routine monitoring, we are conducting baseline evaluations in every province followed by process and outcome evaluations. We want to know what is working and more importantly, what isn’t working and why, so we can ensure that the resources we put into our programs really make improvements in people’s lives and that those interventions make it to the people who need them. We’ll keep you posted on our progress.

Jessica Crawford, MAPS, MPHc
Program Associate

Read full story
08.06 2009

Over at the GiveWell blog, they’ve been asking some difficult but incredibly important questions about international aid projects.  Back in July, they explored the goal of sustainability.  Like GiveWell, we’ve noticed that sustainability is more and more often included as a requirement from funders yet it often remains vaguely defined and difficult to quantify.  VillageReach decided to establish businesses precisely because we believe that for many infrastructure gaps in the health system, they are the only truly sustainable way to address the problem.  And our social business VidaGas is a self sustaining organization, but as GiveWell notes, it has been a larger challenge achieving sustainability on the program side.  Even though we attempted to plan for long-term sustainability from day one in our Mozambique program, it has been challenging to convince the government to maintain the system even when we can show evidence of significant success.  The inertia of the status quo is a powerful obstacle to sustainable change.

So, this begs the question- how do you define and measure sustainability and how important should it be as a goal of a program?  This is especially important when you consider that sustainability often ends up being a trade-off with other qualities- for example, in order to make our program as sustainable as possible, we try to only include elements that we know the government is capable of carrying on after we leave- this can mean sacrificing impactful elements  because they are too expensive, too labor intensive or just too unprecedented for the government to assume control of.   GiveWell concludes that sustainability should be considered “a desirable goal, but not a reasonable requirement.”  The goal of sustainability is fundamental to VillageReach but a more candid conversation about what this truly means could be of enormous benefit to funders and implementers alike, both of whom tend to through the word around without really questioning its value.  On a similar theme- Phillip LaRocco has a humorous note to the “development posse” about lightening up- admitting when things are difficult and cutting through the clichés (of which sustainability certainly must be one of the most ubiquitous!) to truly impact the communities we serve.

GiveWell also explores investing in a small charity.  They note that “giving to VillageReach is a high-risk, high-upside proposition” and honestly, we couldn’t agree more.  VillageReach is proud of its dynamic and innovative approach- our President Allen is fond of saying that our theory of change boils down to “we see, we do, they see, they do.”  Basically, we recognize the problem and because we are small and agile we can create a customized model to address it and then advocate for both the recognition of the problem and the adoption of our model to others.  GiveWell was impressed by our rigorous monitoring and evaluation of our program, an expensive proposition that many small non-profits forgo because they view it as a luxury.  We, however, see quantitative evidence of impact a fundamental necessity in order to convince others of the value of our model.   But it is true that the same things that make us high-impact and allow for change on a scale disproportionate to our size also mean that we don’t have the security of a large, highly diversified non-profit working in a well understood area.  We rely on donors to recognize both the problem of last mile health system strengthening and the value of our solution.  We are thrilled to have GiveWell endorse our approach and we hope to report more success in the future.

Read full story