Thoughts from the Last Mile Welcome to the VillageReach Blog

Tag Archives: Evaluation

05.25 2012

Rapid Diagnostic Test (RDT) Data and Distribution Systems.

As a partner in Mozambique’s health system, VillageReach is undertaking a study about the shortage of RDTs in the rural health centres. This is the second in a series of updates we expect to provide on the work – see the first post here. The study will conclude this fall.
The study involves gathering and analyzing data that health workers collect as they provide testing services, and using that data to analyze consumption as it informs the test distribution. Data about the use of RDTs is critical to operating a well-functioning supply chain and ensuring that health centers have the right tests at the right time. This data allows planners in the health system to purchase the right quantity of tests and distribute them throughout the health system while minimizing wastage and expiry.
Until now, the national health system in Mozambique has not distributed standardized forms for collecting this data. However, the lower levels in the government recognized the need and value of the data and implemented their own systems. In the first visit made by the study team in January this year, it was found that the data collected was of very poor quality, due to lack of standardized instruments or forms to use for recording data and a lack of training for the health workers. The forms used to collect the data were undergoing numerous changes, which complicates the process of data collection at the service delivery point.
Tim2
The system for distributing tests and collecting data about tests was designed with the following flow: the RDTs are amassed at a provincial medical store, redistributed to district storage facilities, and then transported to rural health centers. The image on the left represents the ideal.
Last week we visited again and found that key improvements had taken place in the last four months.
1. Systems and processes are in place to collect data about the use of tests. These are being used properly and routinely. This is no small feat in a country so vast and full of infrastructure challenges that can limit access to the health centers.
2. The Logistics Management Information System is installed and in use at the district level. That is a major accomplishment, and one that will improve the distribution and supply of medicines throughout the country.
But this visit revealed another fundamental observation. Despite improvements over the recording and availability of data about test consumption, there continues to be shortages and stock outs of tests in the health centers. Specifically, I observed situations where there were too few RDTs and a number of them had expired before being used. If the data is there to make the system work, then what is happening?
For many cases, we discovered the question can be answered by the following factors:
• A lack of vehicles for transport limits regular distribution of RDTs and medicines to health centers;
• Where there are vehicles available for transport, there is often a lack of fuel;
• Where there are vehicles and fuel, there is a chronic lack of allowances for personnel to carry out the distribution of medicines.
These factors greatly influence the availability of RDTs at the health center level. With delays in distribution, RDTs may expire before they are used, ultimately reducing the quality of service at the health centers. Tim1
All this ultimately results in poor distribution system of RDTs used in the province and a higher cost of distribution. In addition, the lack of regular distributions results in incomplete and inaccurate data about the health centers being reported back to administration levels within the health system. This image on the right represents what we see in reality.
These factors, all too common occurrences for health systems with limited resources, highlight the importance of improving the quality and capacity of the underlying infrastructure the health centers depend on in order to support so many remote communities.

Timóteo Eduardo Chaluco, Monitoring & Evaluation Officer

Read full story
12.09 2010

In the previous post, Nick shared the exciting news that data from health centers in Mozambique is now available for us to see. Living in an environment where information is easily available at our fingertips, we often take the availability of data like this for granted. In reality, obtaining data from a place where information systems and health records aren’t commonly used is quite challenging!

Yet, the information is crucial not only because it can strengthen the overall performance of the health system by enabling data-driven decision making by our partners on the ground, it is necessary in order to measure progress and impact. And as I’ve written previously, VillageReach is committed to monitoring and evaluating the progress and impact of the Dedicated Logistics System in Mozambique.

The key questions to ask when attempting to evaluate impact are whether or not the Dedicated Logistics System has met its objectives to improve health system performance and increase immunization coverage. In the baseline evaluation, we attempt to answer questions such as: how many children are immunized and how often do health centers run out of vaccines. Unfortunately, this type of information is just not available. There are no immunization registries to search. Health centers don’t keep patient records. Even if they did, we still wouldn’t be able to know how many children are not immunized because there are no vital record systems either. These things just aren’t tracked.

In order to get the information we believe is valuable to answer these questions, we went out and collected it. In order to estimate the immunization coverage, we conducted more than 800 household surveys in randomly selected villages across the two provinces of Cabo Delgado and Niassa. The sample size gives us enough statistical power to make an estimate of the true immunization coverage rate with 95% confidence and because the villages were randomly selected, the sample is representative of the population. In each household, we essentially asked whether or not the children living there had received certain vaccines. In addition, we conducted surveys in more than 60 health centers in those villages to get an idea of how the health centers were performing. We worked with a group in Mozambique who hired and trained local staff to conduct the surveys and complete the data entry. The process took about nine months from the time we first sought Ministry of Health approval for the study until we started seeing the data.

Despite a few challenges along the way including delays in schedules with field teams, traveling time to remote villages, and correcting for concerns with the data such as missing entries and incorrectly completed surveys completed, we have real numbers. We have real data from the very last mile. Finally, we can use this data to answer our questions regarding how many kids aren’t being reached by vaccination services and what we can do to improve this. Using this information, we can tailor our program activities in ways that can make the biggest impact and that is what we intend to do.

Before we can share the results publically, we need approval from the Ministry of Health. We’ve started the process and have already shared the results with provincial leaders in Cabo Delgado and Niassa. We will be presenting the results to the Ministry later this month or early next year. We are looking forward to sharing our results with you as soon as possible. Stay tuned.

Jessica Crawford
Program Associate
VillageReach

Read full story
09.24 2010

After spending nearly two weeks in an intensive course offered by the Institute for Health Metrics and Evaluation for its incoming fellows (and a couple of fortunate UW graduate students like me), I have an even greater appreciation for the role of evaluation in our work in global health and for the complexity and difficulty in doing it well. In her lecture on Evaluating Health Programs, Dr. Emmanula Gakidou, pointed to what is called “The Evaluation Gap” where billions of dollars from international donors and national governments are being channeled into health programs in low and middle income countries but we know relatively little about what programs are working and how well they are working. The reason being is that most of them are not rigorously evaluated.

Even while researchers continue to develop and test new interventions such as vaccines, diagnostic tools, and drugs through thorough clinical research, we don’t know how best to deliver them in countries with weak health systems. The result is an innovation pile-up where proven interventions to prevent and treat disease are available yet millions of people are dying because these interventions don’t reach them.

As a graduate student in public health, it seems to me that the field of global health is turning in this direction and placing a lot more value on measuring impact. As President Obama said in his speech at the MDG Summit; “let’s move beyond the old, narrow debate over how much money we’re spending and let’s instead focus on results-whether we’re actually making improvements in people’s lives.” We need to know what is working and what isn’t so we can better our efforts and get the interventions out to the people who need them.

Unfortunately, evaluation is difficult to do well. As I quickly learned in the IHME course, there are some serious limitations to deal with ranging from poor data quality and availability to the fact that the methodology of conducting a rigorous evaluation just sometimes isn’t possible or is really expensive. As expressed in a Lancet editorial: “Evaluation matters. Evaluation is science. And evaluation costs money. It’s time that the global health community embraced rather than evaded this message.”

VillageReach makes a sincere commitment to evaluation of its programs and has ever since its inception. For example, as we begin to scale-up the Dedicated Logistics System in Mozambique, we are engaged in operations research to inform our program decisions. In addition to routine monitoring, we are conducting baseline evaluations in every province followed by process and outcome evaluations. We want to know what is working and more importantly, what isn’t working and why, so we can ensure that the resources we put into our programs really make improvements in people’s lives and that those interventions make it to the people who need them. We’ll keep you posted on our progress.

Jessica Crawford, MAPS, MPHc
Program Associate
VillageReach

Read full story

Search Blog:

Newsletter Sign Up

By providing your email address you agree to receive periodic email news from VillageReach.


By submitting this form, you are consenting to receive marketing emails from: VillageReach, 2900 Eastlake Ave E, Suite 230, Seattle, WA, 98102, http://www.villagereach.org/. You can revoke your consent to receive emails at any time by using the SafeUnsubscribe® link, found at the bottom of every email. Emails are serviced by Constant Contact

Support Life-Saving Innovation

Your tax-deductible contribution extends the reach of our programs and supports new innovation- from idea to impact.

Donate Today

Malawi healthcare worker