So now we're at level five, replication and evidence of generalizability. At this point, you have evidence that you've caused the outcomes and that your program or your company is quite reliable, you can scale this thing, the results that you have found with one group or in one location, you're going to find those outcomes and we hope impact in multiple locations. This is a really high standard of evidence. It's not that common. It's really impressive. You might have this level of evidence through a meta-analysis, or through a large scale study that has been done in numerous contexts. Because this is pretty rare, to have this level of rigor, let's take a look at one such study that actually proved that this program was quite generalizable across many countries. So, in 2015, a team of researchers, folks who are really well-known and highly regarded for their rigorous field experiments, randomized controlled trials, they published a paper again in Science, this very prestigious journal, and they reported the results of an experimental study that they conducted in six countries, six randomized controlled trials in each of the following countries: Ethiopia, Ghana, Honduras, India, Pakistan, and Peru. These folks, Banerjee, Duflo, Goldberg, Karlan, and others, did a massive study to look at how do we graduate people out of poverty. So, the researchers tested a program. It's actually called or referred to as a Graduation Program. It's a program designed to help the very poorest members in villages to escape extreme poverty. How do they graduate out of extreme poverty? So, let me say a little bit more about the problem here. One-fifth of the world's population lives on less than $1.25 a day. And the challenge is, how do we move the very poorest of the poor out of poverty? Can they graduate out of poverty into sustainable self employment? So, just to give you a sneak preview, the answer is, this program worked. It worked quite well. It worked across the country. What is this program? As I said, it's called the Graduation Program and it has some fascinating features. So, the participants, the beneficiaries of this program, they are chosen through this highly participative process in a village meeting followed by verification by the organization staff. What this really means is that, folks in these villages, they are selecting who is most in need, who could really benefit from this program. Beneficiaries are then randomly selected into two conditions. The folks who are getting the benefit of the Graduation Program, this multifaceted program, they are getting what's called a productive asset, that is, a goat, most commonly a goat, or sheep, or oxen, or bees, or chicken, or guinea pigs, or goods to start an informal store. They're getting assets that they can use to make money. They can sell wool, they can sell honey, they can sell eggs, if they use this productive asset. They also receive a number of other services. They receive training and support regarding how to care for the asset, how to take care of the goat. They receive basic food, or cash support for several months. That's to let them kind of stabilize the household and make it not too attractive to them to just sell off the goat, sell off the bees. No, we want you to keep this, the goats, the sheep, the bees, whatever it is that's going to allow you to earn an income. They also get a visit from a coach to teach them skills, life training to build confidence, they get a savings account where they are actually saving money and they get access to health education or health services. So, it's a quite a multifaceted program. I will say that we tend to know from research that when you do many things, it enhances effectiveness. A real silver bullet type programs don't, with just one intervention, usually don't work as well as this multifaceted program. Data were collected at baseline, again, in these six countries, and two years after the program ended, and one year later. So overall, they are collecting a slew of data from 10,000 households in six countries. But they find a really strong and encouraging results. The people who are in the treatment group compared to those who didn't receive the program, the people who get this program are working longer hours, their income goes up, their hunger goes down, their health improves, their financial inclusion goes up, savings go up a lot. The savings number was the most remarkable in the research results. Households in the treatment condition saved about 95 percent more money than those in the controlled condition. So, big effects from this program, they also were able to show that the program is quite cost effective. It's certainly expensive to provide all this coaching, to give these productive assets, but the benefits are far greater than the costs. So benefits compared to costs are 133 percent in Ghana and as much as 433 percent in India. So, a huge return for investment to have these kinds of results. Now, it's a randomized controlled trial. It shows a lot, but the researchers are clear that there are some questions they can't answer with their work like, is it better to give physical assets like a goat or could you just get cash and would that work as well? They can't answer that. How important was the training and the coaching specifically? They can't answer that. How long do the results last? They know they last for a few years. What about farther down the line? So, the randomized controlled trial certainly doesn't answer every question, but the research has gotten a lot of attention, and it seems to be changing government aid policies. So the program is already being scaled up in Pakistan, in India, in the Philippines, and the government of Ethiopia plans to expand the program as well. So, big effects from this very rigorous research done, as I said, across 10,000 households studied over several years in six countries. This is a great example of level five rigor. It's rare. It's impressive. And hopefully, when there's this big of an effort, it really does lead to policy change. That said, you got to really push for the policy change, too, to make this all worthwhile.
