Before we get started, let me quickly introduce myself. My name is Amol Navathe. I'm an Assistant Professor of Medicine and Health Policy at the University of Pennsylvania, and I'm also a general internist practicing at the Philadelphia VA Medical Center. The subject of this course "evaluation" has been a core interest of mine throughout my career. In addition to academic work, I've worked in the private sector and as an official at the Department of Health and Human Services in the federal government. Throughout these experiences, I have found that evaluation and its techniques have been invaluable in driving insights and informing all sorts of decisions, including federal health policy, health system and health insurance program implementation, and the focusing of clinical resources, just to name a few. These experiences, applying evaluation concepts and techniques in the real world, also serve to ground me in the pragmatic. So, our approach together will be conceptual, yet practical. We will focus heavily on applications and examples. So, our goals together are twofold. First, I aim to get you as excited about evaluation as I am. Second, I want you to have a core conceptual understanding of evaluation, and feel comfortable in applying evaluation techniques in your own work setting. So, in this course, we'll learn about one, why we need to do evaluation, two, methods for evaluation, and three, applying evaluation principles in the real world. Let's start with the first. Why do we need to do evaluation in health policy? Well, some may argue the number one reason is that health policy is always changing. Just think about what's happened in recent years, like changes in the health insurance industry. Such as with the Affordable Care Act, which affected the number of insured and uninsured. Medicaid expansion was a core component there, and so was the launching of health insurance exchanges. Other recent health reforms with large impacts include, bundle payments. Meaning, paying a fixed price for an episode of care, through the bundle payments for care improvement program, a special interest of mine. And fundamental reform of how Medicare pays physicians, and that's just to name a few at the federal level. In addition, non-government stakeholders across the health sector are responding to policy changes and constantly trying to innovate. Here, some examples are, hospitals, innovating through quality improvement initiatives. Startups, rolling out new digital technologies, like text messaging applications directed at underserved populations. Insurance companies, rolling out new disease management programs. There is tonnes of innovation internal to organizations too. For example, implementation of Toyota lean production system principles in physician offices, to make care more efficient and predictable. What types of program or policy changes are happening in your workplace? We'd like to hear about it. The second main reason to study evaluation is that we have to measure. That is, if we're going to learn from all this innovation and change, the mantra of evaluation is going to have to be measure, measure, and measure. So, what if we don't examine carefully? Well, we risk not learning and maybe worse, we may learn and disseminate the wrong things. For example, the VA, or Veterans Affairs, a system of hospitals that take care of former military service men and women recently tested a longitudinal care management program for their high need, high cost patients. The reason is that this small number of patients account for the majority of bad outcomes and use of resources. So, they identified these individuals, then started tracking their spending and health outcomes. Plus, the VA randomly selected some high risk veterans to enroll in a long term care management program, to help them navigate the system. Sounds like a good idea, right? Well, the results were staggering. Spending dropped by 40%, hospitalizations dropped by 30%, success, right? But wait, luckily, the research and implementation group also measured the individuals not randomly selected for the program. Those individuals made up a control group. And guess what? The control group had almost identical declines in spending in hospitalizations. Oops. There was major regression to the mean. Meaning, veterans who are high utilizers in one year, probably weren't in the next year. So, the intervention looked better than it really was, and without a control group, we would have come to the wrong conclusion about its effectiveness. This highlights the importance of a control or comparison group. Let me say that again, this highlights the importance of a control or comparison group. This will be a recurring theme that I cannot emphasize enough. It's that important to evaluation. Otherwise, we risk propagating some very erroneous information. Okay, so now hopefully I've convinced you that proper evaluation technique is important, but another important point, it's not easy either. Let me spend a few moments on why this is. First, let's start with selection bias. Selection bias is the selection of individuals, groups or data for analysis such that the sample obtained is not representative of the population intended to be analyzed. That means, proper randomization is not achieved. Let me give you an example, back in World War II, the American and British forces were discussing how to best reinforce warplanes sent out over Germany with armor. The military leaders sought advice from an unlikely military adviser, statistician Abraham Wald. Together, they began examining planes that returned from flights over German territory. What they were looking at was the pattern of gunfire spray. Ra-ta-ta-ta-ta. These were the parts of the planes that got hit. When you look at the pattern, it seems reasonable to put armor where you see, and they saw the most concentrated area of gunfire hits, right? Well, that's what the American and British military officials thought, but Professor Wald offered the opposite recommendation. He said, "Instead of reinforcing where you see the gun spray, reinforce where you don't." That means the areas of the plane you see that are free from gunfire. Why? "Selection bias" he said, the gunfire, which at tens of thousands of feet was basically random, was essentially selecting out some planes to return. That is, the planes returning were the ones without mortal gunfire hits. So presumably, the planes that were hit in the spots clear of gunfire spray on the returning planes, were the ones that went down over German territory. That my friends is classic selection bias. What other examples of selection bias can you think of? Another core concept that makes evaluation challenging is confounding. In evaluation, we are trying to determine the effect of a cause, call it A, on an outcome, call it B. Say for example in the impact of a smoking cessation program on quit rates. A causal link would be, the smoking cessation program causes more smokers to quit their cigarettes, but, what if the smoking cessation program is expensive? Then maybe only wealthy individuals participate and other factors like, their doctors more frequently prescribe nicotine patches too, or the smokers can afford expensive trips to tropical islands to distract themselves from smoking, also help with quitting smoking. What I'm getting at is that the relationship here between the smoking cessation program and quit rates could be confounded by other attributes, like the wealth of individuals. On the other hand, what if the smoking cessation were offered to everyone at a workplace for free? Then we may find that only very motivated individuals get around to signing up. So sure, program participants may quit smoking more than non-enrollees, but that effect is confounded by motivation. That is, the participants were also more likely to quit independent of the smoking cessation program. Makes sense? Both selection bias and confounding are key concepts. I promise you'll see them again. Fantastic, let's stop here for today. What did we discuss? We discussed why we need evaluation, and why it's hard, including the core concepts of selection bias and confounding. Next, we're going to rewind a bit and consider what the field of evaluation is and what its objectives are. See you next time.
