More specifically, there are a variety of different approaches that people use to address this causality issue. Kind of arrayed them here on a chart. Because there's a real tradeoff here, I think, between difficulty and effectiveness. The things where we really think, yep. We absolutely are convinced that this is what's driving performance. Tend to be quite hard to do. There are a lot of easier things that give us more comfort, but can't be definitive. And so the simplest thing, if we're using some sort of regression framework, is this idea of measuring and controlling the omitted variables. If we believe that there's some other factor, that's driving our predictor, that's a potential omitted variable. If we can include that in our regressions and control for it, or if we can make sure that people, the high and the low performers, have same levels of that omitted variable. Then we can say, no that's not what's going on. And so that's your first strategy. Kind of a variant of that, is trying to hold the person constant, and so sometimes looking at changes in performance of the same individual before and after some kind of intervention, training or something like that, can be a way to avoid a lot of these omitted variables. It's not perfect because as we saw in the example, of kind of people improving in training because it's people are experiencing a dip who go in. There are also some things, even with the same person, but change at the same time as our intervention. But this is kind of the first thing that we do. The reason why this doesn't always work is that there are a lot of things that we can't measure with the precision that we would like to. And so a lot of the early work on this was done looking at the question of the value of education, you know? How much money do we get from each extra year of education. And people are always worried about. Well, is that driven by the effect of what we're learning, or is that driven by smarter people getting more education. Obviously, as a PhD, I'm deeply wedded to that second explanation. The question was can we control for ability? Obviously, you can throw in things like GPA and so on, but we always know GPA and grades are not the same thing as being smart. And so a big challenge was always how do we really feel comfortable that we have measured ability. Because there's always gonna be a lot of things that you haven't measured, and that's gonna compromise this approach. The second thing that you can do that's on the easier side is What's evidence that you can use to rule out alternatives? If you think there's another reason why you might be seeing the same pattern, what can you do to address this? So I talked a little earlier about some work that I did where I was trying to understand whether promotion or hiring was more likely to lead to high performance. One of the challenges with that work is I was using performance evaluations and so I find that people who are promoted perform more highly. Well, couldn't that just be that their evaluation's more biased? The man just know them better, they like them better they give them better evaluations. It's hard to rule that out. One thing that I could look at, is at least say. Well, sometimes these evaluations are more objective and sometimes they are more subjective. So, you're going to be more objective when you're thinking about evaluations that are tied to results. Did they achieve results and less objective when you're thinking about competence. They're going to be more objective when your looking at a job like sales, with this hard number at the end of the year. Less objective when you're looking at something like advisory business. And so what I did was I at least compared what the effects look like when they're more objective versus less objective. The effects were stronger on measures that were more objective, which gave me some comfort that this wasn't a bias story. So always think about, okay, what other things do those alternative explanations suggest and can I find evidence for them in the data is another way to at least get some comfort around causality. Ultimately, to really feel confident here, what you need is something that looks more like an experiment. In an experiment what we do is randomly assign people to a treatment condition and then we have controls. We compare the difference between them. One way to do this, is look for places where the experiment has already been done with yes. So people describe this as natural experiments. Weather conditions under which randomly assigned to the treatment or is there an assignment to getting treatment has nothing to do with performance, a famous example of this in a different setting. Again, this is a question people are interested in. How much does education affect earnings? Well, it turned out, during the Vietnam War in the U.S., there was conscription and there was a draft, and how likely you were to be drafted depended on a lottery number. That you've got. One way to get out of military service was through going to university. And so depending on the draft number people got, they were more versus less likely to actually go to university. And so people have just been randomly assigned a higher versus lower probability of going to university. And this was used to try and estimate, okay, if they're being randomly assigned this, then we have more confidence that we can get a sense of what the effect of going to university on pay is. Because we know there is some component of this that has nothing to do with our underlying ability, ability to learn, or anything like that. And so looking for those kinds of trials, those natural experiments where we know something is being randomly assigned, Is a great way to be more confident about causality. The big challenge here, obviously, is you need to be lucky. Not everything is randomly assigned. Chances are your training was not randomly assigned and if it isn't, you cannot use this kind of approach. In that case, what you can do is conduct an experiment. If this is something that matters enough for you then persuade people that you want to randomly assign training, and see what the impact is. And that way you really can have a decent control and treatment group. You can even make sure that those groups are really balanced in terms of some of the other characteristics of individuals that you think might effect the outcome. This is kind of the gold standards. When we do medical trials, we really wanna know the answer, it's the right thing to do. Obviously, the challenge here is it's expensive, right? You need to persuade people to let you do the experiment. It's gonna take you time to set up the experiment. It's gonna take time to run in a way that often archival data, it's all already there, okay? And so that's the tradeoff here. But always something that you kind of want to think about as you look at these studies is why am I seeing the results that I'm seeing? And if this really matters, what can I do to be absolutely sure that my predictor variable is driving this performance in the way I think it is? How much am I willing to invest in finding that out?
