Let's take a look at some ways to improve the consistency of your labels. Here's a general process you can use. If you are worried about labels being inconsistent, find a few examples and have multiple labelers label the same example. In some cases you can also have the same labeler label an example, wait a while until they have hopefully forgotten or technical term is wash out, but have them take a break and then come back and re-label it and see if they're even consistent with themselves. When you find that there's disagreements, have the people responsible for labeling, this could be the machine label engineer, it could be the subject matter expert, such as the manufacturing expert that is responsible for labeling what is a stretch and what isn't a stretch, and/or the dedicated labelers, discuss together what they think should be a more consistent definition of a label y, and try to have them reach an agreement. Ideally, also document and write down that agreement, and this definition of y can then become an updated set of labeling instructions that they can go back to label new data or to relabel old data. During this discussion, in some cases the labelers will come back and say they don't think the input x has enough information. If that's the case, consider changing the input x. For example, when we saw the pictures of phones, they were was so dark that we couldn't even tell what was going on, that was a sign that we should consider increasing the illumination, the lighting with which the pictures were taken. But of course, I know this isn't always possible, but sometimes this can be a big help. Then all this is an iterative process. So after improve x or after improving the label instructions, you will ask the team to label more data. If you think there are still disagreements, then repeat the whole process of having multiple labelers label the same example, major disagreement and so on. Let's look at some examples. One common outcome of this type of exercise is to standardize the definition of labels. Between these ways of labeling the audio clip you heard on the earlier video, perhaps the labelers will standardize on this as the convention, or maybe they'll pick a different one and that could be okay too. But at least this makes the data more consistent. Another common decision that I've seen come out of a process like this is merging classes. If in your labeling guidelines you asked labelers to label deep scratches on the surface of the phone, as well as shallow scratches on the surface of the phone, but if the definition between what constitutes a deep scratch versus a shallow scratch, barely visible here I know, is unclear, then you end up with labelers very inconsistently labeling things as deep versus shallow scratches. Sometimes the factory does really need to distinguish between deep versus shallow scratches. Sometimes factories need to do this to figure out what was the cause of the defect. But sometimes I found that you don't really need to distinguish between these two classes, and you can instead merge the two classes into a single class, say, the scratch class, and this gets rid of all of the inconsistencies with different labelers labeling the same thing deep versus shallow. Merging classes isn't always applicable, but when it is, it simplifies the task for the learning algorithm. One of the technique I've used is to create a new class, or create a new label to capture uncertainty. For example, let's say you asked labelers to label phones as defective or not based on the length of the scratch. Here's a sequence of smartphones with larger and larger scratches. Not sure if you can see these on your display, but let me just make them a little bit more visible here. I know that all of these are really large scratches if this is a real phone you're buying. This is just for illustrative purposes. Maybe everyone agrees that the giant scratch is a defect, a tiny scratch is not a defect, but they don't agree on what's in between. If it was possible to get them to agree, then that would be one way to reduce label ambiguity. But if that turns out to be difficult, then here's another option; which is to create a new class where you now have three labels. You can say, it's clearly not a defect, or clearly a defect, or just acknowledge there's some examples are ambiguous and put them in a new borderline class. If it becomes easier to come up with consistent instructions for this three class problem, because maybe some examples are genuinely borderline, then that could potentially improve labeling consistency. Let me use speech illustration to illustrate this further. Given this audio clip, [inaudible] I really can't tell what they said. [inaudible] If you were to force everyone to transcribe it, some labelers would transcribe, "Nearly go." Some maybe they'll say, "Nearest grocery," and it's very difficult to get to consistency because the audio clip is genuinely ambiguous. To improve labeling consistency, it may be better to create a new tag, the unintelligible tag, and just ask everyone to label this as nearest [inaudible] unintelligible. This can result in more consistent labels than if we were to ask everyone to guess what they heard when it really is unintelligible. Let me wrap up with some suggestions for working with small versus big datasets to improve label consistency. We've just been talking about unstructured data or problems where we can count on people to label the data. For small datasets there's usually a small number of labelers. So when you find an inconsistency, you can ask the labelers to sit down and discuss a specific image or a specific audio clip, and try to drive to an agreement. For big datasets, it would be more common to try to get to consistent definition with a small group, and then send the labeling instructions to a larger group of labelers. One other technique that is commonly used, but I think overused in my opinion, is that you can have multiple labelers label every example and then let them vote. Voting is sometimes called consensus labeling, in order to increase accuracy. I find that this type of voting mechanism technique, it can work, but it's probably over used in machine learning today. Where what I've seen a lot of teams do is have inconsistent labeling instructions, and then try to have a lot of labelers and then voting, to try to make it more consistent. But before resorting to this, which I do use, but more of a last resort, I would use the first, try to get to more consistent label definitions, to try to make the individual labelers choices less noisy in the first place, rather than take a lot of noisy data and then try to use voting to reduce the noise. I hope that the tools you just learnt for improving label consistency will help you to get better data for your machine learning task. One of the gaps I see in the machine learning world today is that there's still a lack of tools, and there are also machine learning ops tools for helping teams to carry out this type of process more consistently and repeatedly. It's not us trying to figure this out in the Jupyter Notebook, but instead to have tools help us to detect when labels are inconsistent and to help facilitate the process in improving the quality of the data. This is something I look forward to hopefully our community working on and developing. In terms of improving label quality, one of the questions that often comes up is: What is human level performance on the task? I find human level performance to be important and sometimes misused concept. Let's take a deeper look at this in the next video.
