To describe interactions between random variables, we have to introduce a new notion, the notion of joint probability distribution. To do so let me begin with this example. Let us consider a random experiment that consists of tossing of fair coin. Let us consider a three random variables associated with these experiment. The first random variable X will be equal to one if the first tossing gives us a head and zero otherwise. I will introduce a new notation for such kind of events. I will write the condition that we check in square brackets. So the value of this expression, the value of this square brackets I is equal to one if this condition is satisfied and it is equal to zero if it is not satisfied. So this expression is just a shortcut for this expression. Now let us consider another variable Y. Y equal to 1 minus X and let us consider a new variable Z. Z is the same as X, but it is applied not to the first tossing, but to the second tossing. So Z is second tossing gives head. Let us find the probability distribution of all these three variables. We can write it like this, the coin is fair so the probability of getting head and tail in every tossing is equal to one-half. So when the variable X takes value one with probability one-half and to value zero with probability also one-half. The same holds for probability variable Y and the same holds for Z. Now if we look at these three probability distributions, we don't see any difference between these three random variables. But if we think about pairs of these random variables, we see the difference. For example, what is the probability that X equals to zero and at the same time Y equals to zero? This probability is equal to zero because if X equal to zero then Y is equal to 1, so this probability is equal to zero. Let us ask the same question for pair X and Z. What is the probability that X equal to zero and at the same time Z equal to zero? These events mean that on first tossing we have tail and on the second tossing we have also tail. So the probability of this event is equal to probability of elementary outcome tail, tail and this probability is equal to one over four. So we see that similar questions that are asked to the pair of X and Y and the pair of X and Z gives different answers. This show us that despite the similarity between their probability distributions, these variables are in some sense distinct. To capture this distinction, we have to introduce the notion of joint probability. Now let us introduce the definition of joint probability distribution. Let us consider two random variables X and Y, let us assume that X takes value X_1 and so on X_n. We will write it in the following way. Support of X is equal to X_1, X_2 and so on X_m. Support of X is just a set of all distinct values that X can take. In the same way let us assume that support of Y is equal to values y_1, y_2 and so on y_n. Then for each pair x_i and y_j we can ask, what is the probability that X is equal to xy and Y is equal to y_j? So what is the probability that X equal to x_i and at the same time Y is equal to y_j? We can denote this probability by P_ij. Here i changes from one to m and j changes from one to n. Now the sequence of these numbers P_ij such that all of them are greater than or equal to zero and the sum of all of them is equal to one. This numbers together with this supports gives us the probability distribution which is called joint distribution of two variables X and Y. Let us consider the example that we discussed before, let us find joint distribution of variables X and Y. It is convenient to draw joint distribution as a tables like this, here we have values that can X take, this is zero and one and here are values that Y can take, here is zero and one and in the cells of this table we will write these values P_ij. So for example here we should write the probability that X equal to zero and Y equal to zero, what is this probability? According to the thing that we discussed already this probability is equal to zero. What about this probability? The probability that X equal to zero and at the same time Y is equal to one. We can rewrite it as probability that X equal to zero and we know that Y is equal to one minus x. It means that if X equal to zero then Y equals to one. So Y equals to one is the same thing as X equals to zero. So we see that in this intersection we have the same event and this intersection is just equal to this event. So this probability is just probability of X equal to zero and this is equal to one-half. In the same way we can show that here the probability is one-half and here the probability is zero. Now let us find joint probability distribution for a pair of X and Z. What should we place into this cell? As we discussed before, the event that X equal to zero and Z equal to zero corresponds to exactly one outcome of our experiment and the corresponding probability is equal to one over four. It is easy to see that all other cells will be filled in the same way. For example, consider this cell, what is the probability that at the same time X is equal to one and Z is equal to zero? X is equal to one means that at the first tossing we have head and Z equal to zero means that at the second tossing we have tail, so this probability is exactly equal to probability of exactly one outcome and this outcome is head, tail. This probability is equal to one over four, just like this one. So we have here one over four,we have here one over four and we have here one over four. Now when we look at the joint probability we see the distinction between Y and Z, when we relate it to variable X we see that they have different joint probabilities.
