In this video, I want to talk about the concept of orthogonal projections. The idea being that we have a vector in a big vector space and we want to project that vector down into a subspace of that original vector space. So, let's start with the big vector space. Capital V is assumed to be an n-dimensional vector space, and capital W is a p-dimensional subspace of big V. So, W is the subspace of V. Using a process such as the Gram-Schmidt process, we can construct an orthonormal basis for the subspace W. It's p-dimensional so, there are p-basis vectors. Let's write s_1, s_2 through s_p as the basis vectors for W, orthonormal basis vectors. Now, what is the meaning of orthogonal projection? So, let little v then be a vector in the big vector space. Then what is the orthogonal projection of little v onto W? So, the orthogonal projection of little v onto the subspace W is given by, little v, so I'm going to say projected onto W, so I'll use this notation. S are the orthonormal basis for W, so to project v onto W, what we do is take the, so these are supposed to be column vectors but even though this works for general vectors, we're always working with column vectors here. So, we do the v transpose times S_1. This is a scalar now which is the value essentially a v along S_1, and then this times the vector S_1, so a scalar times S_1, plus v transpose S_2, times the vector S_2, plus finally the last one would be v transpose S_p times the vector S_p. So that's the definition of the orthogonal projection of v onto W. The way to think about this is that it's the p's of v along S_1, and then you multiply by S_1, plus the p's of v along S_2, you multiply by S_2, et cetera. We can be even more concrete than this. I wrote down the basis for W, we can write down a basis for V, and W is a sub set of V. So, I can extend the basis of W to be a basis for V. So we extend the basis for W to a basis for V in the following way. The basis for V, will be the original basis for W. So, s_1, s_2 through s_p, the p-dimensional vector space, plus the additional basis vectors, we need to span all of V. So, to here, we add t_1, t_2. Then the last vector t should be so that the total number of basis vectors is n, because V is an n-dimensional vector space, so this should be n minus p, okay? This is an extension of the basis for W. So, that we have a basis for V. Okay. So, let's repeat this process. So, we have little v is a vector in V. If little v is a vector in V, then we can write it as a linear combination of the basis vectors for V. So, we can write, little v, any vector in the big vector space capital V as a linear combination of this basis vectors. So, I'm going to write it as a scalar a_1 times the vector s_1, plus a scalar a_2, times the vector s_2, plus a scalar a sub p, times the vector s sub p, plus another scalar b_1 times t_1, plus another scalar b_2 times t_2, plus the last scalar, b sub n minus p, times t sub n minus p. Any vector in the big vector space V, can be written as a linear combination of these basis vectors. In this notation then, what is v projected onto W? V projected onto W then, is just the p's of this vector that lies in W. All of these terms are proportional to the t vectors outside of W. Only the S vectors lie in W, so V projected onto W is just a_1, s_1, plus a_2, s_2, plus a_p, s_p, and we throw away the t vectors. Okay? So that's the same formula here. You can see that because this v transpose S_1, then this is v right?. V transpose s_1 because the s's are an orthonormal basis vectors. So, if we multiply v transpose against s_1, then s_1 transpose s_1 gives us a_1, s_1 transpose s_2 is zero. Sorry, we looking at v transpose s_1. So, v transpose a_1, s_1 transpose s_1 is a_1, a_2, s_2 transpose s_1 is zero because s_1 and s_2 are orthogona, a_p, s_p transpose s_1 is also zero, b1 transpose s_1 is zero, all of those vectors are also zero because this is an orthonormal basis set. So, this v transpose s_1 is just a_1, so this one is the same here, v transpose s_2 is just a_2 and v transpose s_p is just a_p if this is what is v. So these are two equivalent formulations. I think the second one is easiest to understand, right?. So, the orthogonal projection of vector v in the vector space capital V is just the p's of the vector v that lies in W, right? So, the p's that's in W. So these are just the basis vectors of W. There is an important result that one can derive, not very difficult but I don't think I will do that here. This v projected onto W is the vector in W, that is closest to the original vector v. If you have a vector v right and the big vector space capital V, if you want to find a vector in W that is closest to that vector v, it's a different vector but you want to find the one that is closest to v, then you project v onto W. That's the closest vector. That will be useful when we talk about the least squares line. So, let me summarize, an orthogonal projection then is a way of projecting a vector in a bigger vector space onto a subspace of that vector space. Basically, you throw away all the piece of that vector that is not in the subspace. I am Jeff Chasnov. Thanks for watching and I'll see you in the next video.