Hey guys. Now we are going to talk about those mysterious beasts named eigenvector and eigenvalues. And just to make things easy so we don't have to write things out too much, we're going to abbreviate eigenvector with ev with a little vector sign. An eigenvalues as just ev. So, eigenvectors, eigenvalues. Okay, so what are these things? What are these mysterious beasts. Well, first of all, the eigenvalues and eigenvectors are properties of a matrix. Specifically, they're properties of a square matrix. So, an n by n matrix. And it will turn out that the eigenvalues are invariant to a change of basis. So, we saw in the change of basis video that you could pick a new coordinate frame, a new basis and represent your matrix in that basis. And different representations will give you different groups of numbers in your matrix. However, the eigenvalues of the matrix are the same regardless of what representation your matrix is in. So, that's going to be a very useful property later on. Okay, but since we're taking about matrices, let's remind ourselves of what a matrix is, or what it does. So, what does a matrix do? Let's say we have the matrix A, what does it do? Well, a matrix takes in vectors, that's v1, and maps in to other vectors. So, maybe when you multiply A by v1 you get this new vector, Av1. Maybe you have v2 over here and when you multiply A times v2, you get a vector over here. So, this is Av2. So, what the matrix A is doing to each vector is rotating it by a certain amount and then stretching or shrinking it. So, for V1, it rotates at by maybe 30 degrees and then doubles its length and something similar for V2. Maybe it rotates it by 20 degrees and then scales its length by two and a half. So, what does A do? It rotates and stretches or shrinks vectors. So, one way to talk about A is to make a big list of all the vectors you can possibly think of and then say how much each is rotated and each one stretches or shrinks when acted upon by A. And so, you can do that if you have a lot of time on your hands, but it's really not a very good idea because the list of all possible vectors is infinite. Instead what we can do is to concentrate on a very specific set of vectors or a very special set of vectors if you like. And talking about how each one of those rotates or stretches will actually completely describe what A can do to any arbitrary vector. So, let's say we have our matrix A and we're trying to figure out what the heck is going on with it. And we know that if you have just some lame vector over here, v sublime. When you multiply A times B sublime, you get another kind of, so that's A times B sublime. So, it rotates it and stretches it. Now, it just so happens that there are certain vectors in our vector space, that A will only stretch or shrink. A will not rotate them at all and so these vectors are very special. So, let's call this V special. This will be the first special vector. And what happens when we act on V1 special with A? Well, It doesn't rotate it. All it does is maybe triples its length. So, this is A times v 1 special and in this case there will actually be at least one other special vector. So, this is v2 special, and maybe when you operated on v2 special with A, what it does is to shrink the vector. So, maybe that's A times v2 special, so all the special vectors, A just shrinks or stretches. It does not rotate them, and so what we have is that for a matrix A of dimension N x N. So, remember we're dealing with square matrices now, there are at least N vectors that are only shrunken or stretched by A. So, an A acts upon these vectors, it only stretches or shrinks them, it does not rotate them and so these are special vectors and you know what? In German, the work for special is eigen, or something like that. So, these are called eigenvectors, and the specialness, well they're special for a lot of reason. But the first reason they're special is that A only stretches or shrinks them. So, let's call the set of eigenvector. E1 through en. So, may be this special one was just equal to e1 and this special two was equal to e2. So, if the matrix A only stretches or shrinks eigenvectors, we can write out A times an eigenvector. Let's start with the first eigenvector, is equal to some constant times that vector. So, the output is in the same direction as the input. And we have A times e2 is equal to maybe a different constant. Times that eigenvector. So, in our case e1 would have a landa one grater than one, because it got stretched. And e2 would have a landa two less than one, because it got shrunk. And the landas are called eigenvalues. The eigenvalue is the amoun that the eigenvector is stretched by or shrunken by, when it's multiplied by A. Okay, and now I'm not going to prove this to you but it's something you should do for yourself at some point if you are into this math stuff. But if A is symmetric, so the entries below the diagonal are reflections of the entries above the diagonal. That means that it's eigenvectors will be orthogonal. So, if A is symmetric And we drew its igenvectors, there will always be a right angle between them. And that you weigh her in 3D space, there would be a right angle between all of the eigenvectors. All of the eigenvectors would be mutually orthogonal to one another. So, what I claim is that in this case, knowing the eigenvectors and the eigenvalues tells you what A will do to any vector. So if we know all of the eigenvalues and the eigenvectors, how can we figure out what A times an arbitrary vector will be. Well, since our eigenvectors are all orthogonal and we usually take, I forgot to mention this. We usually normalize the eigenvectors so that they have length one. Since they're all orthogonal to one another and there are at least n of them, we can write v as a linear combination of them or a weighted sum. So v might be v1 times e1 + v2 times e2 plus so on and so forth plus vn times vn. And plugging that into our equation gives us Av = A(v1e1 + ...+ VN, BN). And we can use a distributive property to say that that equals V1 times Ae1 + ...+VN times AeN, where I've move to the Vs to the left of the As because they're just numbers. So we can move them around without any problem. But look at all these Ae1 and Ae2. Since there are eigenvectors, we can replace those with lambda e1 and lambda e2 and so one and so forth. To lambda N, sorry that's lambda 1e1, all the way up to lambda, NeN. And of course keeping the v prefactor out front. So, just like that, we have figured out what Av equals, just by knowing the eigenvalues and the eigenvectors. Okay, okay, so let's say that we have some symmetric matrix and it has a couple of eigenvectors. So e1 and e2. And maybe, in this basis, e1 is let's say, 3 over square root of 10, 1 over square root of 10. And in that case, e2 would be minus 1 over square root of 10 3 over square root of 10. And so that's an example of the actual numbers that you might put in your description of your eigenvector. And, so if we were to change basis, so if we had this be x one new and this be x 2 new. Then, our representations of the eigenvectors would change. And so the actual numbers that go into your eigenvectors depend on what basis you're looking at. However, in a certain way, even though the numbers changed, there's still the same eigenvectors. e1 is going to be right there no matter what my coordinate frame is and e2 is still right there no matter what my coordinate frame is. But most importantly, the action of A upon those vectors is going to be the same regardless of what basis I'm in. So as long a I write A in the correct basis, and remember I can do that by saying A in the new basis is equal to some change of basis matrix, times A in the old basis, times the inverse of that matrix. So as long as I write my A on the correct basis, it's going to do the same thing to my eigenvectors. And what is it do, well it'll scale them by their respective eigenvalues. So this leads us to the conclusion that the eigenvalues, so lambda 1 lambda 2 all the way to lambda n, are independent of the basis you're in. So the actual numbers you put down in the eigenvectors will depend on the basis, but the amount that they are stretched or scaled by, provided you represent A in the new basis, will not change. So the eigenvalues of the matrix are independent of the basis that you're in. That's pretty cool. That's kind of like saying the side links of the triangle are independent of how that triangle is rotated in space. We're finding invariant quantities of the matrix. And so, if you noticed, when we wrote out our eigenvectors in terms of the actual numbers up here that was kind of messy, in our standard x1, x2 basis. I don't like square roots and everything like that and I especially don't like negative numbers. And I especially don't like it when every element of the vector has a weird ugly number. So the last question we'll ask is can we find a basis in which the eigenvectors will be less messy? And the answer is of course, we can. So, let's clean things up just a little bit. So, we don't get confused, the point is to not be messy, right. And let's say, where were our eigenvectors there, that was e1 and e2. We had x1 and x2. So what we can do is choose the basis that lines up with our eigenvectors. So this is the eigenbasis. And in this case, what is e1 in the eigenbasis? Well that's easy, since it's only along one direction, the first direction, it will just be 1,0. And what's e2 in the eigenbasis? Well, it's only along the other dimension, so that'll just be (0,1). So this is a much cleaner representation of our eigenvectors than representing them as stuff with square roots of 10 in them. This is, in a way, the most natural representation. So how to think of this is as the fact that the eigenvectors of a matrix form the most natural representation of the matrix. And so why is it natural? Well, what this means is that all of the special vectors, so all of the vectors that just get scaled are those which lie a long the axis you're using to represent your vector space. So that's pretty neat, wouldn't you say? And here is the icing on the cake. We showed How to represent a matrix in a new basis. And so if you're representing A eigen in a new basis, that would be equal to x times the original a in your standard basis or in your original basis, times x inverse. So these are the. And so here the x's are the matrices that change basis into the eigenbasis. And what's really, really cool about this is that a in the eigenbasis, Is diagonal. So all of the entries except the diagonal entries are zero. And what do you think goes in the diagonal entries? Well, that would happen to be the eigenvalues. And everything else is zero. And so how would we actually find this change of basis matrix? Well, our new basis is just the Eigen vectors. So we would have e1 transposed, so a row. A row vector of our first Eigen vector as the first element. And e2 transposed as the second row. So, in this case, what do we say? We said that E1 is equal to what? 3 / square root of 10, 1 / square root of 10, and E2 was equal to -1 / square root of 10, 3 / square root of 10. So if this guy is x Eigen, then A Eigen will just be this times your original A, times the inverse of x Eigen. And this will be diagonal, which is super nice. So let's quickly summarize what we've come up with so far. First of all, the eigenvectors of a matrix A are the vectors that are only scaled by A, and not rotated. And we have that the scaling factors are called eigenvalues, so, ev's. And we also had that knowing the eigenvectors and the eigen values tells you how a acts on any vector because that vector because that vector can be written as a linear combination of the eigenvector. And lastly, we had that the eigenvector's formed the most natural basis for A because they caused. A and eigen basis to be diagonal. And lastly we had, this is very important, the eigenvalues were invariant to a change of basis. So you can pick any basis you want to represent a n. And your eigenvalues will always be the same. Because, remember, when we represent A in a new basis, what we're doing is finding the transformation of A that preserves its action. And its action can be defined by what it does to the eigenvectors. Namely, multiplying them by their eigenvalues, I use. So since the action of A is the same regardless of what basis it's in so will the eigen values be. So hopefully they're not too scary yet. If you want to check if something is an eigen vector that's pretty easy we go. So for example, let's say A 2, 3, 1 1. And we want to check if the vector v = (1 1) is an eigen vector of A, well what do we say? So we can write out Av = (2 1, 3 1) (1 1) and that's equal to, 5 2. And 5 2 is clearly not a multiple of 1 1. So v is not an eigenvector. I'm not going to go through all of the details about how to calculate eigenvectors. These videos are more to explain the intuition behind these concepts. But the starting point is to write out your eigenvector equation. So A times e1 = lambda1 e1. This could be rearranged such that A minus lambda times the identity times e1 = 0. And by solving this equation for lambda 1 and e1 you get your eigenvector and your eigenvalue. And in fact what you'll find is that this equation has several solutions. Specifically, it will have at least as many solutions as the dimensionality of your vector. So if A is a 2x2 matrix and it operates on two dimensional vectors you will have at least two eigenvectors. So lastly just an example of why these are useful, well here are a few examples. So here are some examples of when eigenvectors are useful So one is it can help you decouple a set of differential equations. Which is what they're used for in lecture two this week. When we're finding the eigenvalues of the recurrent connection matrix. It can also be used to decorrelate a random vector. And so this is what we did when we were doing PCA. In PCA what we had was a bunch of random vectors drawn from some distribution. And maybe these vectors were all correlated. So here the x1 component is very collated with the x2 component. When x1 goes up so does x2 when x1 goes down so does x2. And it just so happen that Eigen vectors of the co-variance matrix are these random vectors were arranged such that if these vectors were from a gausion distribution they would suddenly be decorrelated in the new bases, the eigen bases. And it's nice so to work with uncorrelated random variables when you're dealing with probabilities. That's another instance when eigenvectors are useful. Another example is in dynamical systems theory. Specifically, in a linear system The eigenvectors tell you the direction along which the motion of the system will be in a straight line. And so that could be a very useful thing to know when you're trying to figure out how a system behaves around a fixed point. So that's all I've got for eigenvectors. Again, the purpose of this was not to go through and solve a bunch of complicated mathematical problems, but rather just to give you a perspective of what eigenvectors were all about. So I hope you've enjoyed it and I recommend finding a nice sunny pategras sometime and weighing down it and staring up at the sky and really thinking about what eigenvectors and eigenvalues mean. Linear algebra is a fascinating subject. And understanding the properties of a system, or the properties of a matrix, that are invariant with respect to change of representation is really an interesting thing to meditate on. So thanks for listening. I'll see you next time.