Now we'll introduce the essential matrix as well as the related concept called a fundamental matrix. We'll talk about how to compute the essential matrix and the fundamental matrix, fun image quality itself. Return to the same street corners we have Bob and Mike forming a ray in 3D space. And we know that the two rays from Mike's point of view x2, and Bob's ray x1 can be combined into a bilinear equation. X 2 transpose E times X 1 equal to zero, and this is called the essential matrix equation. If we were just looking at the picture itself. If Mike and Bob only have a picture of themselves looking to the world and only shared a picture with themself, we can only look at pixel coordinates, not 3D dimensional coordinates. And then we know the 3 dimensional coordinates x1 is simply project into the image so the matrix modification of k. Okay reduce a 3 dimensional vector from ops view, find X1 to below X1, each is 2D image coordinates, and similarly in Mikes point of view, the camera calibration matrix K transform the 3 dimensional array into the pixel coordinate, X2 of Mike. Again the pixel coordinates is a two dimensional quantity but we will represent them here as a three dimensional homogenous coordinates for that point. Taken one step into our [INAUDIBLE] we can transform the essential matrix substitute the three dimensional vector with it's two dimensional transformed vectors through the image coordinates. We have the image coordinates X2 transposed times K inverse transposed times C times K inverse X1 = 0. And we're going to lump all the three matrix in the middle into one combined matrix called the fundamental matrix. By doing so, we take all the relationship between the two cameras, the camera calibration matrix, the rotation, the translation. Each individually they are three dimensional matrix. There are three of them. All are all going to lump together in a three by three matrix. Which we call a fundamental matrix. So we hide all the relationships that we need to know between Mike and Bob. The physical location between the two cameras. The [INAUDIBLE] matrix all into a giant three by three matrix. And with the three by three matrix we can relate image coordinates and pixels from Mike's point of view, as well as boss point of view into linear equation. And it's called a fundamental equation. X2 transpose f equal to times x1 = 0. In the following slides, we will see given two dimensional image parameters, xy of corresponding points Mike and Bob's image we can compute essential matrix and as well as the fundamental matrix each one three by three matrix. And later on we see how to derive the camera rotation translation fundamental or essential matrix. So the goal we want to do is now click on a few points in the image. Corresponding to each other and find that computed central matrix or fundamental matrix, and found we can deduce the people lines and add people. We show how to do this in next few slides. So return to the fundamental matrix equation. We have a corresponding pair of points x1, x2 Mikes pint of view vectors are written in the homogeneous coordinates vectors. Fundamental matrix f is a three by three matrix. But it has a rank two, right. Why? Because we know that the epipole times F equal zero. So given any given pair of corresponding points X1, X2, Bob and Mike's field of view. X1 na d x2 are each two dimensional measurements well represent i a homogeneous quantities through by one matrix. We have x2 transpose f times xy equals to zero. F is a three by three matrix is a combination of camera calibration matrix k. Translation t cross and rotation r. As we see rank of to f matrix is only two because f times a people equals to zero. Furthermore, f is up o a scale so since we are looking at homogeneous representation of the image points. So has nine elements. But one scale factor can be removed, so we have a total of eight degree of freedom in the matrix F. For every corresponding points in Mike's, Bob's image, we can establish one equation finite. And therefore if I click on eight different points I can arrive eight linear equations all equal to zero. And this will allow us to compute F from the image measurements itself So again take two-dimensional images of Mike and Bob. We click on corresponding points. And we continue until we achieve eight corresponding points between the two views. This a two dimensional correspondence. Next, what we need to do, is take the binary equation that we have, with f in the middle, set of linear equation, such that we can get to the generic form where we have ax equal to zero. This is done by simply writing down all the linear terms of unknown f and recollect them into a matrix form. And now we have on the left the matrix A. Products between measurements in two images U and V, XY location of appointed image. And all the entries in F, nine of them, is reshaped in vector form, KX, and we have this new square problem where EX equals zero. And recall we had seen this in the previous lectures AX=0 can be computed soon as single value decompositions. And that we are looking for the last columns of the singular vector V corresponds to the smallest single values of D. So again we constructed a matrix from the a corresponding points. One corresponding provide one row in this matrix so we have a total of eight rows and the total is nine columns. So a's sized 8 by 9. We're going to take single value decomposition of this matrix a and single values will be stored in a dynamatrix d with the smallest single value being the last elements in the d. Therefore, the solution to our lease square problem is the last column of V. We can do that in math lab conveniently. Once you obtain this vector, we re-shape it into a three by three matrix into the fundamental matrix we want. But before I continue, we just recognize the fundamental matrix F has a rank 2. When we solved the least square problem, we have never put a constraints of rank 2 into this least square problem. So the F matrix obtained thru the V square might not be rank two. So what we do is then take this fundamental matrix that we computed V square, and reply a single composition to it. And this time the purpose is doing a clean up. We want to find the closest rank two matrix F to the F computed so that you would guarantee that it has only two dimensional subspace with a single value not zero or rank two. This is done simply by taking SVD and take the last elements of D and set to zero. And reconstruct F matrix. Now, through this step, first we do SVD to compute the least square solution for F. Second, take the F matrix and the SVD of that to a cleaning up, such that we guarantee a fundamental matrix of rank two. In practice, this requires us to painfully click on eight corresponding points. If we're very precise for the eight corresponding points, the procedure before can allow to compute the fundamentals exactly. If we want to automate this process, we need to have an automatic feature, correspondence algorithm, and for automatic feature algorithms, it's not guaranteed we have exact respondents. Some will be noise, outliers, and we need to apply the [INAUDIBLE] algorithm, to reject those outliers. We now go through a set of examples. Return to the scene that we saw before, where I click eight corresponding points as we marked an image left and right. From the least square computation, we can compute F matrix. As a vector and reshape into a three by three matrix. For this particular image we obtain the value as shown here. The fundamental matrix obtained through the lee square is not guaranteed rank 3. In fact we can check if there's by calling function rank f. Again, we clean it up through taking a second SVD, and setting the singular value, the third element to zero. This give us a reconstructive fundamental matrix, which guarantees to have a rank two. And as you can see, the values are fairly close to the fundamental matrix we compute a lee square, but it has an additional property that has rank too. Recall that, given any corresponding point, the left image, we have an epipole aligned in the right image. So in this case we have a point, x1. With the following coordinate systems measured. And through the transform matrix F time X1. Well X1 is represent a homogenous coordinates will obtain a three dimensional vector. And these three dimensional vector can be thought as as if you pull a line in the right hand image. And here we took the epipolar line. And re-express it such that the line equation has the property that the first two elements sum up to the number 1, which gives us the orientation of this line, co-sin and sin. Similarly, if you click on the righthand side of the image and obtain the following corner system. We can obtain a fine mapping through F transpose x2, to align in the left image. And the line has three dimensional vector representation and is plotted show in image here. Where's that people? So recall that people of the left image is simply the null space of the fundamentals f. Now we can compute that as a three dimensional vector, homogeneous coordinates and then we nomilize the last element to be one, to plot it in the image yourself. Where is the pinpoint in the right image? That's now space of F transpose and that can compute a SVD again, and that give us three dimensional vectors. Homogeneous representation of a point in the image. Again we need to normalize the last image coordinate to be 1 and bring it back to the image xy plane. So now we have computed epipole, epipole aligns for the left and right image. How do we obtain information that we are interested in to begin with, which is rotation and translation in 3D relation to the views? We return to this in the next lecture.
