So we'll go through a simple example, where in this case we live in a two dimensional world. And the points we're looking at in 2D in x y space and a camera is a one dimensional camera. But all we care about is projection in the one dimensional axis. And this camera is shown here. There is three different views as you move in space. And the point is illustrated in the diagram here. I was shown in the boat for how the camera itself, we show the camera orientations, 30 degrees, 60 degrees, 120 degrees, its orientation estimated currently, as well as the current position of a camera sensors in the two dimensional space. What's shown in a dashed line is measured projection of this point in the image space. And due to the imprecision of the camera lens distortion, or the measurements of featured points itself, we can see this dashed line do not fully intersect at the correct two dimensional points in this space. And what we want to do is then, move the cameras as well as the point that we're currently estimating, such that these dash lines, as well as the camera projections line up in that space. So this Jacobian that have is consists of three cameras at one points in a 2D space. So it consists of in this case, three rows. One row for each of the constraints. And the constraints is for each of the cameras. So we have a total of three rows in the Jacobian. And the camera orientations and the camera centers are then laid out in the grids we show in the different columns. Camera 1 is consist of x, y positions, camera centers as well as the rotation angle, so total of three elements that we have for Camera 1. Camera 2 similarly have three constraints. Camera 3 will have three constraints. And again, Jacobian's fall in form, we see you have zeros in the off diagonal blocks, one by three each, because Camera 1's projection coordinate system is not affected by the Camera 2and 3's coordinate system. The 2D points itself has a Jacobian, throughout for each of the cameras. So that is a total of two columns, and their three rows of notes. We have again, a one dimensional cameras, and one dimensional measurements. Measurements is only in X direction of the cameras. And further, we simplified this Jacobian, the example where we just focus the camera centers and camera orientation are given and we are only trying to look at the two dimensional points, XY, in the 3-D space. Where shall we move them such that they're more aligned with the three dash rays in the space? So imagine we want to minimize this cost function for every possible locations of the 2D points in this simplified space. And imagine we have computed cost function for every possible location of this XY in space. And this is color coded in the figure here. Blue means low energy, low cost, meaning if I put a points in that location and that point back the projected image will line up with a dashed rays better. As you can see, our current estimate is not the optimal ones but has a fairly good value. And a cost function would rapidly increase as we move the point further away from this initialized point, to the left, to the right, to the bottom. Those points location, when we back projected image, were not coincide with the dash rate at all. So, this is a blue force method, if we have a blue force search mechanism checking through every points, this will be a simple problem where we can just find the minimum values, as said that today, solution to the new x points. The point is we do not want to enumerate all possible location of the points in 2D space, but we want to have efficient mechanism given where we are currently. Follow the energy function to its local minima and reach the local minima quickly. And this is done, again, through the normal equation which consists of the Jacobian matrix, transposed time to self and Jacobian matrix transposed minus the arrow function and minus direction. And recall the Jacobian Matrix in our case is two columns, one for X one for Y. And three rows, one for Camera 1, for Camera 2, and one for Camera 3. So we call it f1, f2, f3. So this is the Jacobian for our systems, three rows. Two columns, one for x, one for y. Let's look at this particular first value in the Jacobian first columns, delta f1 with respect to x. What is measuring here, shown here, is all the values of this Jacobian matrix for all possible x of x y direction in this whole entire space. So this marker in red is our current guess made of this point x y. And what's shown here is all the possible derivative functions the Jacobian for all the possible points, not at these initialized points but every other point as well. So, what is this function measuring is if I move in the space, in delta X direction, meaning that if I start off on the initialized points, I simply take a step left or right, how much is my projection in images changes? Again, our projections are one-dimensional. Simply project to the camera in a one-dimensional ray. Think about this extreme case where I simply move the horizontal direction, in this case, that is simply in the horizontal direction for a ray that's shown here on the bottom. So as you move an X, the projection into this particular camera or into this particular direction, projection of the point is actually the same. It's not changing. So as you can see, the value of a Jacobian along this ray is constant, so it's zero. That's [INAUDIBLE] ray in space. For that particular point on a ray shown here, if I simply move horizontally delta X again by the same amount, as you can see its projection into the 1D camera will change. In fact, this change is associated with this delta, theta. The view angle changed, which is back to this particular point. Meaning if I were standing at that point in this three dimensional space, I simply take a step in a horizontal direction and the camera which is orientated this particular direction, its projection will move slightly. In fact, the movements of the projection is associated with the degree of view-angle changes caused by that rate. Showing the Delta theta one. As you change the point again in the space, if I was to take the same Delta-X for this particular camera, we see that it results in a different Delta theta. Mark it delta theta two. As you move in a different points in this two dimensional space, each time making it a delta X by the same amount, will lead to a different view angle changes with respect to this camera, by different delta theta. And this delta theta change is color coded in this picture, in terms of value. Green meaning small values, red or yellow means larger values. As you can see, as we move to the left side of this camera, every time we take a delta X, this will lead to a fairly large change in terms of projection into the image. And that's meeting with with this particular Jacobian. Recall we have initialized ourself at this gray point and what we're interested really is the Jacobian at that point but not entire cause function space. We illustrate this just for illustration purpose. Similarly, we can look at a Jacobian with respect to delta Y. If I move this conjecture point in the delta Y, direction up and down, how much is projection in image 1, respect to camera 1 is changing. And we showed this again for Camera 2 for its movement XY. If I adjust this initialize point in the XY directions, how much the image coordinates in 1D will change. And this is how much is to change for the Y direction respect to Camera 3. So this Jacobian told us that if I remove a points in the search space by a little in bit in X and little in Y, how much is image projection will change. We illustrate this for all the points in the search space. So this is a combined Jacobian in a transposed form. In transverse one, we have two rows and three columns. The error function we can also visualize. The error function says that we have a projection point and we have a measurement points indicated by the dash lines, and we can take the difference between the projection point and the measurement points. Here we illustrate again the whole error function, not just for the point we are initializing, our entire solution space that we have. Red means highly positive, blue means highly negative, [INAUDIBLE] in this case is close to zero. So you want to be in the search range, we want our variable x to roughly in the zero range, in the middle. This is for each of the cameras. This is a three by one columns. Given a Jacobian and error function, we construct this Jacobian transpose J itself and obtain a two by two matrix. This 2 by 2 matrix is formed for every two dimensional point in the search space. And we visualize that as a total of four pictures. And the normal equation then is searched for the delta X that matches this equation shown here, and we visualize each of the matrix form in terms of pictures as illustrated here. We'll have the Jacobian transpose itself, give rise to a two by x two array, you have the Jacobian transposed, in this case two by three, and error function, three by one. And what we're looking for is the scale of value delta X and the scale of value of delta Y, such as the images match up, as it's shown here. And what it's just showing here, again, is imagine we have searched for every possible initialization. And therefore for every points we have a Jacobian matrix and error functions. And for those value, we need to look for delta X and delta Y. Back to our particular search point, that the points we mark in the gray [INAUDIBLE], the delta X we computed through the normal equation is shown here. Is a fact is moving a top by a positive amount, through to the left and then delta X in a minus direction. And this is adjustments given the initial points grade and given the least square solution that we are computing in terms of the normal equations. So that's if we have one points projected to three cameras. What if we have another point in a space projected to the same three cameras? The total cost function we have is that minimize the error for back projection for points I equals 1-3. And J equals 1-2 for the two points in space. The Jacobian matrix itself is further expanded. We have three rows, as we'll say with defer points respect to Camera 1,2 and 3 forming the first three rows. And, we have Point 2 projected to Camera 1 2 3, and forming constraints are Jacobian respect to those Camera 1 2 3 and the next three rows. The colors of the J matrix associate with unknown variable while searching over. And this consist of the camera translation and rotation for each of the three cameras, color coded orange and cyan. But also consists in this case two points in space that were reconstructed for 1 and Point 2, and that forms the columns color coded in blue in the matrix J. In this case, we can see that for the two points that we have, it provides a total of four Jacobians, XY for the point 1 and an XY for the Point 2. J one, row four, corresponding to the Camera 1's constraints respected to the two points. Point 1 in row one, Point 2 in row four. What's shown here the color images is the camera position of one. If I were to move the position of Camera 1, keep its orientation, how much the error function in the back projection would change? As you can see, in our case, the current estimate of Camera 1 is very good. It might not be the local minima, but is a fairly good values, and as you move further away from this cost function will increase. Well, take the Jacobians, with respect to the initialized points of Point 1 and point 2, obtained in this case, a shift of delta x for Point 1, shown here and for delta 2 shown here. Again we don't need to in this case compute the entire cost function, the entire energy function. What's shown here is energy function for a reference, to show that the gradient DR delta x that we computed in fact is diving towards the local minimum in a fairly fast movement. And this is delta X respect to the camera center of one. It's a small shift, but that small shift will further reduce the error in the projection. So back to Bob, Mike's Jacobians. For one single point in space we have Bob's Jacobian respects the camera orientation centers, Mike's Jacobian respect to camera centers and orientations as well, and a three-column vector associated with each point. If you have multiple in the space, Bob and Mike can build up the Jacobian Matrix in the fashion that we've seen in a simplified example. Again, we have Jacobians respect to our camera orientation translation, as well as for every points we add to the space, we add different columns in a Jacobian. First x y z for the first parts and then next x y z for the second parts. So as we increase the number of points, the Jacobian further, in terms of column-wise, increases its size. But because the camera projection matrix is only interact with the camera itself, we have this off diagonal elements in the Jacobian all set to zero. And this is one particular example we've seen before, where the SIFT features that we detect is marked in red and the initialized spec projections to the least square methods is shown in blue. As you can see, they're fairly close, but not exactly on top of each other. The error that we're minimizing here is using the least square methods, is minimizing the ray direction in space with respect to each other. And as such it's not exactly minimized the error in the image space. By padding up this adjustments using the bundle adjustments respect to the image pixel errors, we obtain a much tighter alignments of back projectional points with measure points. And that results in a much more precise camera orientation translation as much as was the three-dimensional estimates of points in space. This concludes the detailed description of bundle adjustments for where am I and where is in 3-D?
