This lecture we're going to show how to measure things with two views. Two views are the focus of the fourth week of our course. But today we're going to show how to measure very simple things without knowing the relative orientation between the two cameras, which is where we're going to solve for in the first week. We have already learned that if we take a picture from the ground plane, from the road, this will be a projective transformation between coordinates on the road and coordinates in your scene. Because this projective transformation is transitive, we can infer that if we have two pictures from the same plane, like the facade of this building at the University of Pennsylvania, then corresponding points on this facade underlie all projective transformations. This is very easy to see in this geometric picture where we have a point projected from a plane through the left camera, a point projected to the right camera. Since from the left to the plane is colineation, and from the plane to the right is a colineation, obviously from the left to the right camera it is a colineation. It is true only for projections of points of the plane, not for points outside the plane. Let us refer to some history. In this case, it is not a history of paintings or perspective, but the history of soccer. In 1966, in the final of the Wold Cup in Wembley Stadium between England and Germany it was two, two England, Germany and it went into extension time. And in extension time you see this goal, where it is not clear whether the ball really crossed the line or not. The referee consulted the linesman and he declared it a goal. But this remain forever a question whether this one was a real goal or not. You see in this picture very blurry the image of the goal while it is bouncing down. There was no camera exactly of this position so that we can be fair. Whether the goal was behind the goal line, or not. In 1996, 30 years later, Ian Reid and Andrew Zisserman from the University of Oxford, they wrote the paper goal-directed video metrology where they tried to find whether the goal was behind or in front of the goal line in this very controversial soccer case. So this was the footage. The footage was black and white, and you can see images from, one camera on the left column and from another viewpoint on the right column. If you look at the soccer field as a plane and you look at these two different cameras, then we have a classic case of a projected transformation or homography between the left image plane and the right image plane. Let us see if we can compute the position of the ball. It is impossible to compute the position of the ball in the real circuit field. But what we're going to do is, we take the ball and we will throw a vertical line and we'll see if we can compute the projection of this ball on the ground plane. Or the intersection of this vertical line with the soccer field. Let B be the ball in this picture, and let B be this vertical projection on the ground plane. If we find the projection of P in the left image, or in the right image, because we know where is the goal line we're going to immediately know whether the ball is behind or in front of the goal. Let us also assume that we can find the vertical vanishing points, V and V prime, to both images. How can we do that? We can do that by fitting lines on the goal posts in both views and just intersecting these lines. If the lines are parallel they might infinity, otherwise it is at some point we compute as intersection of these two lines. This way, we can take the vanishing point, and we connect it with the projection of the ball. And we get these two segments in the scene which are inside these blue rectangles. These are the projections of the vertical through the ball in the real world. Now in Reid and Zisserman they came with a brilliant idea, that we can regard this point P, the point of intersection on the circuit field, as the intersection of the two shadows of this vertical line. You see the camera senses where light source is, so the light and create two shadows of these two lights. So the question now is if we can find these two shadows in the image plane. These two shadows intersect at the point B, and we have one of the projections of these shadows, which is the vertical line in one of the images. And we only need the other projections which we denote here with a orange line. If we take the projection center, and you connect it with this orange line, you have a triangle, and this triangle intersects this image plane of this red line. If we know this red line, we know already through the vanishing point, the length of the ball. I'm going to find the projection of the line P. While it looks beautiful in this picture, we don't know this red line, the oblique red line which is the projection of the shadow of the vertical line, the shadow caused by the left camera. So these are exactly the lines. The line ls which we do not know and the line l prime which we know. However, ls is the projection of this shadow and we know the projection of this shadow in the left lane, we have image plane, ground plane, image plane. This is how we're started this lecture. But then between the two image planes there is a projection formation. And this is what we're going to compute. We'll take the lines around the small box of the goal, and from these lines we can compute the projected transformation corresponding to the soccer field. Let's call this the transformation edge. If edge is the homography between points, then it is known that the homography between lines is H inverse transposed. And this is what we're going to do. We'll take the vertical and the left demands and we'll multiply with the aids in vertical pose and we'll find this projection of the shadow. The only thing then we have to do is to intersect the vertical line with this projection, so l primed with H inverse transpose l. And voila, this is the projection of this point on the ground plane, which is the vertical from the bottom. If we know this point p, we will know whether it will be behind or in front of the goal line. So was it a goal? No, the ball actually did not cross the line in full. There was a lot more engineering behind this geometry. We didn't know the synchronization between the two video streams. So because of that time, we didn't have timestamps. So first, we have to synchronize the two cameras. Then we had to make several assumptions about the exact size of the ball. And the size of the ball indeed counts because the whole ball has to cross the goal line. And we had a lot of motion blur. Now history is repeating. And as a matter of fact, and they get England, Germany game of the World Cup 2010. Frank Lampert from England scored the goal which is obviously a goal. You have seen this picture and you have seen it from another viewpoint very soon, but the referee did not give this goal. This was a real revenge, as opposed to a decision to give to England the goal in 1966 when the goal was not a real goal. You see from this viewpoint that this is a real goal, which the referee did not see. And from this viewpoint as well. This led to the international organization of soccer, FIFA, who use computer vision to really find out whether such situations are a goal or not. There are several technologies about this. One of them is the hawk eye, used also in the challenges in tennis. And you see in the left the cameras that are installed on a stadium. And you see on the right how this is really found. The cameras are exactly synchronized and they can really find the exact position of the ball in 3D. Because the relative poses from the multiple cameras are tracked with very, very high accuracy. This is a very expensive system, and no comparison to the original result from Ian Reid and Andrew Zisserman, who really did it with just regular cameras used by the broadcasters in that historic game. This is one of the most beautiful results of projection formations, a result where we don't need any metric information, we only click in the images on the lines of the boxes, we click on the goalpost, and it is pure math which finds the position of the goal.
