Camera Calibration

Loading...

From the course by University of Pennsylvania

Robotics: Perception

368 ratings

University of Pennsylvania

Robotics: Perception

368 ratings

Course 4 of 6 in the Specialization Robotics

How can robots perceive the world and their own movements so that they accomplish navigation and manipulation tasks? In this module, we will study how images and videos acquired by cameras mounted on robots are transformed into representations like features and optical flow. Such 2D representations allow us then to extract 3D information about where the camera is and in which direction the robot moves. You will come to understand how grasping objects is facilitated by the computation of 3D posing of objects and navigation can be accomplished by visual odometry and landmark-based localization.

From the lesson

Geometry of Image Formation

Welcome to Robotics: Perception! We will begin this course with a tutorial on the standard camera models used in computer vision. These models allow us to understand, in a geometric fashion, how light from a scene enters a camera and projects onto a 2D image. By defining these models mathematically, we will be able understand exactly how a point in 3D corresponds to a point in the image and how an image will change as we move a camera in a 3D environment. In the later modules, we will be able to use this information to perform complex perception tasks such as reconstructing 3D scenes from video.

Introduction10:11

Camera Modeling10:47

Single View Geometry14:18

More on Perspective Projection8:09

Glimpse on Vanishing Points10:17

Perspective Projection I14:42

Perspective Projection II14:30

Point-Line Duality8:42

Rotations and Translations18:11

Pinhole Camera Model10:25

Focal Length and Dolly Zoom Effect8:58

Intrinsic Camera Parameter13:03

3D World to First Person Transformation13:41

How to Compute Intrinsics from Vanishing Points12:11

Camera Calibration11:50

Meet the Instructors

Kostas Daniilidis
Professor of Computer and Information Science
School of Engineering and Applied Science
Jianbo Shi
Professor of Computer and Information Science
School of Engineering and Applied Science

In the last lecture, we have talked about the camera projection model,

the model of perspective projection.

Today, we're going to talk about camera calibration,

how to characterize a camera, how to find the parameters of the camera.

Whether it be the focal length, the image center, the image distortion, or

even the position of the camera with respect to fixed reference frame.

Let us look again at the projection matrix.

The projection matrix is a 3 by 4 matrix P,

which if I will multiply the world coordinates,

we get lambda times uv1, where u and v is given as the pixel position.

So we have to think about this the following way.

Let's say somebody gives us the world coordinates of some point.

If P were known, we would immediately get u and v.

Let's say somebody gives us u and v.

Then, if P were known, we will not be able to immediately get

Xw Yw Zw because there is one unknown there, the lambda, the unknown depth.

But instead of this,

we're going to get a range space through the central projection and the pixel.

But in both cases, we need P, the 3 by 4 projection matrix.

Now this 3 by 4 projection matrix depends on the camera parameters,

the f, the u0 and vo, and the parameters with respect to the world,

what we call extrinsic parameters, R and t.

In addition to those parameters,

we might encounter some other parameters which cannot be modeled linearly,

not with a nice, linear algebraic equation we have seen in the last picture.

This happens when we have lenses like fisheye lenses,

lenses which have a very big field of view.

Then, you might have already observed in cameras, like the GoPro, that when we get

straight lines in the world, you don't get straight lines anymore in your picture.

You get really curved lines.

But these curved lines have a particular characteristic.

They all of them are distorted with respect to the center.

And this distortion we're going to call the radial distortion,

which means that the pixel point is distorted proportionally to

the radius from the center.

We model this radial distortion as opposed to the perspective distortion,

which is naturally perspective projection.

We model just with a polynomial, which is 1+k1

radius +k2 radius squared + k3 radius cubed, and so on.

This k1, k2, k3 are unknown parameters,

which we have to find with a procedure called calibration.

r is the radius, and

it can easily be found as the square root of u square + v square.

The calibration estimates the intrinsic parameters of a camera.

In the past when people didn't have computers, they were trying to compute

these parameters by just looking at the specification of the camera.

For example, the focal length would be given in millimeters.

One would know that the film has a dimension of 35 millimeters, then somebody

would really use this focal length exactly to find the projection in the field.

The image center you could find it with some adjusting procedure, and the k1,

k2, again, by measuring something directly on the film or on the print of the camera.

These days, having computers and having a lot of software doing geometric

transformations and optimization, we can do this on a computer.

So we never rely anymore on the readings of the camera specifications.

But we apply a procedure called calibration to estimate the focal length

f, which are going to be measured in pixels, the image center in pixels.

And k1, k2, etc, the parameters that are unitless,

which describe the radial distortion.

These intrinsic parameters are as opposed to the extrinsics

which really depend on where is the camera with respect to a fixed frame.

Let's see what is the result of such a calibration looks like.

On the left, we see a picture taken with a very wide field of view camera.

You see all the lines in the picture up here, really curved.

And on the right, you see the result after removing the radial distortion.

You see all the lines appearing straight.

And this is what we want in many applications because this radial

distortion is actually distracting us from the linearity of our equations.

It's a nonlinear effect which we try to remove first before we make any

geometric computations.

Let's see now how exactly we peform this calibration.

To perform this calibration, we need to compute the intrinsic and

extrinsic parameters given world coordinates of some points in

the world and the corresponding coordinates in images.

After we find, again just to remind you, after we find the rotation translation and

the intrinsic K, RT and K, and having removed the radial distortion.

Then we can find the projection of the rays in the world,

it can do any computation we want.

For example, compute the motion of the camera, or

compute a regulation of the point.

Let's see how we calibrate in Matlab.

We are going to use what is called the calibration toolbox.

After we have download the calibration toolbox and while in the directory where

the calibration toolbox lies, we call a GUI, a graphics user interface.

And then we have a menu where we set and select to load our images.

These images are the image of a checkerboard that can be, for example,

8, 10, 12, or 20.

And then we have saved them in a directory, and we load them all in memory.

What do we exactly want to find?

Remember that we want to find these matrix parameters and

the radial distortion parameters.

We know the world coordinates because we have measured our checkerboard,

and we know the pixel coordinates if we click on them.

Now this procedure, we saved us some clicking,

which is the selection of extracting corners.

When we're extracting corners, every image is presented by us, and

we call with a pointer and select four characteristic points.

We're going to see later in a lecture about projective transformation,

why we really select these particular four points.

The upper left point,

the first one that we click has always to be at the same position and visible.

The other points can be, for example, shorter or less deep in the image.

The images have to be taken from several viewpoints so that the checkerboard

appears quite obliquely and varies in appearance in all the ten images.

If possible, not only rolling the image around optical axis, but

also tilting it and panning it.

After we have extracted the corners of the checkerboard,

what this procedure really does internally is to compute all the pixels,

all intersections of the checkerboard in the images.

So instead of having us clicking in all 64 positions,

we clicked only 4, and does a computation of the 64 by itself.

In the next point, we click calibrate.

And what really happens at this point is really that given the world coordinates

and the pixel coordinates, the software computes the camera parameters.

You will see that these camera parameters are in a vector of the focal length,

the image center c, the radial distortion,

and then a rotation matrix in the translation.

A small detail here which we have not presented, for the focal length,

we actually see two parameters, a different focal length for x, and

a different focal length for y.

And this still stems from the time when pixels were not really square.

In the old TV, the pixels were rectangular, and

that's resulted in two different scaling factors for the x and the y.

So we still keep it, and usually these two values the way you

see it here are really almost equal up to some decimal places.

Another interesting phenomena that you're going to see is that the u v that you

are actually seeing there is not the exact center of the image.

And indeed, this might be true

that actually the optical axis does not exactly intersect the chip in the middle,

because the chip might have not been positioned exactly.

But it might be also just an effect of the optimization that happens here,

where all parameters are seated in a way that their projection error is minimized.

The projection error of these two last values in pixels you would really want it

to be very small.

And this is the optimization criterion to find these parameters,

that if we take the points in the world and we apply the estimate parameters that

we're going to fall very, very close to original pixel positions.

As a result of this calibration you can see also all the estimated poses of

the camera you have used.

You might have taken the same images by waving the calibration board,

the checkerboard in front of you, or

by moving the camera in front of a checkerboard.

But these two relative motions are exactly the same.

So what we really see in the visualization here are really

just calibration poses with respect to a fixed coordinate system.

The R and T you really find from this procedure is the pose of the camera,

the very first image where you were holding it

when you started taking the images.

So, this is what you get out of the calibration procedure, and

you save these parameters.

And you have to use them on many other problems, like the absolute pose

estimation, structure for motion, triangulation, or stereo.

So, for verification, what you can do is take all of the parameters that

you've found and press Reproject Coordinates,

which is going to show the undistorted image without the radial distortion.

And you will see how close the points in the world are projected to the original

points in the image.

You can use this also in other pictures, and

see how well this projection is close to the original projection of your images.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2018 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play