About the Camera Projection Model

This post is intended as a summary of my learning in the image processing using an RGB camera. My current workplace uses RGB cameras quite a lot in their engineering research. The brand I'm using is the Intel RealSense camera for the purpose of capturing human movement. The computation is done solely in Python using the "OpenCV" library.  Although image processing is a huge topic in itself, I'm interested merely in the calibration of the three-dimensional (3D) world system into the 2D camera image using OpenCV. Only one camera is assumed in this discussion.

1. Basic definitions
The way our human eyes see near things as bigger compared to those far away is called perspective projection. Whereas transformation is the transfer (or mapping!) of an object from one reference frame to another. When talking about space, we always talk about 3D space. On the other hand, an image produced by the camera is 2D. So overall, the perspective transformation deals with the conversion of the 3D world system into a 2D image.

In working with space, we need a so-called frame of reference. The 3D coordinate system has a 3D frame of reference. The basic principle of how human vision works is the same as how any camera works. Here we will use a pinhole camera concept, the simplest form of a camera. Refer to the figure. In the pinhole camera, the frame of reference commonly has an origin at the hole itself, which defines the center of projection of the camera coordinate system. In this way, the position of the pinhole camera in the camera coordinate system is at the origin. The 2D plane on which an image is clearly formed at the back of the camera is called the image plane. The plane is at a distance defined by the focal length from the projection center.

Fig-1: A simple representation of a pinhole camera.

2. The pinhole camera model

The pinhole camera model defines the perspective transformation from the camera coordinate system to an image coordinate system. 

The center of projection C is called the camera center or the optical center. The line connecting the camera center perpendicular to the image plane is called the principal axis or optical axis in Physics. The point where the principal axis passes through the image plane is called the principal point p. We assume that p is the origin of the image coordinate system on the image plane. An object can change its size on the image plane depending on whether it is near or far from the camera along the principal axis.

Fig-2: The pinhole camera model defines the mapping between the 3D coordinates into 2D coordinates. Look at the right panel. Suppose M is at (X, Y, Z) and m is at (x, y). Point C serves as the origin of the coordinate system. The proportionate triangle rule says that y / Y = f / Z; hence y = f .Y / Z. The same rule applies to x.



Let the center of projection C be the origin of the camera coordinate system, and the image plane is f away from point C. The image plane has been brought forward, noting that the position behind or in front of the camera center is congruent. Let M be a point in a 3D space in the camera coordinate system. We want to map this into a point m in the image plane. Using the proportionate triangle rule, the projection mapping from 3D space to 2D image coordinates is:

The above transformation above does not seem linear because of the division by Z. Let's introduce a new coordinate called the homogenous coordinate system. In this new coordinate, a new dimension is added in the coordinate such that any point (a, b) becomes (a, b, 1). Here the extra dimension of a point scales the existing dimensions. The above equation can then be expressed in the homogeneous coordinates format as a matrix multiplication:

3. The pixel coordinate system

 

Fig-3: Schematic shows forward projection from the real world coordinate to the pixel coordinate system.

The previous formula assumes that the origins of the camera and image coordinate systems are exactly along Z. In reality, this is not true (Fig-3). This is because an image is captured by a sensitive semiconductor device which has its own coordinate system. Although both the image and pixel coordinate systems are 2D, they are not exactly identical.

Suppose the principal point p has shifted, then the above formula can generalize into x = f.X/Z + px;  y = f.Y/Z + py.

Notice that the term focal length has changed. The f_x , f_y represent the focal length of the camera in terms of pixel dimensions. The pixel coordinate system is what the computer system hooked up with the camera will understand.

4. Representing the world system
In the equations previously, the subscript cam is written to denote that the points are represented in the camera's frame of reference. In general, a point in the 3D has its own frame of reference in the world coordinate system. For example, you can tilt either the camera in a certain direction, or the object, or both at the same time. We need another transformation matrix that maps the world in the camera reference frame. The transformation usually involves rotation and translation (like the affine transformation in the functional MRI data).

Recall that point M in Fig-2 is stated in the camera coordinate system. However, its position is not the position in the actual world coordinate system. Look at the figure below. The camera and world coordinate systems are related through a transformation matrix [ R | t ] = .

Fig-4: Representation of the transformation between the 3D world coordinate and 3D camera coordinate systems. The transformation matrix consists of R and t, where R denotes a 3×3 rotation matrix, and t is 3×1 translation matrix. The two matrices together defines the extrinsic parameters of a camera in the external world.



Accordingly, we can compute X_cam = R_x.U + t_x  ; Y_cam = R_y.V + t_y ; and  Z_cam = R_z.W + t_z .

Another way to look at the [ R | t ] matrix is the following. The R matrix represents 3 unit vectors (in each axis) of the camera coordinates in the world reference frame. The t, on the other hand, represents the origin of the camera coordinate system in the world reference frame.

Altogether, this leads us to the final transformation matrix. Suppose point M is located at (U, V, W) in the world coordinate system and you want to map this to a point m at a location (x, y) in the image coordinate system.

The above equation can be written as m = K [ R | t ] M. Here, let us define two more crucial terminologies:

1. Intrinsic parameter K, shows the internal orientation of a particular camera. It is fixed and unique for the camera, and is independent of the scene in the real world as long as the focal lengths do not change. 
2. Extrinsic parameters, R and t, show camera orientation or position to a world coordinate system. It translates coordinates of a point in the real world (world coordinate) to a coordinate system that is fixed with respect to the camera (camera reference).

This is the final equation for the forward projection. An inverse process, backward projection, retrieve the world coordinate system from the image position.

Camera calibration can be performed easily with the OpenCV package using a piece of checkerboard to obtain the transformation matrix. Often, this calibration also includes lens distortion correction inherent in any optical system.

Source:
1. HedVision Github
2. Calibration and 3D reconstruction