These experiments showed that a global solution could be achieved using L∞ to estimate the relative motion of multi-camera systems. In this thesis, we investigate the relative motion estimation problem of multi-camera systems to develop linear methods and a global solution.
Problem definition
Contributions
Overview
Projection of points by a camera
It should be noted that x still uses the same unit (say "meter") as that of the world coordinate system in (2.2). The units of the focal lengths fx and fy must be converted from meters, the unit of the world coordinate system, to pixels, the unit of measurement of images.
Rigid transformation of points
Rigid transformation of cameras
In particular, note that the camera is first translated by t and then rotated by R relative to the world coordinate system. Therefore, v can be considered as an image vector coming from the center of the camera to the point X.
Epipolar geometry of two views
- Definitions of views and cameras
- History of epipolar geometry
- Interpretation of epipolar geometry
- Mathematical notation of epipolar geometry
- Pure translation (no rotation) case
- Pure rotation (no translation) case
- Euclidean motion (rotation and translation) case
- Essential matrix from two camera matrices
- Fundamental matrix
The epipolar line corresponds to the image of the ball as seen by the queen. After all, Rvis is the image vector in the first image rotated in a coordinate system of the second camera.
Estimation of essential matrix
Horn’s nonlinear 5-point method
The rotation of the coordinate system of the first camera relative to the second camera is known as the relative orientation. Given a pair of matching points and v′, Rvis the image vector in the first view rotated in the right view (or camera) coordinate system, where is the relative rotation with respect to the other view.
Normalized 8-point method
The minimal five-point solver requires five point correspondences and returns up to 10 actual solutions. There are three epipolar constraints for the minimal five-point problem: the coplanar constraint, the rank constraint, and the trace constraint.
The L ∞ method using a branch-and-bound algorithm
However, in this chapter, we give a brief introduction to the use of two-camera systems, a trifocal tensor, and three-camera systems in multiple geometry. In this chapter, we discuss the characteristics and rigid motion of two / three camera systems.
Motion estimation using stereo cameras
Three-camera systems (trinocular)
Trifocal tensor
A LineLin 3D space is projected as the linesl′and l′′in the images of the second and third camera, respectively. In the point-line-point case, consider that a pointXin can project 3D space to pointsxandx′′in the image planes of the first and third cameras, respectively.
Motion estimation using three cameras
Advantages of multi-camera systems
Multi-camera systems usually have more than three cameras and they do not need to share a field of view. The greater the number of cameras used, the greater the complexity of the multi-camera system.
Geometry of multi-camera systems
Rigid transformation of multi-camera systems
Rigid transformation of cameras was discussed in section 2.1.3, and the rigid transformation of multi-camera systems will be explained in this section. Using (4.1) and (4.2), all the cameras in the multi-camera system are moved to new positions by motionMas follows:.
Essential matrices in multi-camera systems
Non-perspective camera systems
Plane-based projective reconstruction
The four points located on the reference plane are used for matching all images. Once the four points are identified, the planar homography is estimated from the four points. Only four points on a plane visible in all images are needed to solve the projective reconstruction problem.
Linear multi-view reconstruction and camera recovery
We now know the first 3×3 half of the camera matrix and the coordinates of the pairs of matching points. The unknown parameters are the translation part of the camera matrix and the coordinates of the 3D points. Using Rother's method, the two unknowns – the 3D point and the translation part – can be calculated simultaneously.
Recovering camera motion using L ∞ minimization
Estimation of rotation
Averaging rotations
The weighted averages of more than two rotations can also be obtained using the two methods mentioned above.
Lie-algebraic averaging of motions
General imaging model
Convex optimization in multiple view geometry
Instead, it only becomes a convex function if a certain sublevel of the quasi-convex function is considered. Therefore, the strategy is to determine an a-sublevel set of the quasi-convex function to obtain a global minimum. Assume that the center of the unit sphere is the center of the omnidirectional camera.
A translation estimation method
- Bilinear relations in omnidirectional images
- Trilinear relations
- Constructing an equation
- A simple SVD-based least-square minimization
Therefore, the fundamental matrix for the two omnidirectional cameras is expressed by the 3-vector because we already know the rotations. Given point correspondences x↔x′in the two omnidirectional viewsandj, we can calculate a fundamental matrix Fji from x′⊤Fjix = 0where Fji is a skew-symmetric matrix according to lemma 1. The trilinear relations are expressed in the same way as bilinear. relations using a trifocal tensor instead of a fundamental matrix.
A constrained minimization
Algorithm
Experiments
Synthetic experiments
In Figure 6.3, we show the omnidirectional motion of the camera recovered using a basic matrix-based method for varying the noise levels in the data. In Figure 6.4, we show the omnidirectional camera motion that is recovered using constrained minimization and a basic matrix-based method for different noise levels. In Figure 6.5, we show the omnidirectional motion of the camera recovered using a trifocal tensor varying the noise in the data.
Real experiments
Notice in particular that there are no spirals or unexpected changing trajectories in Figure 6-6 as in Figure 6-5. Especially at the same noise level, Figure 6.6-(d) shows a significant improvement over Figure 6.5-(d). These images come from a camera in the forward direction of the person's movement.
Conclusion
Some approaches use stereo/multi-camera systems to estimate the ego-motion of the camera system. In this chapter, we propose an algorithm to estimate the motion of 6 degrees of freedom of multi-camera systems. The next section will present our new approach to estimate the 6 degrees of freedom of a multi-camera system.
Two camera system – Theory
Geometric interpretation
If we think of the camera moving along the line C′2(λ), then this ray traces a plane Π; 3D pointX must lie on this plane. On the other hand, pointX is also seen (as image pointx) from the start position of the second camera and therefore lies along a ray through C2. The point where this ray meets the plane must be the position of point X.
Critical configurations
Algorithm
Experiments
Synthetic data
In this section, we compare our method with Stew'enius' method in the same configuration of synthetic data. These six rays, represented by Pl¨ucker line coordinates, are used to estimate a relative motion of the generalized camera using Stew'enius' method. The comparison between our method and Stew´enius's method is shown in Figure 7.7(a) and Figure 7.7(b).
Real data
Then, a core matrix from a camera on the left side of the vehicle (a rear camera is selected in this experiment) is estimated from the five-point correspondences using the minimum five-point solution method [ 71 ]. With the essential matrix estimated, the degree of translation direction is estimated from a point selected by the other camera on the right side of the vehicle. If the ratio is equal to one, then it tells us that the scale estimate for the translation is close to the correct solution.
Conclusion
Therefore, given 17 points, it is in principle possible to construct a system of linear equations that estimate the generalized essential matrix. Nevertheless, in this chapter, we discovered, remarkably, that there is a linear approximation to solving the generalized essential matrix in the case of common multi-camera configurations. Many nonlinear algorithms exist for solving the generalized essential matrix for generalized cameras.
Generalized essential matrix for multi-camera systems
Pl¨ucker coordinates
Pl¨ucker vectors represent a straight line with a 6-vector, which is a pair of 3-vectors, q and q', which are called the direction vector and the moment vector, respectively. Pl¨ucker's coordinates are homogeneous and therefore multiplying all six coordinates by any real number gives a new one. The three-dimensional space lineLin and two vectorsqandq′ for the Pl¨ucker coordinates of the line L. The vectorq represents the direction of the line, and the vectorq' is the moment vector, represented by the shaded triangle and the normal vector to the triangle.
Pless equation
It is important to note that if the last three elements of the Pl¨ucker line are equal to zero, i.e. due to the use of Pl¨ucker lines, the generalized essential matrix can represent the ratios for a pair of matched light rays in multiple cameras. systems. The point can be determined by finding the point of intersection of the two rays.
Stew´enius’s method
The two light rays L1 and L2 should intersect at one point in the world coordinate system. For continuous motion, Pless also derived the differential generalized epipolar constraint similar to the generalized epipolar constraint for discrete motion.
Four types of generalized cameras
- The most-general case
- The locally-central case
- The axial case
- The locally-central-and-axial case
The order of incoming light rays in correspondence is not preserved and all light rays meet on axis in each camera. d) Local-central and axial case;. Once the order is preserved, we can consider another configuration, which we call the "local-central and axial case". This "local-central and axial case" of generalized cameras is a special case of the "axial case" by preserving the order of the incoming light rays.
Algorithms
Linear algorithm for generalized cameras
This specific minimization problem can be solved by using the least-squares solution of homogeneous equations subject to a constraint, as discussed in Appendix A.5.4.2 on page 595, [27]. The details of the least-squares solution to our problem are explained in the following section, Section 8.4.2. For the local-central and axial case allvandv′ must be the same point and must lie on a single line.
Alternate method improving the result of the linear algorithm
Experiments
Synthetic experiments
Real experiments
The path is printed on a piece of paper and used for the path of the LadybugTM2 camera in the experiment. With all this real data, the estimated motion of the LadybugTM2 camera and its comparison with the ground truth is shown in Figure 8.15. We showed a 3D view of the estimated motion and positions of all 6 cameras in the LadybugTM2 camera system in Figure 8.16.
Conclusion
Geometric concept
This means that finding the motion of a set of cameras is equivalent to solving the problem of triangulating the translation direction vectors derived from each view. We then use Ei, fori >1 to estimate the vector along which the final position of the first camera can be found. In short, each essential matrixEi constrains the final position of the first camera to lie along the line.
Algebraic derivations
Triangulation problem
Second-order cone programming
Summarized mathematical derivation
Algorithm
Experiments
Real data
Somewhere between frame 57 and frame 80 an error occurred indicating the calculation of the position of frame 80. In Figure 9-5-(c) we have shown the difference between the ground truth and the estimated position of the cameras in this experiment. As can be seen, the position of the cameras is estimated to within 57 frames.
Discussion
Hartley and Kahl recently showed that it is possible to find an optimal solution of the essential matrix for a single camera under L∞ using a branch-and-bound algorithm, by searching for the optimal rotation over the rotation space [21]. Hartley and Kahl performed a study to obtain a global solution for the essential matrix in terms of the geometric relations between two views [21]. Equation (10.8) uses a zero-order approximation for the rotations in the region D of the rotation space.
Branch-and-bound algorithm
Theory
This is a cross from only one camera, and the existence of the cross tells us if a problem is possible for the optimal essential matrix in one camera. We find the center of the first camera after the last move by an intersection of all wedges defined by all pairs of matched points. Solving this feasibility problem tells us the location of the center of the first at the last movement.
Algorithm
Experiments
Synthetic data experiments
The angular difference between the estimated translation direction and the true translation direction of cameras, (c) the distance between the estimated centers and the true center of cameras, and (d) the scale ratio between the estimated translation and the true translation are compared by varying noise parametersσ from 0 to 0.5 , which means that about 99.7% of the pixels have errors from 0 to ±1.5 pixels due to 3σ. As shown in Figure 10.4, several experiments are performed 10 times on the same synthetic data by increasing the noise parameters in pixels, and the distance error of centers is compared with the ground truth and its mean values are displayed. As seen in Figure 10.5, our proposed method provides a better estimate of rotation and translation than E+SOCP.
Real data experiments
- First real data set
- Second real data set
For 108 images, the motion of the 6-camera system is estimated and the results are shown and compared with the results of the "E+SOCP" method in Figure 10.14. The estimated trajectories of cameras are superimposed on the ground truth of the measured trajectories of the cameras in Figure 10.12. The estimated trajectories of the LadybugTM2 camera and its existing 6 cameras with the tag are shown in Figure 10.15.
![Figure 10.5: Comparison of two methods which are the SOCP based on the single essential matrix method by [40]](https://thumb-ap.123doks.com/thumbv2/5docco/10307099.0/172.918.182.771.265.791/figure-comparison-methods-socp-single-essential-matrix-method.webp)
Conclusion
Third, linear methods for non-overlapping multicamera setups are presented to estimate the relative motion of multicamera systems. All the six degrees of freedom of the rigid motion of multi-camera systems can be estimated. In this work, we gave a new direction to camera motion estimation for multicamera systems.
