2.3 Local Algorithms
2.3.1 Problem with different window sizes
2.3.1.3 Cost aggregation
As explained earlier, the fixed window affects the smoothness in computed disparity values in low textured regions, and it also blurs the disparity discontinuities. So, cost aggregation is generally used to perform the smoothness operation while preserving the discontinuous regions. It aims to select a best set of pixels, and their cost values are accumulated. This step reduces the error at low textured regions and preserves the edgesi.e.,depth discontinuities. With the invent of cost aggregation methodologies, there is a significant breakthrough of the earlier proposed local stereo correspondence methods. Consequently, the performance of the local algorithms are now comparable to the global algorithms.
The concept of cost aggregation is introduced by Yoon and Kweon [10]. The weight for each of the pixels in a support region is computed. The weight is a combination of the colour and spatial distances of the neighboring pixels in that support region with respect to the center pixel. This is equivalent to the process of weight computation in bilateral filtering. The cost aggregation method proposed in [10] is very similar to the filtering of the disparity space images (DSI) with joint bilateral filter [83]. Mathematically, this can be expressed as follows:
Cagg(p, d) = P
q∈Np
P
q′∈Np′
wl(p,q)wr(p′,q′)C(q, d) P
q∈Np
P
q′∈Np′
wl(p,q)wr(p′,q′) (2.22)
where Cagg is the aggregated cost, p′ and q′ are the candidate matching pixels in the target image corresponding to the pixelsp and qin the reference image, wl denotes the weight between the pixels p and qof a support window in the left image, and wr denotes the weight between the pixelsp′ and
q′ of a support window in the right image. The weights are calculated as follows:
w(p,q) = exp
− ∆cpq
γc +∆gpq
γp
(2.23) where ∆gpqand ∆cpqrepresents the spatial distance and colour difference between the pixelspandqin the CIELab colour space respectively. Also,γcandγp are two constant parameters. The disadvantage of this method is that the computational complexity increases quadratically with the increase of window size. Additionally, this method requires a large window size to handle non-textured regions.
An integral histogram is used as an approximation of the joint bilateral filter [84–86]. The bilateral filter weight is a combination of geometric and range weights. Geometric weight is based on the spatial distance between the pixels, while range weight is based on the intensity difference. The bilateral filter depends on the histogram of the difference image which is independent of the window size. In this case, it is assumed that the geometric weights are same for all the pixels. Richardtet al. presented a stereo matching algorithm which includes the reformulation of the adaptive support weights algorithm [87].
In this method, filtering is done by employing bilateral grid. To preserve the edges in both the stereo images, bilateral grid is extended to include both the input images while performing cost aggregation.
The poor performance of this method at the object boundaries is due to the bilateral grid which is designed using only gray scale images. Hence, colour having similar gray values are difficult to differentiate. Including colour information in the grid increases the memory requirement, and hence only two colour channels are involved. This is still slower, and produces inferior results compared to the gray scale approach. Mattoccia et al. proposed a cost aggregation algorithm which is also based on the joint bilateral filter [88]. The support weights are calculated according to a spatial and a range filter. Spatial weights are calculated based on the spatial distance between the center and neighboring pixels. The range weights are calculated by dividing the support window into blocks. Each block is assigned a single weight based on the colour distance between the center pixel and the mean value of all the pixels in the block. The abovementioned methods sacrifice quality in order to achieve higher computational speed.
Inspired by the performance of bilateral filter for cost aggregation, Hosni et al. performed cost aggregation using a guided filter (GF) [11]. In this method, weights of the support window are computed as follows:
w(p,q) = 1
|Np|2
P
q∈Np
h
1 + Ip−µpT
(Σp+εU)−1 Iq−µpi
(2.24) where µp and Σp are the mean vector and the covariance matrix of all the pixels in the windowNp,
2.3 Local Algorithms
U is a 3×3 identity matrix. |Np|is the number of pixels in the window Np, and εis a user-defined smoothness parameter. Yang et al. accomplished cost volume filtering by employing a full image- based guided filter [89, 90]. The support weights are calculated by a weight propagation scheme. This scheme uses four-connected grid based on the guided image. The weight propagation starts from the source pixel, and travels in the horizontal direction towards the target pixel. This propagation is continued until the propagation path encounters the column where the target pixel is located.
Then the path is propagated in the vertical direction until it reaches the target pixel. Inspired by the abovementioned method, Huang et al. proposed a new eight-connected weight propagation scheme [91]. The propagation path starts from the source pixel either in horizontal or vertical direction until it reaches the pixel which is diagonal to the target pixel. Then the path is proceeded in the diagonal direction until it reaches the target pixel. The computational complexity of the guided filter depends on the size of the image and the number of disparity values used.
Gerrits and Bekaert proposed a colour segmentation-based cost aggregation method [92]. This method is based on the assumption that the depth discontinuities coincide with the object boundaries.
In this method, the reference image is segmented using a mean-shift algorithm, and the support weights are computed on the basis of the pixel locations. Pixels which belong to the same segment as that of the center pixel are assigned one as a weight, while pixels lying outside the segment are given a weight value of zero. Tombari et al. also performs cost aggregation based on image segmentation. But, both the stereo images are considered for computing the support weights [93]. In this method, information of a pixel connectivity and shape of a segment is considered by using both the stereo images rather than using the information of colour and spatial distances. For this, pixels which occupy the same segment as that of the center pixel are assigned a weight value of one, while pixels lying outside the segment are assigned dynamic weights based on the colour distance. In [94], an efficient cost aggregation method using a colour segmentation method is presented. This method consists of two terms, one for segmentation, and another term is for weight correction. Segmentation term ensures that the shape of the support window is adapted according to the image local characteristics. The correction term adds additional weights to those pixels which are spatially near to the pixel under consideration, but it does not satisfy the segmentation assumption. Segmentation assumption tells that the abrupt change in the disparity values generally occurs at the object boundaries. Muninder et al. proposed pixel-based disparity map computation for the pixels of a segmented image region [95]. A set of plane label is generated with the help of initial disparity map and the segmented image regions. The cost for assigning a plane to each of the pixels is computed. Then this cost is aggregated by a spanning tree-based approach, and a label of a plane is assigned to each of the pixels. Segmentation-based
methods fail when the image segments overlap at the disparity discontinuities.