Commission V, WG V/4
KEY WORDS: high resolution orthoimages, Hough algorithm, automatic image orientation, SIFT, orthorectification
ABSTRACT:
At present, digital documentation recorded in the form of raster or vector files is the obligatory way of inventorying historical objects. The orthoimage is a cartometric form of photographic presentation of information in the two-dimensional reference system. The paper will discuss the issue of automation of the orthoimage generation basing on the TLS data and digital images. At present attempts are made to apply modern technologies not only for the needs of surveys, but also during the data processing. This paper will present attempts aiming at utilisation of appropriate algorithms and the author’s application for automatic generation of the projection plane, for the needs of acquisition of intensity orthoimages from the TLS data. Such planes are defined manually in the majority of popular TLS data processing applications. A separate issue related to the RGB image generation is the orientation of digital images in relation to scans. It is important, in particular in such cases when scans and photographs are not taken simultaneously. This paper will present experiments concerning the utilisation of the SIFT algorithm for automatic matching of intensity orthoimages of the intensity and digital (RGB) photographs. Satisfactory results of the process of automation, as well as in relation to the quality of resulting orthoimages have been obtained.
1. INTRODUCTION
Within the community of architects a tendency may be observed aiming at substituting the conventional vector drawing with raster products. The architects appreciate the value of orthoimages and they consider these products as the valuable alternative. It is an attractive tool for development of archaeological and architectural documentation, as the combination of the geometric accuracy and the visual quality. In the process of generation of the orthoimages it is important to determined two basic issues: the accuracy and resolution of data. The resolution should be considered as the possibility to discriminate details. It is defined by means of the (object) field pixel size of the resulting orthoimage. The accurate modelling of the surface is the key issue in the process of the orthoimage generation. It directly influences their quality, both, in relation their geometry and their visual correctness. The locally imprecise description of highly complicated surfaces leads to geometric inaccuracies of „stretched.” or „melted.” type of some parts of the orthoimage. Currently, when the digital object surface model is generated, data from the terrestrial laser scanning is used, which is not always sufficient for the complete description of surfaces of architectural objects. Combination of the point clouds obtained from the laser scanner (as the digital object surface model) and digital images seems to be more useful for the purposes of orthoimage generation. However, the automation of data processing during generation of photogrammetric products is not a separate issue.
2. METHODOLOGY OF ORTHOIMAGES GENERATION
According to the theoretical approach an orthoimage is an image, which projection plane is parallel to the reference plane and all rays are perpendicular to those two planes. In the course of orthoimages generation for the needs of the cultural heritage objects many issues appear, which result from the
nature of close-range measurements, and, in particular, from short distances from objects and their complicated geometric characteristics. (Mavromati et al., 2002, 2003). Difficulties related to generation of orthoimages were widely discussed in the context of "true ortho" generation for architectural objects, when orthoimages, as well as other digital projections are the basic tools for documenting the cultural heritage objects (Gianniou et al., 2007; Georgopoulos et al., 2004,2005). The advantages of TLS is the possibility to support the generation of correct orthoimages. It allows to eliminates many problems, which existed in relation to utilisation of digital images only. The great number of points, the high sampling frequency results in the sufficient density of spatial information concerning the geometry of measured objects. Many professional publications exist which prove the contribution of the TLS data and point clouds from dense image matching to architectural objects, improving, at the same time, automatic generation of orthoimages.
A separate issue concerns integration of the TLS data with "arbitrary" photographs. Procedures are required which would allow for combination of the TLS point clouds with "arbitrary photographs", in order to create a high-resolution orthoimage. Achievement of the sufficient geometric accuracy and the visual quality requires sophisticated abilities at the surface modelling stages and generation of orthoprojections.
Fig. 1 A sketch of the ruins of the castle in Iłża (source: A) Own source, B) Regional Museum in Iłża)
orthoimages acquisition of the cultural heritage objects and automation of "arbitrary photographs" orientation in relation to the scans.
3. RUINS OF THE CASTLE IN IŁŻA
The objective of performed experiments was to inventory and generate architectural documentation of the ruins of the castle in Iłża. The ruins of the Kraków Bishops’ Castle in Iłża is the medieval fortification and, at the same time, the relics of the renaissance residential object. At present, the conservator takes permanent care of this object. It consists of two basic parts: the upper and the lower castle. The following elements, which remained from the past and which are currently exposed on the terrain surface, are: the final defence tower, constructed on the circular plan, a fragment of the upper castle, which has been partially reconstructed, and relics of the lower castle (Fig.2). They are mostly the walls which do not have much architectural adornments. They mostly consist of stones, which create outlines of walls of particular chambers and the cellar.
All above mentioned elements were the subjects of laser measurements (close-range scanner Z+F 5006), photogrammetric measurements (images was taken parallel to walls), photogrammetric documentation and the initial analyses, performed on the basis of acquired data. Due to the existing (conventional) documentation of the object and the current conditions of the ruins, which are permanently accompanied by processes of destruction resulting from the lack of protection against atmospheric conditions or against microorganisms, attempts were made to develop the accurate digital documentation together with the additional analysis of the monuments. These operations aimed at the additional diagnosis of the current conditions of the ruins (Markiewicz et al, 2014a).
4. DESCRIPTION OF PERFORMED EXPERIMENTS
The discussed experimental works were performed according to the diagram presented in Fig.2. specified, which allow for its spatial, 3D orientation. In the case of processing orthoimages of architectural objects, this requires intensive works in order to determine many planes. The authors
Fig. 2 Diagram of the performed experiment
Fig. 3. Planes detected by the PCL algorithm ( planes are marked by the same colour)
Since vertical and horizontal planes are important for documentary purposes, the author's algorithm was developed, which utilises the modified Hough transformation for these purposes. In order to limit the time of searching for vertical planes, the projection onto the XY plane is used (Fig. 4a). The histogram is considered as an image according to the brightness scale; it serves as an image in which straight lines are determined using the Hugh algorithm (Fig. 4c). Their equation coefficients allow for the initial determination of the plane equations.
The algorithm has been developed in two versions: - for "non-ordered" points,
- for "ordered" images, which operates with .zfs files obtained from the Z+F scanner.
The effects of processing are practically identical; they only differ with respect to the time of operations (ordered images require shorter computation time: 2 minutes comparing to 6 minutes).
Possible horizontal planes are detected basing on the analysis of the histogram of points distribution at various altitudes – peaks on the diagram show the possible elevations (Fig.5). The results of searching are presented in Fig.6 (in order to clarify the figures, two of three determined vertical planes and one of three horizontal planes are presented)
Fig. 5. The histogram of the number of points as the Z co-ordinate function
Fig. 6. Planes detected by own algorithm (vertical in red and orange, blue – for the vertical plane
It was assumed during calculations that the maximum distance of points to the theoretical plane equals to 50 mm.
Considering that the wall is constructed of irregular bricks, the distribution of deviations does not correspond to Gaussian distribution (Fig. 7).
Fig. 8. Another wall of the "test" cellar
Figure 8 presents the results of searching for planes for another part of the cellar, as seen from two different points of view. In Appendix A was presented results for another architectural object (the building of the Warsaw University of Technology); in this case the wall is not only constructed of smooth bricks, but it also have many parallel planes.
4.2 Automatic generation of orthoimages in the intensity
The process of generation of orthoimages was automatically performed using the LupoScan software. The plane, which was interpolated before and the spatially oriented point cloud were used to perform this process. The TLS data of the ancient cellar of the Castle ruins in Iłża has been utilised; the scanning resolution equalled to 3 mm per 10 m. The following parameters were assumed for the generation of orthoimages:
• NN interpolation
• automatically generated plane with the assumed deviation of 5 cm
• the buffer of projected points +- 3 * 5 cm
• the resolution of orthoimages for the generated object (the ancient cellar) – 2 mm
As a result, orthoimages were obtained which were used for further experiments concerning the automation of orientation of close-range digital photographs (Fig. 9). Besides information concerning the laser beam reflectance intensity, information concerning the depth was also recorded for each pixel.
4.3 Orientation of photographs in relation to orthoimages
The objective of our experiments was to investigate the possibilities to apply "arbitrary" terrestrial photographs, in the process of orthorectification or colouring the point clouds. Although scanners are equipped with embedded digital cameras, which acquire photographs during scanning, they are characterised by the lower resolution and geometric accuracy (Fig. 10).
Fig. 9 Orthoimages in the intensity - GSD 2 mm
Fig. 10 Fragments of RGB orthoimages a) from a camera embedded in the Z+F 5006h scanner b) from the Canon 5D
Mark II camera (source: Markiewicz et al., 2014) The process of orientation of photographs in relation to orthoimages was performed in two stages:
A) Detection of tie points with the use of the SIFT algorithm (the authors' application).
The SIFT (Lowe, 2004) algorithm was applied for searching for and matching the tie points. This algorithm was implemented in the OpenCV function library, with default parameters. In the proposed approach, the plain information concerning images, recorded in raster forms, was utilised. Therefore it was possible to obtain positions of pixels in images. Generated orthoimages were recorded with georeference data and, additionally, they contained information about the depth, what allowed not only to determine co-ordinates of terrain points in the orthoimage plane, but also to add the third dimension.
Fig. 11 Utilised photographs with points used for orientation (in green) and points deleted from the process of adjustment (in red)
Fig. 12 Histograms of control points deviations (Image 1): A) X direction in pixels (blue), B) Y direction in pixels (green), C) X direction in μm (blue), D) Y direction in μm (green) - resection
Two experiments were performed:
I: for the first test site (Fig. 11 b) 8 photographs were acquired from different positions, with the use of a zoom lens, with three different focal lengths (each photograph covered the entire orthoimage area). II: for the second test site a lens with a constant focal
length was applied (4 photographs covered the entire orthoimage area).
It was assumed for the approach, which was based on the adjustment using the resection, that the interior orientation elements, which were determined earlier, are determined in an independent calibration process. In the proposed approach we do not have a network of images or the possibility to perform self-calibration, but we acquire only one image, which covers the entire object; we use the camera, which was earlier calibrated, to acquire this photograph.
In the second approach, which is based on the DLT method, all parameters are determined in the process of adjustment and the impacts of distortion elements is considered in coefficients of the polynomial (Chen et al, 1994, Karara et al, 1971).
Variant I
8 photographs were used for the analysis and testing the assumptions. Results of searching for the number of points are presented in Table 1.
Table 1. Statistic of detected points Photograph
no.
Number of detected
points
Number of incorrectly detected points
Percentage of incorrectly detected points
1 1152 4 0.35 %
2 933 3 0.32%
3 1002 35 3.49%
4 477 47 9.58%
5 300 10 3.33%
6 310 30 9.67%
7 530 9 1.70%
8 517 13 2.51%
Fig. 13 Utilised photographs with points used for orientation (in green) and points deleted from the process of adjustment (in red)
Fig. 14 Histograms of control points deviations (Image 1): A) X direction in pixels (blue), B) Y direction in pixels (green) - 3D DLT Due to the high number of tie points it was decided to present
the results in the form of histograms of deviations, with respect to the x and y axes, respectively. Figure 12 presents examples of histogram for the photograph 1. All analyses were performed in own application made in Matlab software.
After analysing the histograms it could be noticed that their shapes corresponded to Gaussian distribution. This may suggest the correct detection of tie points. It may be noticed that about 95% of points are characterised by the accuracy higher than 2 pixels (12 μm).
Variant II
4 photographs were used for the analysis and testing the assumptions. Figure 13 presents examples of the images with marked points used for orientation . Results of searching for the number of points are presented in Table 2.
Table 2. Statistic of detected points Photograph
Figure 14 presents examples of histogram for the photograph 4. After analysing the histograms it could be noticed that their shapes are similar to Gaussian distribution. This may suggest
the correct detection of tie points. It may be noticed that about 95% of points are characterised by the accuracy higher than 3 pixels (18 μm).
4.3 Generation of RGB orthoimages
The process of generation of colour orthoimages was performed automatically, using the LupoScan software, basing on the Digital Model of Object Surfaces, performed earlier and on oriented digital images.
In order to perform the accuracy analysis of the generated RGB orthoimage, it was compared with the intensity orthoimage. The SURF algorithm was used for that purpose, in order to perform independent check (Fig. 15). Obtained results were compared with the DLT photographs and the resection (Table 3).
After analysing the table it could be noticed that that about 95% of points are characterised by the accuracy higher than 2 pixels (4 μm).
Table 3. Average deviation between check points on intensity orthoimages and RGB orthoimages
Id Avarage > 2 pix
Fig. 15 RGB orthoimages with marked check points (green) used for accuracy analysis (Varian I)
Fig. 16 RGB orthoimages with marked overlaps (Variant II) Figure 15 presents final result of the single image
orthorectification. The final orthomosaic result is shown in figure 16. The red squares show the places where two different images overlap. After analysis of this fragments it can be noticed that there are no parallaxes but only difference in colours.
5. CONCLUSIONS
Performed experiments proved that the process of generation of orthoimages may be automated, from generation of a reference plane to automatic generation of RGB orthoimages.
The proposed, original approach, based on the transformed Hough algorithm, allows for high improvements of the process of searching for planes within entire scans, without the need to divide them into fragments. Thus, the divided scan with automatically generated planes, also specifies the number of orthoimages, and the maximum deviations from the plane define intervals (in front of and behind the plane) of points which are required for generation of orthoimages. This new approach accelerates the process of orthoimage
generation and eliminates noises resulting during the stage of data acquisition using close-range scanners.
Utilisation of orthoimages in the form of raster data, amended with the third dimension, allows for the use of CV algorithms to search for homologous points and spatial orientation of any photographs. Such an approach eliminates the necessity to measure the control points. The proposed approach allows for detecting the great number of evenly distributed points, what allows for detection of self calibration; this is particularly important for zoom lens cameras.
Drap P., Sgrenzaroli M., Canciani M., Cannata G., Seinturier J., 2003. Laser Scanning and Close Range Photogrammetry: Towards a Single Measuring Tool Dedicated to Architecture and Archaeology; CIPA XIXth International Symposium; Antalya, Turkey.
Georgopoulos A., tsakiri M., Ioannidis C., Kakli A., 2004, Large Scale orthophotography Using DTM from Terrestrial Laser Scanning. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Istambul, Turkey, Vol. XXXV, Part B5, pp. 467-472.
Georgopoulos, A., Makris, G.N., Dermentzopoulos, A., 2005. An alternative method for large scale orthophoto production, Proceedings of CIPA 2005 XX International Symposium, 26 September – 01 October, 2005, Torino, Italy.
Gianniou P., Georgopoulos A., Tsakiri M., della K., 2007 The documentation of the medieval entrance of the Rhodes fortification complex, XXI International CIPA Symposium 1-6 October 2007, Athens Greece
Lowe D. G. Distinctive Image Features from Scale_Invariant Keypoints, 2004.
Krakow bishops' castle in Iłża, Poland. In: GeoConference on Informatics, Geoinformatics and Remote Sensing, Albema, Bulgary, Vol. 3, pp. 391-396
Mavromati, D,. Petsa, E., Karras, G., 2002. Theoretical and practical aspects of archaeological orthoimaging The International Archives of the Photogrammetry, Remote Sensing and Spatial InformationSciences, 34(B5), pp. 413-418.
Mavromati D., Petsa E., Karras G. E., 2003. Experiences in Photogrammetric Archaeological Recording. XIX International CIPA Symposium, Antalya, Turkey, pp. 666- 669
Petsa E., Grammatikopoulos L., Kalisperakis I., Karras G., Pagounis V., 2007 Laser Scanning And Automatic Multi-Image Texturing Of Surface Projections XXI InternationalCIPA Symposium, 01-06 October 2007, Athens, Greece
Fig. 1. Results of searching for planes for a building.
Fig. 2. Interim stages of determination of planes for the second object: (a) – the 2D histogram, (b) the 1D histogram – visible peaks correspond to the road and pavement surfaces
In this case the wall is "more regular"; it is constructed of identical bricks. Therefore the majority of points fall within the interval +-15 mm from the theoretical value (Fig.8)
