In this dissertation, we proposed novel reflection removal algorithms for LS3DPCs captured by high- resolution terrestrial LiDAR scanner [31] and dynamic 3D point clouds captured by real-time LiDAR scanner [50].
In Chapter 2, we proposed a reflection removal algorithm for LS3DPCs captured by a terrestrial LiDAR scanner [31]. We investigated the characteristics of received echo pulses associated with glasses, and computed the reliability to estimate valid glass regions of dominant reflection artifacts. Then we measured a reflection symmetry score and a geometric similarity score for each point, and detected the virtual points reliably which should be removed. Experimental results demonstrated that the proposed algorithm successfully detected and removed the reflection artifacts in LS3DPCs.
In Chapter 3, we generalized our previous method [42] to remove the reflection artifacts in LS3DPCs captured with multiple glass planes. We investigated the reflection characteristics of laser pulses emitted from a terrestrial LiDAR scanner, and estimated valid glass regions located in multiple glass planes, respectively. Then, for each point, we found all the possible trajectories of reflection, and selected the optimal trajectory by computing the validity scores combining the trajectory reliability, reflection symmetry, and geometric similarity. We finally detected the virtual points associated with the optimal trajectories, which are then removed. We captured diverse LS3DPC models with various numbers of glass planes and perform intensive experiments. Experimental results demonstrated that the proposed algorithm successfully detected the virtual points associated with multiple glass planes, and removed the geometric artifacts of glass reflection in LS3DPCs. Moreover, the proposed algorithm yields a much better performance compared with the existing method qualitatively and quantitatively.
In Chapter 4, we proposed reflection artifact removal method for 3D dynamic point clouds. We first find the set of intensity peaks by collecting intensity peaks from every frames. Then, we fit multiple lines to the set of intensity peaks. To estimate glass plane, we use the characteristics that the intensity of echo pulse is maximized when the emitted laser pulse and glass plane are perpendicular. Thus, we select the optimal intensity peak from the inliers of each line and estimate candidate glass plane using the optimal intensity peak. To detect and remove virtual points, we first cluster 3D point clouds into multiple clusters using supervoxel algorithm, and detect virtual clusters by estimating all possible trajectories of laser pulse. The experimental results show that the proposed method successfully estimates glass planes from dynamic 3D point clouds and removes reflection artifacts clearly.
71
References
[1] A. Levin, A. Zomet and Y. Weiss, "Separating reflections from a single image using local features," in Proc. IEEE CVPR, 2004.
[2] A. Levin and Y. Weiss, "User assisted separation of reflections from a single image using a sparsity prior," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 9, pp. 1647-1654, 2007.
[3] Y. Li and S. Brown, "Single image layer separation using relative smoothness," in Proc. IEEE CVPR, 2014.
[4] Y. Shih, D. Krishnan, F. Durand and T. Freeman, "Reflection removal using ghosting cues," in Proc. IEEE CVPR, 2015.
[5] N. Arvanitopoulos, R. Achanta and S. Susstrunk, "Single Image Reflection Suppression," in Proc.
IEEE CVPR, 2017.
[6] Q. Fan, J. Yang, G. Hua, B. Chen and D. Wipf, "A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing," in Proc. IEEE ICCV, 2017.
[7] R. Wan, B. Shi, L.-Y. Duan, A.-H. Tan and C. Kot, "CRRN: Multi-Scale Guided Concurrent Reflection Removal Network," in Proc. IEEE CVPR, 2018.
[8] X. Zhang, R. Ng and Q. Chen, "Single Image Reflection Separation With Perceptual Losses," in Proc. IEEE CVPR, 2018.
[9] J. Yang, D. Gong, L. Liu and Q. Shi, "Seeing Deeply and Bidirectionally: A Deep Learning Approach for Single Image Reflection Removal," in Proc. ECCV, 2018.
[10] B.-J. Han and J.-Y. Sim, "Single image reflection removal using non-linearly synthesized glass images and semantic context," IEEE Access, vol. 7, pp. 170796-170806, 2019.
[11] N. Kong, Y.-W. Tai and S. Shin, "A physically-based approach to reflection separation: from physical modeling to constrained optimization," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 2, pp. 209-221, 2014.
[12] A. Agrawal, R. Raskar, K. Nayar and Y. Li, "Removing photography artifacts using gradient projection and flash-exposure sampling," ACM Trans. Graph., vol. 24, no. 3, pp. 828-835, 2005.
[13] Y. Schechner, N. Kiryati and R. Basri, "Separation of transparent layers using focus," Int. J.
Comput. Vis., vol. 39, no. 1, pp. 24-39, 2000.
[14] P. Wieschollek, O. Gallo, J. Gu and J. Kautz, "Separating Reflection and Transmission Images in the Wild," in Proc. ECCV, 2018.
72
[15] K. Gai, Z. Shi and C. Zhang, "Blind separation of superimposed moving images using image statistics," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 1, pp. 19-32, 2012.
[16] N. Sinha, J. Kopf, M. Goesele, D. Scharstein and R. Szeliski, "Image-based rendering for scenes with reflections.," ACM Trans. Graph., vol. 31, no. 4, pp. 100-1, 2012.
[17] Y. Li and S. Brown, "Exploiting reflection change for automatic reflection removal," in Proc.
IEEE ICCV, 2013.
[18] X. Guo, X. Cao and Y. Ma, "Robust separation of reflection from multiple images," in Proc. IEEE CVPR, 2014.
[19] T. Xue, M. Rubinstein, C. Liu and T. Freeman, "A computational approach for obstruction-free photography," AMC Trans. Graph., vol. 34, no. 4, pp. 79:1-79:11, 2015.
[20] B.-J. Han and J.-Y. Sim, "Reflection Removal Using Low-Rank Matrix Completion," in Proc.
IEEE CVPR, 2017.
[21] B.-J. Han and J.-Y. Sim, "Glass Reflection Removal Using Co-Saliency-Based Image Alignment and Low-Rank Matrix Completion in Gradient Domain," IEEE Trans. Image Process., vol. 27, no. 10, pp. 4873-4888, 2018.
[22] A. Nandoriya, M. Elgharib, C. Kim, M. Hefeeda and W. Matusik, "Video Reflection Removal Through Spatio-Temporal Optimization," in Proc. IEEE ICCV, 2017.
[23] A. Geiger, P. Lenz and R. Urtasun, "Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite," in Proc. IEEE CVPR, 2012.
[24] X. Chen, H. Ma, J. Wan, B. Li and T. Xia, "Multi-View 3D Object Detection Network for Autonomous Driving," in Proc. IEEE CVPR, 2017.
[25] L. Chen, J. Yang and H. Kong, "LiDAR-histogram for fast road and obstacle detection," in Proc.
IEEE ICRA, 2017.
[26] L. Caltagirone, S. Scheidegger, L. Svensson and M. Wahde, "Fast LiDAR-based road detection using fully convolutional neural networks," in Proc. IEEE IV, 2017.
[27] E. Shtrom, G. Leifman and A. Tal, "Saliency detection in large point sets," in Proc. IEEE ICCV, 2013.
[28] J.-S. Yun and J.-Y. Sim, "Supervoxel-based saliency detection for large-scale colored 3D point clouds," in Proc. IEEE ICIP, 2016.
[29] J.-K. Ahn, K.-Y. L. , J.-Y. Sim and C.-S. Kim, "Large-scale 3D point cloud compression using adaptive radial distance prediction in hybrid coordinate domains," IEEE J. Sel. Topics Signal Process., vol. 9, no. 3, pp. 422-434, 2015.
73
[30] J. Papon, A. Abramov, M. Schoeler and F. Worgotter, "Voxel cloud connectivity segmentation - Supervoxels for point clouds," in Proc. IEEE CVPR, 2013.
[31] RIEGL, "RIEGL VZ-400 3D Terrestrial Laser Scanner," [Online]. Available:
http://www.riegl.com/nc/products/terrestrial-scanning/produktdetail/product/scanner/5/.
[32] M. Bishop, "Pattern recognition and machine learning," springer, 2006, pp. 431-439.
[33] P. Dempster, M. Laird and B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. R. Stat. Soc. Series B (Methodol.), vol. 39, no. 1, pp. 1-38, 1977.
[34] A. Fischler and C. Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Commun. ACM, vol. 24, no. 6, pp.
381-395, 1981.
[35] S. Householder, "Unitary triangularization of a nonsymmetric matrix," Journal of the ACM, vol.
5, no. 4, pp. 339-342, 1958.
[36] E. Johnson, Spin-images: a representation for 3-D surface matching, Pittsburgh, PA: Carnegie Mellon University, 1997.
[37] A. Frome, D. Huber, R. Kolluri, T. Bulow and J. Malik, "Recognizing objects in range data using regional point descriptors," in Proc. ECCV, 2004.
[38] Z.-C. Marton, D. Pangercic, N. Blodow, J. Kleinehellefort and M. Beetz, "General 3D modelling of novel objects from a single view," in Proc. IEEE/RSJ IROS, 2010.
[39] B. Rusu, N. Blodow and M. Beetz, "Fast point feature histograms (FPFH) for 3D registration," in Proc. IEEE ICRA, 2009.
[40] E. Hellinger, "Neue Begrundung der Theorie quadratischer Formen von unendlichvielen Veranderlichen," Journal fur die reine und angewandte Mathematik, vol. 136, pp. 210-271, 1909.
[41] J. Besag, "On the statistical analysis of dirty pictures," J. R. Stat. Soc. Series B (Methodol.), vol.
48, no. 3, pp. 259-302, 1986.
[42] J.-S. Yun and J.-Y. Sim, "Reflection removal for large-scale 3D point cloud," in Proc. IEEE CVPR, 2018.
[43] R. Toldo and A. Fusiello, "Robust multiple structures estimation with j-linkage," in Proc. ECCV, 2008.
[44] "Amazon Lumberyard Bistro, Open Research Content Archive (ORCA)," Amazon Lumberyard, 2017. [Online]. Available: http://developer.nvidia.com/orca/amazon-lumberyard-bistro.
[45] J.-S. Yun and J.-Y. Sim, "Virtual point removal for large-scale 3D point clouds with multiple glass
74
planes," IEEE Trans. Pattern Anal. Mach. Intell. (Early Access), pp. 1-1, 2019.
[46] J.-S. Yun and J.-Y. Sim, "Cluster-wise removal of reflection artifacts in large-scale 3D point clouds using superpixel-based glass region estimation," in Proc. IEEE ICIP, 2019.
[47] P. Foster, Z. Sun, J. Park and B. Kuipers, "Visagge: Visible angle grid for glass environments," in Proc. IEEE ICRA, 2013.
[48] R. Rusu, "Semantic 3D object maps for everyday manipulation in human living environments,"
Ph.D. thesis, Computer Science department, Technische Universitaet, Muenchen, Germany, 2009.
[49] T. Shan and B. Englot, "LeGO-LOAM: Lightweight and ground-optimized LiDAR odometry and mapping on variable terrain," in Proc. IEEE/RSJ IROS, 2018.
[50] Velodyne LiDAR, "Puck LITE Lightweight Surround LiDAR sensor," [Online]. Available:
https://velodynelidar.com/products/puck-lite/.
75
Acknowledgements
Throughout the writing of this dissertation, I have received a great deal of support and assistance.
I would first like to express sincere gratitude to my advisor, Professor Jae-Young Sim. His continuous support and guidance for my Ph.D. studies, related researches and projects, helped me to have logical and broad thoughts about researches. I could not imagine having a better advisor and mentor for my Ph.D. studies. I also would like to express thank the rest of my thesis committee: Professor Seungjoon Yang, Professor Se Young Chun, Professor Seong-Jin Kim and Professor Won-Ki Jeong for their insightful comments and feedbacks about this dissertation form various perspective.
I thank my lab mates: Kyu-Yul Lee, Hyo-Gi Lee, Dae-Sik Lee, Garam Kim, Phil Joon Jung, Se-Won Jeong, Byeong-Ju Han, Eunpil Park, KuHyeun Ko and Eun Sung Jo for the sleepless nights together before due dates and for all the fun we have had in the last six years. Also, I thank my friends Sangyeong Jeong, Hanvit Kim, Hyeonuk Sim, Hyeon Joong Cho, Hoon Yi, and Taeyong Kim.
Last but not the least, I would like to thank my family: my parents Guang-Og Yun and Soon-Rye Sah and my sister Hye-In Yun for moral supporting and encouragement throughout writing this dissertation and my life in general.