Autonomous navigation of the UAV on the course avoiding obstacles using deep neural networks. Small UAVs (Unmanned Aerial Vehicles) have been widely used in various places recently due to their light weight, small size and cheap price, and autonomously operating unmanned aerial vehicles is one of the active research areas. They can be used in various activities such as rescue, search and surveillance, monitoring and human assistance.
On the other hand, the monocular vision sensor is light, cheap, low-power, and especially can obtain abundant information about the surrounding environment, such as Recently, many studies have been conducted based on vision sensor to find and follow the path from the general paths through deep learning. For stable operations, UAVs must continue to fly by avoiding obstacles when there is an obstacle in the way or by actively judging themselves when they are off-track due to a sudden disturbance situation.
The direction of the UAV is controlled to follow the given route, while maintaining its position near the center of the route with the CNN. In addition, in order to return to the UAV's original path when it goes out of the way due to disturbance, the swing rates of the CNN are stored for a certain period of time. Through these methods, the UAV handles various situations encountered during the journey on the road, and the proposed approach is verified by simulations using ROS and Gazebo simulator and experiments in real environments.
Motivation
Research Objectives
Outline of the Thesis
Autonomous Navigation on the Trail with Monocular Vision
Obstacle Avoidance Methods with Monocular Vision
Optical Flow
Artificial Neural Network
The recent introduction of deep learning technologies into various fields has brought significant progress, with many studies of basic multi-layer perceptrons known as ANNs or modified versions of ANNs such as CNN, which is well known for its performance in image processing, and RNN (Recurrent Neural Network), which due to its repetitive structure is mainly used in the analysis of sequential data. In a typical ANN, each node in the second layer is connected to each other, each providing the output of one neuron as the input of another. In artificial neural networks, the activation of a node defines the output of that node in terms of its input or set of inputs.
In other words, the activation function is a mathematical equation that determines the output of the neural network. This function is connected to each neuron in the network and determines whether the input of each neuron should be activated based on whether it is relevant to the model's prediction. The activation function also helps normalize the output of each neuron to a range between 1 and 0 or between -1 and 1.
Convolutional Neural Network
Another feature of CNN is that the pooling operation is performed after the convolution operation. Pooling refers to the process of downsampling input data by reducing dimensions according to certain rules. Pooling is an integral part of CNN because raw input data or images contain an unnecessary amount of unnecessary data, so downsampling the data helps prevent overfitting problems.
Problem Statement
Trail Navigation Algorithm
Data Acquisition and Preprocessing
The image in Figure 11 is an example of the data set obtained from a virtual environment to determine the heading direction of UAV travel. The labels such as turn left, go straight, turn right, and non-route corresponded to the images showing right, front, left, and off-road, respectively. The image looking to the front from the left side of the cycle path, the middle of the cycle path and the right side of the cycle path have labels such as turn right, go straight and turn left.
Of these, 4000 images and labels were used for training and 1256 images and labels were used for testing. In the case of Figure 12, 3,480 images and labels from a total of 4,220 data were used for training and 740 images and labels were used for testing. The data acquisition method in the route head routing decision algorithm was collected for two cases, real environment and virtual environment.
The dataset as Figure 11 obtained in the virtual environment and dataset as Figure 12 were used to train the neural network used in the simulation in Section 5.2 and the dataset as Figure 12 and Figure 13 obtained in the real environment were used to train the neural network used in the experiment itself in section 5.3.
Architecture and Training
Trail Following Strategy
The final yaw rate is determined by comparing the ratio of the left and right turn of the neural network for head direction control, and the ratio of the left and right turn of the neural network for lateral offset.
Disturbance Recovery
Recovery Strategy
Equations (3)-(5) show how the disturbance recovery swing rate is determined. where yHD is the convolutional neural network prediction for head direction control and label. turn left t, go straight, turn right, no way}. and the moving average filter for disturbance recovery is specified as:.
Obstacle Avoidance
Optical Flow Estimation via CNN
In the case of FlowNet simple, it is a neural network that learns the optical flow and the relationship between two images together by inputting overlapping consecutive images at the same time, while in the case of FlowNet correlation, each image is separately processed and combined with important features at a later stage. Of the two types of neural networks, we used FlotNet Correlation using an empirical method with simulation.
![Figure 15: Flying chairs dataset [32]](https://thumb-ap.123doks.com/thumbv2/123dokinfo/10495596.0/28.892.177.717.136.438/figure-15-flying-chairs-dataset-32.webp)
Horizontal Avoidance Strategy
Frontal Obstacle Avoidance Strategy
Integrated Algorithms
If there are no obstacles and there is no track in the camera's view, the UAV will attempt to return to the track with jam recovery. If there are obstacles on the route or if there is a path in the camera's view, the UAV tries out the integrated algorithms that combine route navigation and obstacle avoidance with weighting factors.
Training Result
Training Result of Trail Navigation Algorithm
Training Result of FlowNet Algorithm
Simulation Setup
Two simulation maps have been constructed for track navigation and two simulation maps for obstacle avoidance. In each case, a total of five simulations were performed, track navigation analyzed the distance from the center of the track, and obstacle avoidance algorithm through FlowNet analyzed the performance of each algorithm by analyzing the distance from the obstacle. In the case of perturbation recovery, we examined the direction of progress when the perturbation in the z-axis moment of the body frame applied for 5 seconds for the three-category dataset used in existing research[11][12] and the four-category dataset that added non- the trace dataset.
We also analyzed the success rate and return time to get back on the road based on the number of moving average filters. To test the integrated algorithm, we designed four simulation maps for bicycle paths with obstacles. Two simulation maps with simplified obstacles such as rollers and white walls on the bike path, and two simulation maps with realistic obstacles such as trees, people, cars and walls.
Simulation Result
- Trail Navigation Simulation
- Obstacle Avoidance Simulation
- Disturbance Recovery Simulation
- Integrated Alogorithm Simulation
The obstacle avoidance algorithm via FlowNet was implemented for L-shaped corridors and T-shaped corridors with walls, as shown in Figures 26 and 27. Each simulation was performed five times, and Figure 28 and Table 7 show the minimum and standard deviation of the minimum distance to obstacles in each simulation. The success rates and return times for the number of moving average filters used for the perturbation recovery algorithm are shown in Table 8 .
When the average number of filters is 2, it is often impossible to get back on the road through the wrong off-road measured value. If the number is more than 3, the convergence success rate is kept at 100%, but the larger the number of filters, the longer the turnaround time. Figures 29 and 30 show the results of using the existing three-category data and the result of using the perturbation recovery through the four-category dataset for the case where the perturbation is given at the z-axis moment in the body frame for 5 seconds. to leave the lane during normal driving.
Figures 31 and 32 show the results of using the existing three-category data set and the result of using the disturbance recovery through the four-category data set in the case of pushing forward in the corner of the path. Finally, we simulated a total of four maps using algorithms that incorporate track navigation, obstacle avoidance, and disturbance recovery.
Experiment Setup
Experiment Result
For various environments from forest to downtown, roads or paths are one of the environments that have the least unspecified factors such as obstacles. The second is obstacle avoidance through predicted optical flow through the convolutional neural network. Finally, when the UAV is off the road due to disturbance, the algorithm quickly returns to the road using data from the previous time.
In future works, we will develop algorithms for experiments and apply the proposed algorithms to forest trails. The speed of the optical flow estimation algorithm is low due to the limitations of the computer board. Then, the forest path is more difficult than the normal roads, because the difference between the forest paths is more difficult than the distinction of the general roads, and there are many unstructured obstacles on the forest path.
For stable autonomous navigation on forest trails, we plan to research and develop the suggested algorithms. A reactive biomimetic navigation system using optical flow for a rotary-wing UAV in the urban environment. I was very lucky to meet these people in my life and I would like to thank them.
I would also like to thank the people who worked together in the Autonomous Systems Laboratory (ASL). Finally, I would like to thank my family for supporting me during my graduate studies.
