3.5 Experimental evaluation
3.5.1 Experiment with simulated slip models
To evaluate the performance of the proposed algorithm for learning from automatic mechanical supervision we perform an experiment in which we use actual visual patches collected by the rover (as in Figure 2.4), but the slip behavior models are created from known nonlinear models (Figure 3.7). This experiment is partially con- trolled, to be able to report classification error, comparing to human-labeled terrains, and to measure goodness-of-fit to the mechanical behavior models. The models used are generated to simulate actual ‘slip vs. slope’ behavior, as measured and reported in [81] for MER. In the next section we will show experiments on slip measurements collected by the rover in the field.
The image patches are collected from three actual terrains, i.e., while driving on sand, soil and gravel. The visual representation used is two dimensional and is composed of the average normalized red and green of the terrain patch.
The experimental setup (Figure 3.7) generally follows Figure 3.2: a set of slip and slope measurements are generated from each of the curves and are paired with appearance patches coming from actual terrains. It is not known to the algorithm which terrain classes the input examples belong to.
Table 3.1 gives a summary of the results when learning with and without me-
Norm. red
Norm. green
Input vision data Sand
Soil Gravel
20 40 60 80 100 120
−20
−10 0 10 20 30
Norm. red (rescaled)
Norm. green (rescaled)
Input vision data
SandSoil Gravel
0 10 20 30
0 20 40 60 80
100 Slip space data
Slope (deg)
Slip (%)
SandSoil Gravel
Figure 3.7: Experimental setup for learning with simulated slip models. Top row:
Example terrain patches displayed in the initial vision space with ground truth clas- sification (only a subset of the patches is shown). Bottom row: 2D color representation of the terrain patches (left) and the assigned nonlinear slip models (right).
0 10 20 30 0
50 100
Slope (deg)
Slip (%)
Abs=10.6 % Sand
0 10 20 30
0 50 100
Slope (deg) Abs=6.24 %
Gravel
0 10 20 30
0 50 100
Slope (deg) Abs=0.76 %
Soil
0 10 20 30
0 50 100
Slope (deg)
Slip (%)
Abs=0.88 % Gravel
0 10 20 30
0 50 100
Slope (deg) Abs=1.34 %
Sand
0 10 20 30
0 50 100
Slope (deg) Abs=0.722 %
Soil
Figure 3.8: The learned nonlinear models for the three classes superimposed on the training data. Learning without supervision (top), learning with automatic mechan- ical supervision (bottom). The slip models could not be estimated correctly in the unsupervised case because of initial errors in the terrain classification. They are learned correctly after adding the supervision.
chanical supervision, comparing to human-labeled ground truth. The slip prediction error is computed as: Err=PNi=1|F(xi,yi)−zi|/N, where F(xi,yi) is the predicted and zi is the target slip for a test example (xi,yi) (see also Equation (2.6)). The results are averaged over 100 independent runs. Each run uses about 400 training and 400 test examples, randomly selected from the data. In the case of learning with- out supervision, essentially an unsupervised clustering is done in the visual space.
The mechanical models are fit after the classification has converged and the data corresponding to each terrain type is used.
The learned nonlinear models for the three terrains are shown in Figure 3.8. When learning without supervision, some initial classification errors in the vision space cause examples from the wrong models to be assigned to a class and, as a result of that, the wrong slip models are estimated. This is not the case if automatic supervision is used during training. Using slip measurements as supervision helps the classification algorithm and the right models can be estimated, leading to a smaller test error (Table 3.1). Learning with automatic supervision achieves 72% of the possible margin
40 60 80 100
−20
−10 0 10 20 30 40
Norm. red (rescaled)
Norm. green (rescaled)
Unsupervised learning
Sand Gravel Soil
40 60 80 100
−30
−20
−10 0 10 20 30 40
Norm. red (rescaled)
Norm. green (rescaled)
Automatic supervision
Gravel SandSoil
Soil Soil Soil Soil
Soil Soil Soil Soil
Soil Soil Soil Soil
Soil Soil Soil Soil
Soil Soil Soil Soil
Soil Soil Soil Soil
Soil Soil Soil Soil
Soil Soil Soil Soil
Figure 3.9: Terrain classification in the vision space after learning without supervision (left column), and after learning with automatic supervision (right column). Some example patches, representatives of the learned soil class corresponding to the two learning scenarios, are shown in the bottom row. When learning with automatic supervision, the algorithm has managed to learn that the two separate clusters in the soil class, containing brighter and darker patches, belong to the same class (bottom right panel).
for improvement between the unsupervised learning and the learning with human supervision.
Figure 3.9 shows the classification in the vision space and some of the appearance patches which have been learned to belong to one of the classes. As seen in the figure, when learning with automatic supervision, the vision-based classifier has been forced to learn that some additional darker patches also belong to the soil class, which was not immediately apparent in the unsupervised classification. This point is important, as real-life data offers a lot of variability in appearance, and even if some limited supervision is admissible, a human operator would not be able to show to the system all possible illumination or view invariances of a terrain patch, for example. The
mechanical supervision can be used to do that instead. As only visual information is used for determining the terrain type in the test mode, the terrain classification errors for all three scenarios (Table 3.1) are much larger compared to the training mode (Figure 3.8). This is due to a significant overlap in the vision space. The terrain classification errors are rather large because the data is quite hard and the 2D color feature space is rather insufficient to make a good discrimination.
From this experiment we can conclude that when there is a large overlap between the visual appearance of the patches, using automatic supervision helps to learn to discriminate them better. As a result of that, more precise slip models can be also learned for each terrain. Learning from automatic supervision outperforms the unsu- pervised learning and is able to retrieve the correct underlying terrain classification achieving performance comparable to human-supervised learning. The slip models used here have very small noise variability. In the next section we perform experi- ments with actual slip measurements collected by the rover.