System Evaluation and Discussion
6.1 Automatic License Plate Recognition .1 Evaluation Criteria
6.2.2 Groundtruth
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
50 6.2 Catch - Overspeeding Detection
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
51 Table 6.2.2.2 Comparison of groundtruth and Catch detection on the treatment group
Treatment Group Driving Speed Groundtruth Detection by Catch
Car 1 30 km/h overspeed overspeed
Car 2 40 km/h overspeed overspeed
Car 3 50 km/h overspeed overspeed
Car 4 60 km/h overspeed overspeed
Car 5 70 km/h overspeed overspeed
Car 6 80 km/h overspeed overspeed
Car 7 85 km/h overspeed overspeed
Car 8 90 km/h overspeed overspeed
Car 9 95 km/h overspeed -
Car 10 100 km/h overspeed -
Only 8 out of 10 cars were accurately detected by Catch as βoverspeedβ. For Car9 and Car10 which drove above 90km/h, since the speed was too fast, our hardware camera was not highly responsive enough to capture a clear image of the car. Hence, the license plate could not be recognized from a blurred image to register the check-in and check-out timestamp. We can say that our speed camera was only able to capture a car with maximum speed of 90km/h. This was due to the limitation of hardware setup, but it would not have too much impact on our following experiment. This was because in real world, it was difficult to speed beyond 80km/h in campus, as there may be cars coming out to the main road from every intersection.
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
52 6.2.3 Synthetic Experiment
In order to compare the performance of Catch and the traditional speed trap in detecting overspeeding cars, we conducted a synthetic experiment in UTAR Kampar campus by inviting 10 users to drive around the whole campus. The speed limit was 25km/h, meanwhile the total complete distance around the whole campus was 3.3km. Thereby, based on the time formula:
ππππππππ =ππππππππππππππππ/ππππππππππ
If drivers obeyed the speed limit, the minimum time required to complete the whole journey was 7min 55sec.
To kick start the experiment, we set up two cameras at two different locations, to serve as check_in point and check_out point, to capture the car images and record the checkpoint times.
Figure 6.2.3.1 Camera at check_in point and the captured car image sample
For benchmarking purpose, we also set up a third camera at the check_in point to simulate the traditional Malaysia speed trap system. The car image was captured from a wide angle to calculate their on-the-spot speed.
Figure 6.2.3.1 traditional speed camera at check_in point and the captured car image sample
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
53 The map below showed the locations of speed cameras set up and the route to be completed by 10 users in this synthetic experiment.
Figure 6.2.2.3 Camera installation locations and synthetic experiment route
Table 6.2.3.1 Comparison of overspeed detection by traditional speed trap and Catch Car On-The-Spot Speed Traditional
Speed Trap Time Taken Average
Speed Catch 1 35 km/h overspeed 5 min 12 sec 38 km/h overspeed 2 18 km/h normal speed 4 min 56 sec 40 km/h overspeed 3 25 km/h normal speed 6 min 27 sec 31 km/h overspeed 4 20 km/h normal speed 7 min 11 sec 28 km/h overspeed 5 30 km/h overspeed 5 min 38 sec 35 km/h overspeed 6 33 km/h overspeed 5 min 04 sec 39 km/h overspeed 7 25 km/h normal speed 4 min 28 sec 44 km/h overspeed 8 19 km/h normal speed 4 min 33 sec 44 km/h overspeed 9 40 km/h overspeed 4 min 45 sec 42 km/h overspeed 10 20 km/h normal speed 6 min 48 sec 29 km/h overspeed
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
54 Based on the comparison results above, we observed that every car which was determined as
βnormal speedβ by the traditional speed trap, their average speed was significantly higher than the standard speed limit as well as the on-the-spot speed detected by the traditional speed trap.
This indicated that the drivers were evasively slowing down when entering the camera zone, but started to speed again after exiting the camera zone. This had proven the limitations of the current speed trap system in determining overspeeding car.
On the other hand, although the overspeeding drivers followed the speed limit in the camera zone, our proposed system β Catch was still able to flag the overspeeding cars based on their average speed in completing the whole journey. In other words, the only way not to be detected as βoverspeedβ by Catch was to obey the speed limit over the distance. Which means, the proposed speed trap system was no longer escapable or evasible.
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
55 6.2.4 Real-Word Experiment
Besides from carrying out synthetic experiment, we also constructed a real-world experiment by installing cameras around the campus to compare the performance of traditional speed trap and Catch in overspeeding detection. In real world, since there will be no driver driving around the whole campus in a single trip, so for this experiment, our Catch camera checkpoints were set at the UTAR Westgate and Eastgate. Meanwhile, the traditional speed camera was set at the check_in points to detect the on-the-spot speed. The total distance of the experimental route was 1.25km, under the circumstance of the speed limit 25km/h, the minimum time taken to complete the route was exactly 3min.
The map below showed the locations of speed cameras set up and the real-world experimental route.
Figure 6.2.3.1 Camera installation locations and real-world experiment route
Noted that there were intersections between our experiment checkpoints, which means the driver may exit early without completing the route. For this concern, we had set a threshold, if the car did not show up at the check_out point after 3min 45sec passing by the check_in point, we will mark the car as βearly exitβ and discard its data in our experiment. The threshold 3min 45sec was calculated from the car default speed 20km/h without pressing accelerator.
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
56 In fact, we had also considered the βearly exitβ issue. But why didnβt we just set up checkpoints before the intersections? The main reason was that the distance between the check_in point and the first intersection was only about 0.5km, which was too short for us to detect overspeeding based on average speed. As shown in the table below, it was too strict to determine whether a car was overspeeding by only a time difference of 3 seconds. Therefore, we increased the route distance to 1.25km, so that there was still a 7-second buffer to detect overspeeding.
Table 6.2.4.1 Time difference to determine overspeed between 0.5km and 1.25km Time Taken
Route distance 0.5 km 1.25 km
Average
Speed 25 km/h (speed limit) 1 min 12 sec 3 min 00 sec 26 km/h (overspeed) 1 min 09 sec 2 min 53 sec Time difference between
normal speed and overspeed 3 second 7 second
The real-world experiment was run from 7.30am to 6.30pm, Monday to Friday, for a month.
Since the working hour was from 8am to 6pm, hence we added 30 minutes of preamble and trailer to collect the car data. Besides, as we were doing car overspeeding detection, thus the motorcycle passing by would not be considered in our experiment.
Figure 6.2.4.2 Number of overspeeding car detected by traditional speed trap and Catch (daily)
The figure above showed the number of overspeeding car detected by the traditional speed trap and Catch in a day. The total number of cars flagged as overspeed by Catch (287) in a day was 91 more than the traditional speed trap (196). Which means, within a day, there were 91 cars
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
57 speeding beyond the standard speed limit outside the camera zone, and were not detected by the traditional speed trap.
The figure below showed the number of overspeeding cars detected by traditional speed trap and Catch from the 1st week to the 4th week.
Figure 6.2.4.3 Number of overspeeding car detected by traditional speed trap and Catch (weekly)
Meanwhile, the following table summed up how many overspeeding cars did Catch caught than the traditional speed trap for every week. We can observe that although the traditional speed trap can detect a fair number of overspeeding cars (700-800), but still significantly less than what had been detected by Catch (1100-1300). In a week, Catch was able to catch around 440-490 overspeeding cars that had been escaped from the traditional speed trap.
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
58 Table 6.2.4.2
Number of overspeeding cars Catch caught more than traditional speed trap in 4 weeks Week 1 Number of overspeeding car detected
Monday Tuesday Wednesday Thursday Friday Total Traditional
Speed Trap 196 172 168 139 114 789
Catch 287 255 262 224 203 1231
Number of cars Catch caught more than Traditional Speed Trap 442
Week 2 Number of overspeeding car detected
Monday Tuesday Wednesday Thursday Friday Total Traditional
Speed Trap 204 162 147 135 106 754
Catch 279 248 253 226 195 1201
Number of cars Catch caught more than Traditional Speed Trap 447
Week 3 Number of overspeeding car detected
Monday Tuesday Wednesday Thursday Friday Total Traditional
Speed Trap 202 196 185 136 111 830
Catch 281 273 271 260 232 1317
Number of cars Catch caught more than Traditional Speed Trap 487
Week 4 Number of overspeeding car detected
Monday Tuesday Wednesday Thursday Friday Total Traditional
Speed Trap 167 173 140 118 104 702
Catch 265 249 253 224 187 1178
Number of cars Catch caught more than Traditional Speed Trap 476
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
59 Figure 6.2.4.4 Number of overspeeding car detected by traditional speed trap and Catch
in a month
Table 6.2.4.3
Total number of overspeeding cars Catch caught more than traditional speed trap in a month Month Number of overspeeding car detected
Week 1 Week 2 Week 3 Week 4 Total
Traditional
Speed Trap 789 754 830 701 3074
Catch 1231 1201 1317 1178 4927
Number of overspeeding cars Catch caught more than Traditional Speed
Trap (in a month) 1853
After running real-world experiment for a month, we found that Catch had caught a total of 4927 overspeeding cars but traditional speed trap only managed to catch 3074 cars. In other words, if we continued with the current speed trap system, in a month, there were 1853 overspeeding cars evading the speed trap by slowing down at the camera zone. But if we transformed into Catch, these 1853 cars can no longer escape anymore.
Based on our experimental results, as long as an overspeeding car can slow down to the speed limit at the camera zone in time, it was impossible to be detected by the traditional speed trap.
However, the speeding behaviour can still be detected by Catch and was no longer evasible.
The only way not to be detected as overspeed by Catch was to obey the standard speed limit along the way. To conclude, by implementing Catch, there was no way for the speeding drivers to evade the speed detection system, and this solved the limitations of current speed trap system in Malaysia.
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
60