In this project, a steel defect detection image recognition system has been developed to detect three types of hot-rolled steel defects: rust, edge, and loose wrap. Meanwhile, the system performs model inference by using the trained SSD model to find and crop the region of interest (ROI) from the input image regarding edge defects and loose packing.
General Introduction
Importance of the Study
Problem Statement
Aims and Objectives
Scope and Limitation of the Study
Contribution of the Study
Outline of the Report
Introduction to Image Processing
Colour Detection
RGB color model is the most widely used color model that stores values for red, green and blue layers. The HSV color model is the most commonly used for color detection, as it is more similar to human color perception.
Canny Edge Detection
As a result, edge points that do not significantly contribute to feature visibility can be discarded. Examine the edge points perpendicular to the edge and see if their gradient is less than (x,y).
Computer Vision
Object Detection
Object localization is the process of locating a detected object in an image with a bounding box as an indication. In addition to object classification, the system performs object localization to predict the height and width of the detected car in the image.
Deep Learning
Loss Function
- Classification Loss
- Binary Classification
- Regression Loss
- Mean Squared Error
- Mean Absolute Error
- Root Mean Square Error
The loss function can be treated as one of the main cores of deep learning algorithms. When it comes to deep learning, one of the most important stages is designing loss functions to solve a specific job. Using the cross-entropy function, it measures the similarity between the model prediction and the real label, which represents the actual probability distribution.
However, as the accuracy of the prediction increases, the cross-entropy value decreases. For example, for K, the binary cross-entropy formula is expressed in the form of . It focuses solely on the average number of errors, regardless of the direction in which they occur.
The root mean square error can be calculated using the square root of the LMSE. To avoid this, it is often to use the square root version of the formula. However, the disadvantage of the constructed network is that the average mean accuracy (mAP) is 1.19% lower than Faster R-CNN.
YOLO
All bounding boxes have attributes such as width, height, class (for example, car, cat, dog, etc.), and the center of the box. Since each grid cell can predict bounding boxes and their associated confidence estimates, it allows YOLO to eliminate bounding boxes that are not identical in size to the current bounding box based on the confidence score. The YOLO-tiny network is a compact or optimized version under the YOLO family, which has a shallow network layer compared to other members of the series.
The model is developed and trained using a Northeastern University (NEU) hot-rolled steel strip surface flaw dataset comprising 1800 images classified into six labels, each class including 300 images divided into six categories. Single Shot MultiBox Detector (SSD) is an object detection approach that detects multiple objects in a single shot. SSD implements several improvements to compensate for the loss of inaccuracy, including multiscale and default box features.
It allows SSD to match the accuracy of Faster R-CNN while working with lower quality images, significantly increasing system speed. For the loss function, the discrepancy between the ground truth field and the predicted boundary field is called the localization loss (𝐿𝑙𝑜𝑐). For each good match prediction, it penalizes a loss based on the confidence rating of the corresponding class in the prediction.
Summary
Introduction
Since most of the technical tasks for the system development are done during the first trimester, the second trimester will focus only on the report writing and FYP poster design work. Spyder IDE is a free integrated development environment (IDE) that allows users to develop Python algorithms such as data acquisition, data analysis, data visualization, image processing, etc. Google Colab is a web-based IDE for Python that allows anyone to create and run unlimited python code using the browser.
This is because Google Colab offers free hardware resources such as Central Processing Unit (CPU), Graphics Processing Unit (GPU), and Tensor Processing Unit (TPU) to run the Python code online. It benefits the deep learning developer to train the CNN network model, a process that requires high intensive use of GPU or TPU. The essential libraries include Tensorflow, OpenCV and labelImg, which are used throughout the development process.
Tensorflow is a free and open source artificial intelligence library that builds models using data flow graphs. It can be used to process images and videos to detect items, faces and even handwriting. Image annotation is an essential data collection process that must be done before training the deep learning object detection model.
Deep Learning Modelling
Like hot-rolled steel detection, this model can help the system determine the ROI of edge defect detection, which can be used for Canny Edge processing to evaluate its severity. However, it is a challenge to collect such amount of data from the industry in addition to the pandemic condition. It is possible to expand the amount of data by slightly modifying the original data or by creating new synthetic data from existing data using some techniques.
For example, data augmentation can be done by slightly adjusting the brightness and making another copy. It is a way of preparing the learning material so that the machine learns to recognize and detect objects. It is an open source python GUI application capable of exporting the annotation files in COCO, XML and YOLO format.
Next, it is necessary to prepare the software environment and tool to train the deep learning model with all the annotated images. Google Colab is chosen as a model training platform because it is available in the form of software and hardware. The trained model can be evaluated based on its average average precision and loss function value.
Rusting Detection
Hold/Release Decision Making for Rusting Condition
The hold/release decision action determines whether the hot rolled steel can be released or must be held. This classification can be done by calculating the percentage of rust on the hot rolled steel surface. If the calculated percentage value is above the preset threshold, the hot rolled steel is classified as the watch category.
Otherwise, the hot-rolled steel with less than 30% of the rusted surface will be released. Based on the output binary image data from the previous step, the percentage of rust on a hot-rolled steel surface can be calculated from the number of white pixels. According to the inspection operation of the hot-rolled steel company, the inspectors are required to find out all the rusted steel and classify which steel should be suspended or released based on the state of rust.
Thus, by calculating the percentage of rust, it can automate this kind of inspection process.
Edge Defects Detection
Hold/Release Decision Making for Edge Defects Condition
In the previous section, it is known that the severity of the edge defects can be inspected by referring to the quantity of the white output pixels from Canny edge detection. So the program can decide whether to keep or release the hot-rolled steel with edge defects based on the percentage of white pixels in the output image from the Canny edge detection:.
Loose Wrap Detection
It is the same process in the rust detection to extract the ROI from the image for further image processing step. In this loose wrap detection, the cut ROI is used for Canny edge detection to find out the severity of the loose wrap issue of the hot rolled steel. Loose casing matter will also form irregular edge lines on the hot-rolled steel, so the Canny edge detection can be implemented.
Similar to edge defect detection, hold/release classification can be based on the percentage of white pixels.
Summary
In this chapter, the performance result for three types of defect detection which are rust detection, edge defect detection and loose coil detection are presented and discussed. For all detection types, the end result of a process is the classification of an image into the "retain" or "release" category. In the experiment, accuracy is the amount of correctly classified images over the total input images as a percentage.
Also, the average processing time is the average duration from loading an image into the system until the system successfully classifies the image. The cause of the false detection is also investigated and discussed in this chapter.
Rusting Detection Performance
Rusting Detection Test Result
During the test, 372 images were entered into the system, which generated the results in an excel file. Based on the generated result, 228 images belong to the waiting state, while 144 images are good for release. The green layer in the output images acts as an indicator for the browning area detected by the system.
Discussion for Rusting Detection Performance
It can be observed that the background of the hot-rolled steel (metal deck roof and wall) contains some green layer regions. In other words, the color of the background is almost close to the rust color. As a result, the developed rust detection system can help the steel mill to inspect more than 72000 hot-rolled steels for rust defects in one working hour.
Edge Defects Detection
Edge Defects Detection Test Results
Discussion for Edge Defects Detection Performance
It is noticeable that the edge lines of the image on the right side are much more visible than on the left side. The image on the right is taken closer to the hot-rolled steel; so it can capture more detail than the image on the left. Hold/release classification accuracy for dented edge and bent edge decreased over time.
Furthermore, the average end time for detecting edge defects ranges from 50 ms to 74.4 ms for different types of defects. The higher the image size, the greater the amount of data pixels in an image.
Loose Wrap Detection
Loose Wrap Detection Test Result
Discussion for Loose Wrap Detection Performance
Summary
Conclusions
Recommendations for future work
