Teachable Machine: Real-Time Attendance of Students Based on Open Source System
Edwin Ariesto Umbu Malahina1,*, Ryan Peterzon Hadjon2, Franki Yusuf Bisilisin1
1 Informatics Engineering, STIKOM Uyelindo Kupang, Kupang City, Indonesia
2 Information Technology, Citra Bangsa University, Kupang City, Indonesia Email: 1,*edwinariesto@gmail.com, 2ryanphd2@gmail.com, 3aenk.funk@gmail.com
Coressponding Author: edwinariesto@gmail.com
Submitted: 03/10/2022; Accepted: 30/11/2022; Published: 30/11/2022
Abstract−The utilization of open source-based services will be very useful, simplifying and accelerating the process of object recognition and complex computational processes, one of them uses the Teachable Machine service. Identification of student faces in real-time attendance is a case study that will be applied to students to recognize and identify accurately and clearly the presence of students during online / offline lectures, by applying Teachable Machine services that have good algorithms with a machine learning approach that utilizes the Tensorflow.js library where the training data testing uses Convolutional Neural Network (CNN). Of the objects identified, the average accuracy of all classes ranged from 91-100%, with the number of samples for each object class being 23 objects or more. Number of sample images in one class. Clothing, object background and lighting intensity around the image object are also very influential in determining the accuracy value of student face recognition later, so that the use of the tensorflow.js library that implements Convolutional Neural Network (CNN) will be very helpful in facial recognition and influencing factors so that the data entered later needs to be further corrected and improved again, so that the results obtained in implementing the online attendance system have been very helpful in detecting student faces with an average accuracy rate of 91.8%.
Keywords: Teachable Machine; Face Detect; Machine Learning; Recognition; Open Source, Tensorflow.Js, Convolutional Neural Network.
1. INTRODUCTION
Object-based open source is a software by utilizing open source code with the aim that certain developers or communities can contribute in adding functions, features and fix bugs in them. Teachable Machine is a web-based tool that provides three (3) object recognition learning models; image, voice and pose by utilizing machine learning technology by utilizing TensorFlow as a library in javascript., that can make machine learning model building fast, easy, and accessible to everyone [1]. The following case study will be tested and implemented in the introduction of student faces to conduct class attendance (offline) or online to be identified in the system which can later be implemented in a website development by applying integration/synchronization sources with Teachable Machine services in it. So that the application of the teachable machine system can also be utilized by students, teachers, lecturers, designers and one of them can be applied to educational institutions so that it can be used as a curriculum creation, tutorials and other resources about machine learning without special technical expertise in recognizing machine learning concepts [2] where the implementation of teachable machine services also provides high levels of accuracy, precision and sensitivity ranging from 97% - 100%, where the teachable machine service developed by Google is capable of making machine learning up to 100% test accuracy level, and can be reduced according to the factors that affect it, namely the lighting around the image object [3].
Open source-based object recognition using teachable machines has been widely researched for various case studies including; image type recognition of plant seeds [4], face object recognition where the accuracy success rate can reach 99.8% by utilizing the teachable machine [5], Object pattern recognition for micro and nano particle types to classify based on their characteristics [6], Machine learning interfaces that are also open source-based can contribute to being implemented for children with special needs, especially the visually impaired, which can be specially designed so that they can communicate based on audio or images as input data that can increase representation to communicate with each other and learn together [7]. The benefits of this open source-based system will contribute to the trend model as the case study used, which will increase the publication of open source-based research available in the public domain, where the availability of more modern computing capabilities can improve the accuracy performance of the GPU (Graphics Process Unit) system which has played an important role, one of which is in the ability to recognize faces or image objects [8].
Not only that, in the concept of machine learning based on facial recognition, users can identify facial change behavior (facial expressions) through an OpenFace service developed based on open source implemented in Computer Vision. [9]
to detect facial expressions and facial behaviors [10], because human facial expressions are a way of conveying nonverbal communication that can be understood and understood when expressing visually.
From the explanation of previous research based on open source services above which includes research on object recognition of plant seeds, recognition of facial objects (not attendance) using teachable machines, recognition of nanoparticles, application to blind people based on audio or image recognition, recognition of facial expressions using the OpenFace service. So that the research being developed tries to contribute to an online attendance system that will detect student faces in real-time using an open source system service on the Teachable Machine system.
In general, the face recognition process will go through the stages of face detection, face alignment, face representation, face identification and verification [8]. In the case study that will be examined and has been implemented, namely detecting the face of each student who is present either offline in class or online, This concept was implemented
due to problems such as; Students who attend sometimes fill in attendance to friends who are not present in class and teachers do not often check student attendance in class, so they do not have valid information in the future to make assessments based on student attendance. So from the problems found and the benefits of the current open source system (teachable machine), it is necessary to develop a system that can detect the face of each student who is present so that it is easy to be recorded in the database system. Which will require data collection in the form of images of each student's face (more than 10 images with different facial poses) then will be entered into classes on the teachable machine service for later training sample image data collected based on the available classification list then will produce output in the form of face recognition percentage value.
The implementation of student face recognition services will be developed using a website and service integration using an open source machine learning system from teachable machine (tensorflow.js - javascript) to be synchronized on the website developed and used by each individual student to take attendance using a face in real-time and data will be stored based on the percentage level of face recognition into the database system. The implementation of this system certainly contributes to recording the authenticity of each student's attendance in class, so that the teacher is able to assess and evaluate student attendance in class better through the system developed. Teachable machine also provides other export models that can be developed in the form of; javascript, p5.js, hard, Android and Coral into different device system models.
2. RESEARCH METHODOLOGY
2.1 Research Stages
The following will explain the stages of the research that will be carried out, to explain the flow of the facial recognition system framework by utilizing the open source teachable machine service, as well as the literature used in supporting the research theory.
Figure 1. Teachable Machine user interface
IMAGE CLASSIFICATION
(DATASET) TRAINING CLASSIFIER PROCESS EVALUATION
DETECTION
IMAGE SAMPLE EXPORT MODEL
(.JS)
Figure 2. Image recognition stage (Teachable Machine)
In the process of image recognition, the author uses the source data process from the Google Teachable Machine (GTM) service. Teachable Machine (teachablemachine.withgoogle.com) is a web-based GUI tool for creating machine learning classification models that can be customized without specialized technical skills. (Machine Learning, or ML, allows a system to learn to analyze data without being explicitly programmed.). Teachable Machine to help students, teachers, designers and others learn about ML by creating and using their own classification models [11].
The first process, researchers must provide the samples needed, namely in the form of sample images of student faces to become material for detection classification which will be imported into the teachable machine service first. After that, the classification process, training and output results will be tested. Then, the general steps in creating an ML classification model go through the process; classification, training, and evaluation (detection process). Then this process must be organized from left to right to show the cause and effect relationship and the sequence of steps that must be passed and performed for new users. To build a model, the user first chooses whether the model should classify images, sounds, or poses as input. Then the user creates the classes that they want the model to learn to classify [12]. In this study using image classification of images taken from a webcam, each student will enter their image in each of the available classes in the teachable machine first. Classes can be made more than one class for many students for each of their faces to be trained and detected later.
Teachable Machine uses Tensorflow.js, a library for machine learning in javascript, to train and run the training results in a model in a web browser. [13]. In this training process, machine learning systems such as Convolutional Neural Networks (CNNs) are also used in Google Teachable Machine (GTM) [12]. After training, the next step is to evaluate (detection), here you can do the testing stage using sample images or through a webcam and the system will display the results in the form of accuracy in percentage form. In addition, it can export further development files by developers in the form of javascript, p5.js, keras, Android and Coral files to different device system models to be developed into different programming languages.
2.2 Tensorflow Architechtur (Transfer learning)
TensorFlow.js is an open source machine learning library that runs anywhere JavaScript can be run. It is based on the original TensorFlow library written in Python and aims to reinvent the developer experience with a set of utilization APIs for the JavaScript ecosystem. [14]
Transfer learning involves learning knowledge to help learn something different but similar. Similarly, if you have a machine learning model that has been trained and has a domain, such as images, it can be reused to perform other tasks.
The traditional Convolutional Neural Network (CNN) architecture can be seen how transfer learning is able to utilize this trained network to learn new objects.
Figure 3. Convolutional Neural Network (CNN) Architecture
Figure 3 above shows the architecture of a CNN being trained to recognize handwritten digits from 0-9. If it can separate the pre-trained lower-level layers of the existing trained model shown on the left from the classification layer near the head of the model shown on the right, it can use those lower-level layers to generate output features for a particular image based on the original data it was trained on.[15]
2.3 Research Flowchart System
A program flowchart acts like a reflection of a computer program in terms of flowchart symbols. This type of flowchart shows the process steps of a program in detail and the steps of solving problem units together that are closely related to each other and interact with each other to achieve a goal. This flowchart shows the flow of the system or what the system does [16].
START LOGIN SUCCESS? FACE DETECTION
ATTENDANCE WEBS
Teachable Machine (Tensorflow.js)
RESULT PROCESS SAVE
DATABASE END
YES
NO
SYNC
Figure 4. Flowchart system online attendance
The initial process begins when the student logs in to verify the user's identity by entering the username and password, if successful, the meal will enter the attendance page with the face detection method in real time by utilizing the tensorflow.js library that was previously created on the Teachable Machine service in the form of an export *.js (javascript) file, if the results of the face detection percentage have appeared then the student can save the data to store student attendance data information (id, name, attendance time, face detection percentage, location and other information) into the system database, then the attendance process system is complete.
3. RESULT AND DISCUSSION
In the process of developing the system into a website, the first step that will be taken is to enter information data in the form of photos / images of student faces in the amount of variations between 22-25 each class (more is better) into the teachable machine service to train sample data first which will be managed and recognized later using Convolutional Neural Network (CNN).
Figure 5. Input data sample and testing (simulation)
In Figure 5 above, researchers conducted a sample data test of 10 classes with different student faces with gender between men and women. Then after the face sample data is taken, the next step is to train the data using the default settings including Epoch: 50 the higher the epoch value, the better the sample data will be tested 50 times, Batch Size: 16 batches are a set of samples used in one training literacy. Suppose in the case study used the sample data is 25 images / class (preferably 80 photos / images of each class) and select Batch size is 16 then it will be divided by 25/16 = 1.4 batches. After all 1.4 batches are fed through the model, exactly 1.4 epochs will be completed (16 is the ideal number so it is recommended not to change it - default) and Learning Rate: 0.0001 (default). Then the results will be visible on the top right to do testing first before being exported to a Javascript file that will be used in the website that will be developed to detect student faces.
(a) (b)
Figure 6. (a) Accuracy per class (b) Confusion matrix
In figure 6, after doing the test training data sample of all classes, it will produce the accuracy value per class and the Confusion matrix where this matrix will divide each class sample into each matrix column. So far, the matrix has placed each photo sample or class data in each matrix column correctly without any errors or confusion. In figure 6 (b) where the Y axis is the axis of the class created (class photo / student face sample), while the X-axis is the prediction that represents the classes to shoot the sample model if it matches the class name on the Y-axis then it is correct for the prediction level. Then the Y and X axes of the matrix columns must match and align to be correct.
(a) (b) Figure 7. (a) Accuracy per epoch (b) Loss per epoch
In figure 7, part (a) is the level of Accuracy per epoch obtained by the model correctly during the sample data training session, for example if the model classifies 70 samples correctly out of 100 then the accuracy is 70/100 = 0.7, if the prediction is accurate the value is one (1) but if it does not eat the value is below one (<1). Then from the results of testing research tests that have been carried out with each of the 10 classes, the accuracy value is 1. Then in the figure part (b) is Loss per epoch (error), where if the prediction value is perfect (accurate) then the value will be 0, otherwise if the prediction is inaccurate then the value is > 0. So from the results of testing research tests that have been carried out on the 10 overall classes used, the Loss per epoch value is 0.5 - 0 in this study.
After all the data in the class has been trained, the next step is to export the data. In this study the author will take the source data in the form of a javascript file (*.js) where in the source file the teachable machine service has prepared a sourcode that will be connected to machine learning from tensorflow using the Convolutional Neural Network (CNN) method approach. Where the file can be run on our localhost computer or cloud-based hosting later.
Figure 8. Export model and download javascript.js file on teachable machine
In figure 8, is a script view of the javascript language that has been included with the tensorflow.js library, so that developers do not need to do coding so long and just understand the core functions in the script file (javascript.js) that is downloaded.
(a) (b)
Figure 9. (a) Login system (b) Attendance face detection online with tenserflow.js (Teachable Machine) In figure 9 in figure (a) the first start, students will log in and verify their identity based on their username and password and access rights as (student or lecturer) if correct, it will be continued in figure (b) where students will enter the online attendance page that has been developed and included with the Tensorflow.js library, which has been created and exported previously from the Teachable Machine service in the form of a javascript file, which is then modified in the website developed above. With the application of this attendance, students can take attendance in real-time which will detect the accuracy of each recognized face and will be stored in the database every time students take attendance.
Table 1. Evaluation Result Face Recognition
No Name / ID Male/Female Dataset Accuracy (%)
1 Student 01 M 25 100
2 Student 02 F 22 99
3 Student 03 F 23 99
4 Student 04 F 23 98
5 Student 05 F 23 82
6 Student 06 F 25 99
7 Student 07 F 23 46
8 Student 08 M 23 100
9 Student 09 M 23 97
10 Student 10 M 23 98
ACCURATE AVERAGE 91.8 %
.
In table 1 above, it can be seen that 10 sample datasets were tested using an average dataset of 23, then the average accuracy was 91.8%. where there is 1 class (number 7, pink color) which has an accuracy of 46%, this is due to the level of similarity with other data is very influential and will be divided, similarity factors greatly affect object recognition so that the accuracy value can decrease and will be divided into other class samples. Of course this must be considered when we take an image object, especially a face, it is necessary to pay attention to both clothing, background, identical faces (similar/twin), lighting will affect object recognition. So it is recommended to take a photo of the top of the head (forehead) to the chin only (closeup photo) with a white or color background only, so that the accuracy value can reach 98% - 100%.
4. CONCLUSION
In this study, several conclusions and suggestions can be drawn that need to be considered together, including; the application of face detection models using Teachable Machine with the Tensorflow.js library is very satisfying and helps in contributing to each model of different types of programming with applications that are easy to understand and also further modified. With the application and test results proven from 10 classes with an average number of samples is 23 samples, detected 9 classes that have an average value of accuracy is 96.8% (9 classes whose detection is good), while 1 other class has a poor accuracy of 46% which is influenced by the same type of clothing used by 2 different students, and also need to be considered other factors such as background, identical faces (similar / twin), lighting will affect object recognition as well. Then it is recommended to take a photo of the top of the head (forehead) to the chin only (closeup photo) with a white background or one color only. Of course, it will be possible later in machine learning using suggestions to minimize these factors, it is certain that the accuracy rate will be better and can reach above 95%. Maximizing the Convolutional Neural Network (CNN) applied to Tensorflow.js also needs to be considered the recommended number of samples is 80 samples or more in one class, so that the machine can learn and train from each dataset or test data more accurately.
REFERENCES
[1] T. Teachable Machine, “What is Teachable Machine?,” Teachable Machine, Sep. 03, 2017.
https://teachablemachine.withgoogle.com/ (accessed Sep. 26, 2022).
[2] M. Carney et al., “Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification,” in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Apr. 2020, pp. 1–8. doi:
10.1145/3334480.3382839.
[3] D. Agustian, P. P. G. P. Pertama, P. N. Crisnapati, and P. D. Novayanti, “Implementation of Machine Learning Using Google’s Teachable Machine Based on Android,” in 2021 3rd International Conference on Cybernetics and Intelligent System (ICORIS), Oct. 2021, pp. 1–7. doi: 10.1109/ICORIS52787.2021.9649528.
[4] C. Chazar and M. H. Rafsanjani, “Penerapan Teachable Machine Pada Klasifikasi Machine Learning Untuk Identifikasi Bibit Tanaman,” Prosiding Seminar Nasional Inovasi dan Adopsi Teknologi (INOTEK), vol. 2, no. 1, pp. 32–40, May 2022, doi:
10.35969/inotek.v2i1.207.
[5] Ö. F. ASLANDOĞDU, H. AYDIN, and A. ÇETİNKAYA, “Accuracy Prediction and Analysis of Teachable Machine (TM) Model developed with Tensorflow Javascript on the Cloud: Face Recognition System Implementation,” Uluslararası Yönetim Bilişim Sistemleri ve Bilgisayar Bilimleri Dergisi, vol. 6, no. 1, pp. 66–77, Jun. 2022, doi: 10.33461/uybisbbd.1106753.
[6] P. P. More and M. D. Jaybhaye, “Wear particles recognition through teachable machine,” Industrial Lubrication and Tribology, vol. 74, no. 2, pp. 274–281, Mar. 2022, doi: 10.1108/ILT-11-2021-0438.
[7] U. Dwivedi, “Introducing Children to Machine Learning Through Machine Teaching,” in Interaction Design and Children, Jun.
2021, pp. 641–643. doi: 10.1145/3459990.3463394.
[8] D. Wanyonyi and T. Celik, “Open-Source Face Recognition Frameworks: A Review of the Landscape,” IEEE Access, vol. 10, pp. 50601–50623, 2022, doi: 10.1109/ACCESS.2022.3170037.
[9] T. Baltrusaitis, P. Robinson, and L.-P. Morency, “OpenFace: An open source facial behavior analysis toolkit,” in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Mar. 2016, pp. 1–10. doi: 10.1109/WACV.2016.7477553.
[10] E. Sariyanidi, H. Gunes, and A. Cavallaro, “Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition,” IEEE Trans Pattern Anal Mach Intell, vol. 37, no. 6, pp. 1113–1133, Jun. 2015, doi:
10.1109/TPAMI.2014.2366127.
[11] M. Carney et al., “Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification,” in Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Apr. 2020, pp. 1–8. doi:
10.1145/3334480.3382839.
[12] Y. P. Pujari, D. Prasad, N. Malleswari, and N. Gupta, “Implementation of Machine Learning Based Google Teachable Machine in Early Childhood Education,” International Journal of Early Childhood Special Education, vol. 14, no. 3, pp. 4132–4138, May 2022.
[13] Teachable Machine, “Faq : Diving Deeper,” Teachable Machine, Nov. 08, 2019. https://teachablemachine.withgoogle.com/faq (accessed Oct. 02, 2022).
[14] C. D. Google, “What is TensorFlow.js?,” Codelabs Google, Mar. 31, 2022.
https://codelabs.developers.google.com/tensorflowjs-transfer-learning-teachable-machine#1 (accessed Oct. 03, 2022).
[15] C. D. Google, “TensorFlow.js: Make your own ‘Teachable Machine’ using transfer learning with TensorFlow.js,” Codelabs Google, Mar. 31, 2022. https://codelabs.developers.google.com/tensorflowjs-transfer-learning-teachable-machine#2 (accessed Oct. 03, 2022).
[16] M. D. Irawan, Flowchart dan Pseudo-Code: Implementasi Notasi Algoritma dan Pemrograman, 1st ed. Bandung: Media Sains Indonesia, 2022.
