Chapter 1 Introduction

It is hereby certified that FONG JIA YEE (ID No: 19ACB04133) has completed this final year project/dissertation/thesis* entitled “WEB-BASED RECEIVING KEY INFORMATION EXTRACTION SYSTEM” under the supervision of Ts Dr Tan Hung Khoon (supervisor) of the Department of Computer Science, Faculty of Information- and Communication Technology, and Pn Noraini Binti Ibrahim (Co-Leader) of the Department of Digital Economy Technology, Faculty of Information and Communication Technology. I declare that this report entitled "WEB-BASED RECEIPT KEY INFORMATION EXTRACTION SYSTEM" is my own work, except as cited in the references. Since this is a joint project carried out by Fong Jia Yee and Tan Ann Gee, the task allocation has been done in which I am responsible for integrating three models to perform key information extraction and to develop web-based receipt key information extraction system, while Ann Gee responsible for the development of mobile phone based receipt key information extraction system.

In the end, my developed system will be able to upload multiple receipts to be processed and store the records for review purposes, while Ann Gee's developed system will be able to upload one receipt and extract key information from the receipt obtain.

Introduction

• Key Information Extraction of Receipts
• Automated Receipt Processing
• Objectives
• Project Scope and Direction
• Contributions
• Report Organization

Since the models work well, the models should be implemented during the development of the web-based key information extraction system. The main objective of this project is to develop a web-based retrieval key information system. Keyword Information Extraction – To take the output from the text localization and OCR patterns and generate the output which is the keyword information extracted from the invoice.

After completing these tasks with the help of existing models, they should be put to use in the development of the key information extraction system of online retrieval. Upload Multiple Invoices – To develop a web based system which is able to upload multiple invoices for extracting keyword information from them simultaneously. Thus, in this project, I am required to develop the key information extraction system of the Internet.

Some models capable of handling text localization and keyword information extraction have been analyzed.

Figure 1.1.2: Documents with different types and layouts.

Literature Review

• Text Localization
• Connectionist Text Proposal Network (CTPN)
• Optical Character Recognition (OCR)
• Keyword Information Extraction
• Convolutional Universal Text Information Extractor (CUTIE)
• Bidirectional Encoder Representations from Transformers (BERT)
• Character-Aware Neural Language Model
• Limitations of Previous Studies

Based on the architecture given in Figure 2.3.1.3, there are four different paths where a high and consistent resolution for all features will be maintained in each path. Since the second sentence is the current future sentence of the first sentence, the target will be set to true. Since the second sentence is randomly selected from the corpus, the target will be set to false.

The selected document will be inserted into a vector by converting part of the texts into a fixed-size vector representing the semantics of the document. Each keyword will be entered using the same pattern previously used to enter the document. For different resulting keywords, maximum sum similarity (MSS) and maximum marginal relevance (MMR) will be applied.

Based on the architecture given in Figure 2.3.3.1, the input will be accepted by the first layer to perform character insertion, followed by convolution operations and max merge operation over time.

Figure 2.3.1.1: Equations to calculate the mapping position of texts in the grid [6].

System Design

System Overview

System Architecture Diagram

• Keyword Information Extraction Module i. Connectionist Text Proposal Network (CTPN)
• User Management Module
• User Management Module
• Use Case Diagram for Project I
• Activity Diagram for Project I i. “Upload files”

After the user logs into the system, the user will be navigated from "login". On the "login" page, the user will be navigated to the "registration" page to register a user account. On the "login" page, the user will be navigated to the "forgot password" page to reset the password.

Otherwise, an existing email will be displayed informing the user that the email address used to register has been used. If the user has successfully logged in, the user will be taken to the "upload" page, which is the main page of the system. Otherwise, a message with an invalid email address or password will be displayed to inform the user that the entry is incorrect.

If the user chooses to log out, the user will be navigated to the "login" page. Otherwise, an unregistered email message will be displayed to inform the user that the email used to reset the password is not a registered email. If the user chooses to view uploaded images, the image will be displayed in a new tab.

If the user chooses to view uploaded records, the uploaded data will be displayed if any records are found. If the user chooses to view deleted data, the deleted data will be displayed if the data is found. When the user has selected the files to upload, after clicking the “upload” button, the files will be validated by the system.

If the user has clicked on the post ID, the user will be navigated to the corresponding "record. If the user has clicked on the image URL, further explanation is provided in the "view uploaded images" activity diagram for project II.

Figure 3.2.1.2: Crop pre-processing flow [13].

Database Design

When the user clicks the "restore" button, the system will search for the related record and compare it with the activated records in the database. Otherwise, the system will activate the associated record and a successful recovery message will be displayed. If there are still records to restore, the system will repeat the previous process.

In the end, all messages will be displayed at the same time if the condition is met.

Table 3.6.1: Database structure.

System Implementation

Hardware Setup

Software Setup

• Visual Studio Code with WSL
• Python
• EasyOCR
• Flask
• MySQL Server

The Python version used should be above or equal to 3.7 which is recommended by the developers who provide the source code of the developed models and below or equal to 3.9 in which when Python 3.10 is used to train the models, some bugs are raised unexpected due to incompatible version of Python. EasyOCR is used as an API to connect both CTPN and Character-Aware Neural Language models by extracting the texts in the input image based on the coordinates of the texts extracted from the CTPN model to be the input of the Character-Aware Neural Language model to extract input image keyword information. Flask is a framework used to deploy trained models and create a web-based key retrieval system.

There are 11 attributes in the "records" table, namely record_id, image, text, upload_date, upload_time, company, address, date, total, user_id and status.

Models Training

After providing valid inputs in the "register" interface, the user is navigated to the "login" page and successfully registered message is displayed. When the files are processed, the progress bar is filled in color and the results are displayed as shown in figure 4.4.15. The user is navigated to “record details” page of record ID 1 after clicking on the URL of record ID 1.

After clicking the delete button on the "record details" page of record ID 1, the record with record ID 1 was removed from the "uploaded records" page. After clicking the delete button on the 'record details' page of Record ID 1, the record with Record ID 1 appears on the 'deleted records' page. After deleting the records with record ID 2 and 3, they were removed from the "uploaded records" page.

The re-uploaded records appear in the "Uploaded Records" page with record IDs 9, 10, and 11. Three records that appear in the "Deleted Records" page are duplicates of records with record IDs 9, 10, and 11. When selecting all records to be restored, records with record IDs 4 and 5 can be recovered and deleted from the "deleted records" page, while records with record IDs 1, 2, and 3 cannot be recovered.

After all records are deleted in "deleted records" page, successfully deleted and no record found messages are displayed. The "uploaded records" page will only display the record uploaded by the new user account, since the new user account is the current logged in account. The records from the previous user account are not displayed in the "uploaded records" page of the new user account.

The user uploaded the same receipt which was saved as a record in "uploaded records" page and duplicate record was detected. It can be clearly observed in figure 4.4.51 that both text files have the same content.

Figure 4.3.1: Training and evaluation outputs for both Connectionist Text Proposal Network (CTPN) and Character-Aware Neural Language models

Concluding Remark

Bachelor of Information Systems (Honours) Business Information Systems Faculty of Information and Communication Technology (Kampar Campus), UTAR. There were numerous errors when importing the source code of the models for training into Visual Studio Code. In the end, most of the errors were eliminated after changing the environment from Visual Studio Code to a Linux environment.

Bachelor of Information Systems (Honours) Business Information Systems Faculty of Information and Communication Technology (Kampar Campus), UTAR.

System Evaluation and Discussion

• System Testing and Performance Metrics
• Testing Setup and Result Use Case Register
• Objectives Evaluation
• Concluding Remark

The record is removed from the "uploaded records" page and appears on the "deleted records" page. The message about the record has not been restored followed by the duplicate of the registered record stored in the "uploaded records" page will be displayed.

Table 5.2.1: Register testing.

Conclusion and Recommendation

Conclusion

Recommendation

FINAL YEAR PROJECT WEEKLY REPORT

• WORK DONE
• WORK TO BE DONE
• PROBLEMS ENCOUNTERED
• SELF EVALUATION OF THE PROGRESS

Start working on Chapter 3 (System Design), Chapter 4 (System Implementation), and Chapter 5 (System Evaluation Discussion) of the project report at the same time. The models code was imported into Visual Studio Code without errors. Proceed with Chapter 3 (System Design), Chapter 4 (System Implementation), and Chapter 5 (System Evaluation Discussion) of the project report simultaneously.

The keyword information extraction model is able to generate key information extracted from each image. Function of the web-based receipt key information extraction system to display the records owned by the user. Function of the web-based receipt key information extraction system to display the record details and delete the record.

Function of the web-based receipt key information extraction system to display the deleted records. Function of the web-based receipt key extraction system to permanently delete the deleted record, restore the deleted record and reset the password. Note Promoter/Candidate(s) is/are required to provide a soft copy of the full set of the Originality Report to the Faculty/Institute.

Based on the above results, I hereby declare that I am satisfied with the originality of the Final Year Project Report submitted by my student(s) as mentioned above. Form title: Supervisor's comments on originality report generated by Turnitin for submission of final year project report (for undergraduate programs).

UNIVERSITI TUNKU ABDUL RAHMAN