INTRODUCTION
- Background Information
- Problem Statement and Motivation
- Project Objectives
- Project Scope and Direction
- Impact, Significance and Contribution
- Report Organization
In addition, the application provides a detailed and more realistic 3D model of the human digestive system, as the 3D model is reconstructed based on medical CT scan data. Chapter 2 provides an overview of the technologies that will be used in this project.
LITERATURE REVIEW
- Review of the Technologies
- Augmented Reality (AR)
- Types of Medical Imaging
- Medical Image Processing and Segmentation
- Review of the Existing Systems/Applications
- The Augmented Reality for Teaching Thai Students about the Human
- Utilising Mobile-Augmented Reality for Learning Human Anatomy
- Augmented Reality for the Study of Human Heart Anatomy
- An Interactive Augmented Reality System for Learning Anatomy Structure
- Human Anatomy Learning Systems Using Augmented Reality on Mobile
- Comparison between Existing Systems and Proposed Application
- Concluding Remark
Another weakness of the application is that the virtual 3D heart model is not accurate and realistic. The user can select each part of the bone and interact with the virtual object.
SYSTEM METHODOLOGY/APPROACH
Methodology and General Work Procedures
The main tools used in this project are 3D Slicer, Unity with Vuforia engine and Visual Studio Code. The technologies involved in this project are Augmented Reality (AR) technology and 3D reconstruction of medical images.
System Requirements
- Hardware
- Software
In the proposed project, Unity package with Vuforia Engine will be used for creating AR in mobile application. In the proposed project, Xcode 13 is used to support Unity game building on iOS devices.
Requirement Definition
- Functional Requirements
- Non-functional Requirements
The system will allow users to show or hide different parts of the AR model structure. The system will allow users to open the video player to play a video of the selected topic.
System Design / Overviews
- Use Case Diagram
- Use Case Descriptions
- Activity Diagrams
- Sequence Diagram
The student starts the activity by clicking the puzzle game button on the home page. Once the marker is detected, the system displays the AR model of the human digestive anatomy on the screen of the student's device.
Implementation Issues and Challenges
If the anatomy puzzle is placed in the wrong position, the system will return the puzzle to its original position. This process is repeated until the entire anatomy puzzle is pulled into the correct position and there are no puzzles left.
Project Timeline
- Timeline for FYP1
- Timeline for FYP2
Last but not least, another challenge in this project is the lack of knowledge and experience in using Unity to develop the proposed application. The focus in FYP2 is on development and implementation of the application, as well as testing of the application. Before starting to develop the application, a refinement of the previous work in FYP1 is first carried out.
Then the presentation and product demonstration are scheduled for the last two weeks, week 13 and week 14.
Concluding Remark
Finally, the project timeline is created to ensure that the project can be delivered on time.
PRELIMINARY WORK
- Overview of Preliminary Work Done
- Process of Image Segmentation using 3D Slicer
- Importing medical datasets to 3D Slicer
- Perform Segmentation on the CT images
- Refining the Segmentation and Applying Smoothing Effects
- Exporting the Segmented 3D model
- Results of Image Segmentation
- Concluding Remark
To perform image segmentation, the Segment Editor module as shown in Figure 4.2.2.1 is used in the 3D Slicer. Furthermore, the 'Scissors' tool is used on the 3D view to further segment the organs into different regions. In addition, Gaussian method is used to smooth all the details on the segments.
Figures 4.3.2 through 4.3.10 show the results of image segmentation of the liver, stomach, pancreas, and gallbladder.
SYSTEM IMPLEMENTATION
Setting Up Unity
Each game object is then associated with a series of components that determine how the game object will behave in the scene and react to other game objects in the scene. Some examples of components that can be linked to a game object are Transform, Physics, Audio, Image, Vertical Layout Group, Scripting etc. Go to File > Build Settings, in the Platform section, click Android and switch the platform to Android as shown shown in Figure 5.1.3.
Then go to the player settings and set the icon by dragging the logo, logo foreground or logo background into the corresponding sections of flexible icons, round icons and legacy icons, as shown in Figure 5.1.5.
Setting Up Unity and Xcode on macOS Virtual Machine
Setting Up Vuforia Engine
- Adding Vuforia App License Key to Unity
- Creating and Adding Target Marker Database to Unity
To get the license key, go to Developer > on the Vuforia Engine developer portal. As shown in Figure 5.3.2.4, the target marker uploaded to this project has the highest five-star rating for augmentable, which means the marker is suitable to be used for target image recognition by AR. As shown in Figure 5.3.2.5, download the database from the Vuforia Engine developer portal and import the database into Unity.
As shown in Figure 5.3.2.6, under the Database section, click Add Database to add the database.
Development of Main Menu Scene
As shown in Figure 5.4.2, the render mode is set to “Screen Space – Camera” to ensure that the user interface always appears within the device's camera view. In addition, as shown in Figure 5.4.3, the user interface scaling mode is set to 'Scaling with screen size', so that the user interface will change proportionally depending on the screen size. When the user clicks the “Exit” button, a confirmation window appears, as shown in Figure 5.4.9.
Then LeanTween is downloaded and imported into the project by Package Manager in the Unit, as shown in Figure 5.4.11.
Development of Play AR Scene
The HorizontalRotateButtonClicked method is added to the game object, as shown in Figure 5.5.13, so that when the user clicks this toggle button, the horizontal autorotation of the 3D model is toggled. After closing the guideline panel, the user interface as shown in Figure 5.5.23 is presented to the user before the user starts scanning the target image. The user can also tap the organ to select the structure part, as shown in Figure 5.5.24.
Additionally, when the user clicks the arrow button on the right side of the screen, the right navigation panel will slide onto the screen as shown in Figure 5.5.28.
Development of Select Topic Scene
The user can check or uncheck the organs listed in the right navigation panel to show or hide the organ on the screen. This script is to create the buttons of each anatomy topic with topic image and topic name, and to store the topic clicked by the user in a static variable to be passed to the next scene. To the On Click() section of the ButtonTemplate game object, ChangeSceneButton method of CanvasTransition script is added, so that the scene will be changed to the Topic Info scene when the user clicks the ButtonTemplate.
Development of Topic Information Scene
CloseButton button to close the panel containing a zoomed version of the anatomy structure image. A ScrollArea game object is added to the canvas to allow the user to scroll up and down through the contents of the topic information. The function of this script is to set the title bar and the content of the Topic Information Scene, such as the video thumbnail, anatomy structure image, and description texts, based on the anatomy topic selected by the user from the previous Select Topic scene.
The title bar text and corresponding video thumbnail sprites and texture sprites are assigned to the Theme Info script.
Development of Video Player Scene
EnterFullScreenButton button to enter the full screen mode of the video by changing the screen orientation to landscape. TimeSlider Slider that shows the progress of playing video and allows the user to slide it to change the current playing video frame. On the PauseButton game object, the PlayVideo script's Pause method is assigned to the On Click() section, as shown in Figure 5.8.6, so that the video will be paused.
Finally, when the user clicks on the close button in the upper left corner of the screen, the video will close and the user will be navigated back to the Topic Info Scene.
Development of Anatomy Puzzle Game Scene
For example, Figure 5.9.5 shows the Puzzle Game script added to the Pancreas game object. When the user enters the Puzzle Game Scene, the Stopwatch script will be called to start the stopwatch. If the user can click the help button in the upper right corner of the screen, a panel with instructions to play the game will appear, as shown in Figure 5.9.12.
After the user has successfully dragged the entire anatomy puzzle into the correct position, a success panel, with the time used to complete the game, will appear to congratulate the user and allow the user to make the choice to play again or appear in the main menu as shown in figure 5.9.13.
Build Configuration
- Build Application on Android Device
- Build Application on iOS Device
After that, as shown in Figure 5.10.1.2, select the running device to be the connected mobile device and click "Build And Run". Next, start collaboration and publish the project to the cloud, as shown in Figure 5.10.2.2. After the project opens successfully, go to File > Build Settings > iOS, then click on 'Switch Platform' to switch to the iOS platform as shown in Figure 5.10.2.5.
Then enable automatically manage signing, the provisioning profile and signing certificate will then be generated automatically as shown in Figure 5.10.2.6 and Figure 5.10.2.7.
Concluding Remark
In Xcode, click Unity-iPhone > Signature & Capabilities, then select 'Team' to be the developer ID and set the package identifier to a unique name. Now, the project is ready to run and build on the connected iPhone device by clicking the Play button in the upper left corner of Xcode. The application will open on the iPhone after the project build is completed successfully.
Furthermore, each of the functionality modules developed is briefly described with screenshots of the application developed.
SYSTEM EVALUATION
- System Testing
- Test Cases for Main Menu
- Test Cases for Play AR Functionality
- Test Cases for Learn Anatomy Functionality
- Test Cases for Anatomy Puzzle Game Functionality
- Survey Questionnaire
- Objectives Evaluation
- Concluding Remark
As shown in the figure, the respondents agree that the application helps them understand anatomy faster: 40%. According to the result obtained, as shown in the figure, the respondents agree that this AR application provides a more realistic-looking 3D representation of the human digestive system, while the remaining 46.7% of the respondents are neutral. It can therefore be concluded that most respondents will probably use this AR application again.
The results obtained from the survey are optimistic as most of the respondents are satisfied with the application.
CONCLUSION AND RECOMMENDATION
Brief Summary
Novelties and Contributions
Future Work
Providing detailed information about the organs of the human digestive system with descriptions, pictures and videos. Would you agree that this AR application provides a more realistic 3D representation of the human digestive system compared to the 2D images in the textbook. A video player function for playing videos of the organs of the human digestive system is made.
The work to be done in the next two weeks is to create the user interface and functionality of the anatomy puzzle module and to test and debug the built module.
