International Journal of Advance Electrical and Electronics Engineering (IJAEEE)
_______________________________________________________________________________________________
Inertial Sensors Based Gesture Recognition
1Saya Mogra, 2Monish Kapadia, 3Adarsh Kalyanshetty, 4Chandrashekhar Beral
1,2,3,4Electronics Department Research and Innovation Center D. J. Sanghvi College of Engineering, Mumbai, India
Abstract-This paper presents the concept of inertial sensor based special character recognition. The hardware consists of a custom board compiled using atmega128 microcontroller, a Bluetooth wireless link and a 6 DOF (degree of freedom) inertial sensor MPU-6050 by InvenSense which is used for motion sensing. Intention of custom board is not to compute and draw inferences but to transfer data to PC where MATLAB functions would be computing the main job. MPU-6050 was tried integrating into pen like structure and users were asked to make gestures in air space. The raw data from the gestures is collected by the atmega128 over I2C protocol and is then transmitted to PC running MATLAB over UART serial protocol following a wireless link. The data collected in MATLAB is stored in an array and are in actual the coordinates of space representing some character. The stored data is then processed using a custom algorithm which identifies the slopes and distances between every two successive values and based on calculation outcomes, inference is drawn for individual characters.
Keywords—Atmega128 Microcontroller; Motion ProcessingUnit (MPU) -6050; Bluetooth Module(HC-06);
handwritten gesture recognition; Inter-Integrated Circuit (I2C) protocol; MATLAB(matrix laboratory); Convhull;
Convex hull; Slope; Distance;
I. INTRODUCTION
It focuses on the concept of air-writing [9] which is making gestures in air-space as if you are writing on an imaginary blackboard [10].The development of technology, especially in the field of computers has led to an indispensable rise in human-computer interface.
[1] Human computer interface is the basic exchange of information and particulars between humans and computers for certain tasks using some mechanism[2].The most important criteria while selecting a HCI device is how convenient and user- friendly the device is. Therefore, it’s of prime importance for researchers to develop advanced HCI technologies that helps users to easily give commands to get the desired output without putting extra time and effort in learning a protocol to handle the particular device. Gestures facilitate new forms of user interfaces, which will be especially suited for mobile and wearable computer systems. Instead of forcing a user to manually operate a device, hand gestures allow operation with empty hands and without the need to focus on tiny
screens and keys. Various sensing techniques (e.g.
cameras, inertial sensors) are used for the purpose of gesture recognition [10]. Today, technology has developed and enables processes such asmotion recognition, gesture recognition, handwriting recognition and audio-visual recognition with the help of human computer interface. [1]
In this paper we focus on the application of human- computer interface through special gestures mapping and its recognition through light weight process minimal intelligence. Air writing (special gesture movements and mapping) is the process of writing characters, symbols and gestures in air-space as if the user is writing on a virtual plane [9]. It is an easy task for the user as he only has to make the desired gestures or motions in air with his bare hands without the requirements of a notebook, keyboard, touch screen or a pen. In air-writing the orientation of the characters drawn is three dimensional and subjective of its user unlike the two-dimensional writing done on the PC. Thus air-writing is convenient, flexible, user friendly and has applications ranging from sign language, virtual reality, biomedical, health care and remote control. [3].Various sensing techniques are used to detect the air-writing. Using an accelerometer and gyroscope combination has proven to be one of the most efficient ways to detect the gestures made with respect to the X, Y, and the Z directions. It includes the combination of linear acceleration in each of these directions with the rotational acceleration about each axis. This amalgamation can be used to infer various results and values depending on their applications. In this application, a MPU-6050 which consists of an accelerometer and gyroscope is used to read and store raw data during air-writing.
Recognizing gestures in air without image or tactual feedback is a complex task because writing in air neither includes a particular sequence nor does it have a distinct reference position [9] and thus many complexities may arise. Since the handwriting form and orientation of the character might differ from user to user, efficient recognition of characters is a complex process and requires algorithms to efficiently recognize the gestures.
Algorithms are self-contained processes that are used for data processing. Here the algorithm draws inference from the received data and processes it to recognize the character that it denotes. The various algorithms that can
be used include Hidden Markov Method, Neural Networks, Modeling and recognition, [9], Spotting [10]
etc. However here we use a simple and more basic algorithm which is unique for every character or gesture.
We use a character-specific approach to identify the basic curves and slope differences of each character and then, using probability and estimation we device methods to successfully detect these characters.
II. WORKING
The hand gestures were made in air space holding the sensor MPU-6050 which was integrated into a pen like structure. Thevalues received were of the data type int16 since the resolution of each axis is 16 bit (2 bytes). So the data resulting per sample is 2bytes x 6 axis = 12 bytes. Initialized MPU system for 100 Hz i.e. 100 samples per second. It means 12x 100 = 1200 bytes per second. We have initialized the data links in the entire system such that it can handle the generated data transfer easily without lag/uncertainties.
Figure 1. Block Diagram
The mpu module is used to track down the gesture made in order to recognize the character drawn by the user. It works on I2C protocol and is initially at sleep. The module is a combination of 3-axis accelerometer and 3- axis gyroscope. The device has 6 Degrees of Freedom(DOF).Three degrees of freedom are assigned to the accelerometer to give the values of acceleration in the x, y and z directions while the other three are assigned to the gyroscope which gives the rotational values about x, y and z directions. A combination of both the accelerometer and the gyroscope readings are required/used to decode and recognize the gestures made. The data received is raw and noisy at this stage.
Therefore, a digital filter must be used to remove the noisy readings. The filter that we used is a complementary filter which is an amalgamation of values of both, the accelerometer as well as the gyroscope. We include gyroscope values because they are very accurate and susceptible and we include accelerometer values because they do not drift [4].Thus the filter uses the combination of 98% of gyroscope values and 2% of accelerometer values to get the final filtered values. The complimentary filter used gives us definite values at regular time intervals which can
further be used for plotting and processing values. The formula is as stated below:
Figure 2.1. Complimentary Filter Formula
Figure 2.2. Complimentary Filter Block Diagram Image by: robotics.stackexchange.com
We use an ATmega 128 to process the received values.
The values are transmitted using the I2C protocol. [5]
The key feature of this protocol is that communication can occur with multiple devices simultaneously using only two lines: SDA (Serial Data) and SCL (Serial Clock).This makes the system fast and reliable. It works on the Master Slave principle with ATmega128 as the master and the MPU-6050 as its slave. Every device is given a unique address through which it is accessed. The slave address of MPU is 0x68.The microcontroller is used to collect data from the sensor through the I2C protocol and then sent it to the Bluetooth module using the UART protocol.
Figure 3. MPU-6050 block diagram Image by: InvenSense
Figure 4. I2C protocol
UART stands for Universal Asynchronous Receiver / Transmitter. UART is the microchip with programming that controls a computer's interface to its attached serial devices. It is a serial communication protocol which is used to ‘talk’ and exchange data with the devices.
The data transmitted from the ATmega via UART is received by the Bluetooth module. It is a short-range communications technology that is used to replace the complexity of cables thus making the system portable
and also maintaining a high level of security. It consumes low power and has a high transfer rate.[6]
Here, we use the Bluetooth module HC-06. The Rx and Tx pins of the module are connected to the Tx and Rx pins respectively of the microcontroller. The processed values are transmitted to MATLAB wirelessly on the PC through Bluetooth.
Table I. Chip Specification [8]
Chips Function Main Specification ATmega 128 Receive data
from the sensor
and then
transmit to Bluetooth module
I2C UART
128kB Flash
Memory
MPU 6050 Sensor
integrated with three-axis accelerometer and three-axis gyroscope
16bits, primary and auxiliary I2C interface. Acc Range:
±2gˈ±4gˈ±8gˈ
±16g; Gyro. Range:
±250 ˈ ±500 ˈ
±1000ˈ±2000º/sec
HC-06 A Bluetooth
module to
transmit data wirelessly
2.4–2.484GHz ISM band
1 Mbps data transfer
III. ALGORITHM
The data that is collected on the computer device through the Bluetooth module is first converted into a text format and further arranged into an array. We have used a 2-D recognition algorithm which is unique for each of the characters that are recognized. The platform used to write the algorithm is MATLAB. MATLAB is a computer software that we use to decode and recognize the characters that are gestured. We use the inbuilt convhull function of the MATLAB Library which enables us to detect the outermost points of the given gesture. This function returns the convex hull of a set of given points as shown in the figure below.
k=convhull (x,y)
Figure 5. convhull function MATLAB
These points are recognized and named individually.
This enables us to individually access each set of coordinates of the given number of points. We use the functions vertx and verty to store the vertices of x and y respectively.
vertx = x; verty = y;
Each of these defined coordinates is used for distance and slope calculations. We use the basic slope calculation formula as follows:
a(i)=(verty(i+1)-verty(i))/(vertx(i+1)-vertx(i));
For every character that is to be recognized here, we develop an estimate for the range of values within which the slope of every line or curve of the character will lie and based on that probability we device an algorithm to detect the particular character. The characters that we detected here are:
(1) Letter 0
Figure 6.1. Letter O output(2) Letter α
Figure 6.2. Letter α output
(3) Letter V
Figure 6.3. Letter V output (4) Letter Z
Figure 6.4. Letter Z output
Once the algorithm is used, the computer displays the letter and the plot of the recognized symbol/character.
Here we have restricted the detection to a limited number of characters but we are currently working on extending this method to recognize all characters, numbers and symbols and thereby create a database for recognizing any gesture that are made.
IV. APPLICATIONS
An accelerometer and gyroscope based gesture recognition pen is one of the most suited application.
Technology has transformed over the years and has now become advance to allow convenience of writing in air using this pen and directly getting it stored on a device nearby through Bluetooth. This can be extended to communicate through Wi-Fi which will extend the range and the mobility of the device. For example, a student sitting in a classroom could automatically store class
notes on his personal computer through a wireless medium by simply making gestures in air. Such applications can also hope to spread through the industrial, medical and government fields in the near future because of its convenience, simplicity and low power requirements. With the deeper perception towards intelligent devices, the people have come to rely on HCI applications and so greater research in this field can be done to make life more comfortable and intelligent. [7]
REFERENCES
[1] 4 IEEE SENSORS JOURNAL, VOL. 15, NO. 1, JANUARY 2015 An Inertial Pen With Dynamic Time Warping Recognizer for Handwriting and Gesture Recognition Yu-Liang Hsu, Cheng-Ling Chu, Yi-Ju Tsai, and Jeen-Shing Wang, Member, IEEE
[2] 2013 3rd International Conference on Computer Science and Network Technology Smart Home Power Management System Design Based on Human-Computer Interaction Model Xiaoyun Zhao, Chunlai Zhou, Wenhui Huang Department of automation Communication University of China Beijing, China, 100024.
[3] Reading a IMU Without Kalman: The Complementary Filter. Peter-Jan.com
[4] IEEE SENSORS JOURNAL, VOL. 12, NO. 5, MAY 2012 MEMS Accelerometer Based Nonspecific-User Hand Gesture Recognition Ruize Xu, Shengli Zhou, and Wen J. Li, Fellow, IEEE
[5] Xuan Khanh Pham, Doan Que Anh Vo and Ngoc Hai Nguyen, The Phong Cao, PID-Neural Controller based on AVR Atmega128, 2008 10th Intl. Conf. on Control, Automation, Robotics and Vision Hanoi, Vietnam, 17–20 December 2008 [6] JoongHan Yoon , Young Chul RHee , Sang Mok
Lee , Woo Su Kim, Tunable Meander-Type Antenna Integrated with a Bluetooth Module in PCB Board, Proceedings of Asia-Pacific Microwave Conference 2010, FR1G-03
[7] 2013 3rd International Conference on Computer Science and Network Technolog. Smart Home Power Management System Design Based on Human-Computer Interaction Model. Xiaoyun Zhao, Chunlai Zhou, Wenhui Huang
[8] Proceedings of the 33rd Chinese Control Conference July 28-30, 2014, Nanjing, China.
Micro-IMU based Wireless Body Sensor Network.FEI Yunfeng1 , SONG Yinhao1 , XU Lin2*, and SUN Guangyi2
[9] IEEE TRANSACTIONS ON HUMAN- MACHINE SYSTEMS, VOL. 46, NO. 3, JUNE 2016. Air-Writing Recognition—Part I:
Modeling and Recognition of Characters, Words, and Connecting Motion. Mingyu Chen, Ghassan AlRegib, Senior Member, IEEE, and Biing- Hwang Juang, Fellow, IEEE.
[10] 2012 16th International Symposium on Wearable Computers. Airwriting: Hands-free Mobile Text Input by Spotting and Continuous Recognition of 3d-Space Handwriting with Inertial Sensors.
Christoph Amma, Marcus Georgi, Tanja Schultz Cognitive Systems Lab, Institute for Anthropomatics.
