Multi-Sperm Tracking Using Hungarian Kalman Filter on ... - iKnow

(1)

Multi-Sperm Tracking Using Hungarian Kalman Filter on Low Frame Rate Video

Grafika Jati*, Alexander A S Gunawan ** ,Silvia Werdhy Lestari ***, WisnuJatmiko *, MH Hilman ****

*Faculty of Computer Science, Universitas Indonesia - Depok Email: [email protected]

**Mathematics Department, School of Computer Science, Bina Nusantara University, Jakarta, Indonesia

***Faculty of Medicine, Universitas Indonesia

****Melbourne School of Engineering, University of Melbourne - Australia

Abstract--One factor of human sperm health is sperm motility.

Motility is the ability of sperm to move. Sperm with healthy motility move forward promptly, not inactive and not moving in circles. In this paper, we would like to analyse sperm motility by considering the problem of multi object tracking in video sequences of human sperms. The challenges in multi-sperm tracking are many human sperms have fast and unpredictable movement. Inaddition, the sperm have similar size and shape comparing by each others. To solve this problem, we used sperm detection in each video sequence to get the position of sperms.

In the same time, the estimated sperm position is calculated based on previous tracking by using Kalman Filter. Finally the positions of detected sperms are compared to estimation results by using Hungarian assignment method. In this way, the trajectory of each sperm can be conclude. This paper analyze sperm motility qualitatively based on the resulted sperm trajectories. The experiment results were conducted on both open video data and our own low-frame-rate video data. The experiment results shows that the proposed method can handle the challenges in multi sperm tracking, create their trajectory and then analyze their behaviors.

I. INTRODUCTION

In the middle-l980, Andrologist introduced computer- assisted semen analysis (CASA) [1]. This system predict sperm motility and evaluated men fertility from general popu- lation. Unfortunately, CASA systems requires substantial user intervention to operate and remain prohibitively expensive.

On the other hand, research of human sperm tracking has been approached by some researchers. Human sperm motility analysis from video data is one of the active research area nowadays. The common researches in this area are single sperm cell detection and tracking.

In sperm tracking research, Zhou et al [2] utilize particle filter to track sperm position by estimating the target distribution.

Particle Filter make tracker away from local optimum trap. So it maintain the distribution. Then Zhou combine Particle Filter with mean shift that facilitates the estimate of local optimal orientation. It enhance the Particle Filter in capturing sperm motion. Ravanfar et al [3] proposed combination of particle filter and watershed segmentation to sperm tracking with collision problem. The tracker utilize sperm head orientation and morphological characteristics to detect occlusion inter- sperms. The following research by Ravanfar et al [4] try to

overcome the challenge that sperm cell has a similar shape and size to each other by applying limitation for search space. Gunawan et al [5] proposed searching driven stochastic sampling framework for abrupt motions on low frame rate sperm video. The above researches tryto overcome the main problems in sperm tracking that are: occlusion, background clutter, and fast motion. Nevertheless, the researches are still limited to a single sperm tracking. The recent single sperm tracking focus on optimizing tracking algorithm using optimization algorithm. Aprinaldi et al [6] tryto optimize sperm tracking based Smooth Stochastic Sampling using Particle Swarm Optimization (PSO). The Smooth Stochastic Sampling estimate fast and unpredictable sperm motion. Then the PSO search the best target in the determined Search Window.

Inobject tracking, Kalman Filter are widely used for real- time tracking application [7],[8], and [9]. Based on Bayesian Filter framework, Kalman Filter estimate state of the system and its variance by employing linear assumption. Although Kalman Filter is not too robust in facing tracking challenges, but it consistently outperformed other tracking algorithms in term of speed. Liu et al [10] has utilized Kalman Filter to quantify the locomotive behavior of human sperm heads and tails. Our previous research [11] also used one variation of Kalman Filter, that is Extended Kalman Filter (EKF) algorithm. In this paper, Kalman Filter is utilized as multi sperm tracking combine with detection method. By using Otsu method for sperm detection, the position of sperms in each video sequence can be determined. In the same time, the estimated sperm position is calculated based on previous tracking by employing Kalman Filter. Finally the positions of detected sperms are compared to estimation results by using Hungarian assignment method.Inthis way, the trajectory of each sperm can be concluded in each video frame. In our experiment, the approach can be overcome the fast and unpredictable sperm movement. Furthermore, it can handle the abrupt sperm motion in our low-frame-rate video sequences.

In section 2, the background method that is used in this paper will be explained such as Kalman filter and Hungarian assignment method. Design of proposed method is described in Section 3. Experiments result is presented in Section 4, and the performance of the proposed method is evaluated for two real

(2)

scenarios, one with open data (good frame rate) and the other c. Update the error covariance one with our own data (low frame rate). The conclusion and

further discussion as open problems of this paper are briefly Pk= (1 - _KkH)Pk-

discussed in the last section of the manuscript. ⁽⁷⁾

Kk =Pk-HT(HPk-HT +R)-l (5)

b. Update the estimate next state after get measurementZk II. BACKGROUND

Kalman Filter is utilized for estimating target position and Hungarian assignment method for concluding the right sperm trajectory. Inthis section, the brief description about Kalman Filter and Hungarian assignment method are described as follows:

A. Kalman Filter

Kalman Filter is an optimal estimator, it predict optimally the state based on series of observed measurement over time [12]. Kalman Filter was developed by Rudolf E Kalman for filtering in 1960.It presume parameters of interest from indirect, inaccurate, and uncertain observation. Kalman Filter run recursively so the new measurement can be integrated as they arrive. It is a part of Gaussian tracker which assumes all involved distributions are Gaussian and the dynamics is linear [13].

In Kalman Filter, it is necessary to build the model that suits the problems. The main model equation is

The above process run between prediction and correction iteratively over timek.The previous estimates will be input for the current state. The

xi:

is prior estimate and alsoP_k- is prior error covariance. Then the prior value is used for measurement equation 5,6,7. To get posterior value in measurement update, equation 5 computes Kalman Gain (K_k). Then the posterior valueXkis calculated in equation 6. Finally we computePkin equation 7 which is necessary for the estimated value in next time(k+ 1).

III. METHODOLOGY

The main pipeline of the proposed method (sperm tracker) can be seen in fig. 1. The pipeline is inspired from multi-object tracking framework [15]. The pipeline starts with gathering video dataset of cell sperms. Then video will be processed into sequence of images. After that, the multi sperm detection is done for each frame. The detection process consist of noise reduction and intensity thresholding. In the same time, the estimated sperm position is calculated based on previous tracking by employing Kalman Filter. Finally the positions of detected sperms are compared to estimation results by using Hungarian assignment method. Output of the propose method is sperm position and trajectory.

A. Multi-Sperm Detection

The appearance of multi sperm is detected in three stages, that is: background subtraction, noise reduction using 2D Gaussian Filter, and threshold in binary image or Otsu's method. Detection process can be seen in fig 2 and it is explained below:

B. Hungarian Assignment Method

Hungarian method or also known as Munkres assignment method is a combinatorial optimization algorithm which is used to solve assignment problem [14]. It is introduced by Harold Kuhn in 1957. Assignment problem deals with assign- ing delivery jobs to path, worker to jobs, machine to tasks, and so on. The goal is to get optimum assignment.Ifgiven nxn matrix, the steps of Hungarian method as follow:

Step 1. For each row, choose the smallest entry in each row and subtract it to all the entries of its row.

Step 2. For each column, choose the smallest entry in each column and subtract it to all the entries of its column.

Step 3. Make line through rows and columns which cover all zeroes in the matrix, then count the number of line.

Step 4. Algorithm finish if minimum number of covering line isn and then choose the zeroes as optimal assignment results.

Ifminimum number of covering line less thann go to step 5.

Step 5. Choose the smallest entry that is not covered by any line. Subtract it from each uncovered row and add to each covered column. Go to step 3 again.

(3) (1) (2)

(4)

(6)

Xk=Axk-l +BUk +Wk-l Zk= HXk +Vk

1. Time Update(prediction) : a. Project the state ahead

xi:

=AXk- 1+BUk b. Project the error covariance matrix ahead

Pi: =Ap,,_IA^T

+

Q 2. Measurement update (correction):

a. Compute the Kalman Gain

A,H,B are general form matrices. Xk (current state) may be evaluated using linear stochastic equation 1. Xk is linear combination of previous state (Xk-I> plus control signal (u) and noise (w). Insperm tracking control signal is not used.

The current measurement Zk can be calculate by 2. It is a linear combination between current state value and measurement noise. After the model is built, we estimate the mean and standard deviation of Gaussian noise function Wk-l and Vk. In here, all noises are assumed as Gaussian signal. Next the Kalman Filter process consist of two main stages: time update (prediction) and measurement update (correction). The steps in time update and measurement update are described as follows

(3)

Input

~

I

~

^{Frame 1} ¹ Sperm Detection Output

iH

Sperm Tracking

~I:

Finding Multi

I

Noise Reduction

estimation .1 ^Hungarian ^{Soerm Path}

I

^Video ^{Frame 2}

I

I~ Gaussian filter

~I

Kalman filter

I _I

^Assignment

_I

\

Thresholdin!!^Intensity

^I

^Method

I

^position^distance

I

detection

I'

^{Frame n}

I

Fig. 1. Main pipeline of propose method

Original Video Frame Image Noise Reduction Intensity Thresholding Point x-y Centroid Detection

Fig. 2. Flowchart of spenn detection process

cr is standard deviation of the distribution, whilex and y is position of the pixel. The mean of distribution is assumed to be zero. Based on Gaussian distribution, it is designed Laplacian of Gaussian f1G(x,y) to blob detection. Finally, the input image is convolved to the Laplacian of Gaussian.

1) Image Noise Reduction using 2D Gaussian Filter and Blob Detection

The first step of detection stage is reducing noise in sperm cell image. Image noise is caused by random electrical disturbance in video imaging system. This phe- nomenon is manifested in bad image whose has random light, dark pixels, pixel or low contrast. There are many sperms in each image frame with many unknown noises.

By using 2D Gaussian filter, many noises can be reduced from the image. 2D Gaussian filter can be written as:

2) Intensity Thresholding

Observing that sperm cell has different pixel intensities from background, threshold operation can be used to detect the sperm cell. Pixel whose intensity value exceed to threshold will be accepted, and threshold limit will be rejected. This procedure is called as Otsu's method that produces binary black and white image.

The determination of the threshold value depends on the brightness of sperm cell image. Thus the threshold is set manually based on experiment result. The threshold value for our own and Kokopeli video dataset [16] is 0.8. In addition, 1.0 means white (high contrast) and 0.0 means black (low contrast).

1 hr

G(x,y)= - 2ze- 2(7

ncr ⁽⁸⁾

3) Point x-y Detection

Sperm detection is done by finding extreme value in an image based on pixel resolution as describe in [17].The extreme value in here is the pixel which has the maximal intensity value.

B. Multi-Sperm Tracking

While sperm has been detected in certain frame, the estimated sperm position is calculated based on previous tracking by employing Kalman Filter in the same time. The estimation on frame n is based on the sperm position in the frame n - 1.

The noise variances in Kalman Filter are assumed based on the variability of how fast the sperm move in centimeters.

second2

Standard Kalman Filter performs poorly when tracking multiple target in close proximity or whose paths intersect.

Thus the standard Kalman Filter can not used in multi sperm tracking directly. The combination between predicted sperm positions from Kalman Filter and detected positions from detection stage can be used to overcome the problems. By using Hungarian assignment method, the detected positions will be assigned to estimated positions. To use Hungarian method, we create the distance (cost) matrices between all pairs which rows are estimated positions, and column are detected positions. The pseudocode of our multi sperm tracking can be seen in Algorithm 1.

IV. RESULT

In order to evaluate the proposed method, experiments has been done by using video data to track the movement of a human sperm cell. The experiments are implemented on Intel i5 2.50 GHz CPU with 4 Gigabytes RAM. The experiments used two video data, the first data is retrieved from Dr. Cipto Mangunkusumo Hospital Lab, which have low

(4)

Fig. 3. Result multi spenn detection in fast motion dataset

For multi sperm tracking the evaluation is divided into three challenges: tracking in fast motion dataset, in low frame rate, and with partially occlusion. Overall the experiment results show that both video can be tracked well.

. . . .

Frame #:40

Frame #12

26 GlIt

26 It0

26

..

. ..

_:

.

%

Frame #5

27

.. .. .

. . .

Frame #5

I: procedure MULTI-TRACKING ALGORITHM

2: define variables for Kalman Filter

3: define starting frame

4: define measurement noise for x and y coordinate

5: define coefficient matrices

6: initialize result variable

7: estimation position

8: determine number of track

9: convert noise to covariance matrix

10: for each frame

11: load image

12: make given detection matrix from detection process

13: do Kalman Filter

14: predict next state of cell sperms with the last state and predict motion

15: predict next covariance

16: calculate Kalman Gain

17: assign detection to estimated track position

18: make distance (cost) matrices. Row

=

tracks and Column= detection

19: do assignment to distance matrices using Hun- garian Assignment Algorithm

20: check if detection far from observation than reject estimation

21: get assignment result

22: find new detection and lost tracking

23: draw track

24: end for

25: end procedure

frame rate about 20 fps)[5] and the second data is retrieved from Kokopelli Technology [16]. Each dataset consist of 300 number of sperm cell image sequence. The qualitative challenges in the experiments are fast motion of many sperm, abrupt motions due to low frame rate in our own dataset, and partially occlusion between sperm during motion.

Algorithm 1 Multi-tracking Algorithm

Fig. 4. Result multi spenn detection in Low Frame Rate dataset

1) Tracking in Fast Motion Dataset: The challenge can be overcame well because many sperms can be detected accurately in same time. By comparing to estimated position from the Kalman Filter, then Hungarian assignment method can conclude the trajectory of sperms well. Nevertheless sometimes the estimated position is not reliable and cause Hungarian method assign the wrong sperm positions.Infig 5, itshow illustrated performance of tracker facing fast motion challenge.Inframe #5 sperms is initial position. In frame #15 sperms had move away from initial position and so on. Each sperm track is visualized by color.

A. Qualitative Evaluation

Multi sperm detection is evaluated before multi sperm tracking. Fig 3 shows that multi detection method can detect high frame rate dataset with a stable. The number of detected sperm can either increases or decreases due to there are sperm enter or exit to the frame. In fig 3, it detected 48 sperms in frame #5, 47 sperms in frame #10, 49 sperms in frame

#20, and 48 sperms in frame #40. However, when observed visually there are some miss-detection.Itis found that some noises are detected as sperms. In low frame rate, proposed method also successfully detect multi sperm. The result can be seen in fig 4. The proposed method detected 26 sperms in frame #5, 26 sperms in frame #12, 27 sperms in frame

#17, and 26 sperms in frame #40. Finally, it is calculated that detection accuracy is about 90% comparing to visual measurement.

Frame #17 Frame #45

(5)

Fig. 5. Result multi spenn tracking in fast motion dataset

is combining multi detection method with Kalman Filter and Hungarian Assignment Method. Detection process follows with estimating process. Finally the positions of detected sperms are compared to estimation results by using Hungarian assignment method. So the trajectory of each sperm can be estimated well.

For future work, there some parameter that can be tuned.

The detection method can be modify with other noise reduction method. So it can get more clear detection. Another assignment method like fuzzy Hungarian Assignment method can be used to get more accurate estimation. Next work we have to calculate some variable such as curvilinear velocity(VCL), straight-line velocity (VSL), Linearity of forward progression (LIN), and amplitude of lateral head displacement (ALH). That variables are important to sperm quality assess- ment. Furthermore the tracking method will be improved using modified PSO. Jatmiko at al [18] modify PSO that solve multi source problem. It is appropriate to case multi sperm tracking.

Fig. 6. Result multi spenn tracking in abrupt motion dataset

V. CONCLUSION

Based on experiments that had performed, the proposed method can perform multi human sperm cell tracking on low frame rate and high frame rate video. The proposed method 3) Tracking in Partially Occlusion: Partially Occlusion is the most difficult problem in this experiment. Our tracker sometime can swap the tracking target between other sperm.

In many image frames, Hungarian method can handle this problem to assign the right track for the right sperm.

REFERENCES

[1] R. P. Amann and D. F. Katz, "Andrology lab comer': Reflections on casa after 25 years," Journal of andrology, vol. 25, no. 3, pp. 317-325, 2004.

[2] X. Zhou and Y. Lu, "Efficient mean shift particle filter for spenn cells tracking," inComputational Intelligence and Security, 2009. CIS '09.

International Conference on, vol. I, Dec 2009, pp. 335-339.

[3] M. Ravanfar and M. Moradi, "Low contrast spenn detection and tracking by watershed algorithm and particle filter," inBiomedical Engineering (ICBME), 2011 18th Iranian Conference of, Dec 2011, pp. 260-263.

[4] M. Ravanfar, L. Azinfar, M. H. Moradi, and R. Fazel-Rezai, "Occlusion robust low-contrast spenn tracking using switchable weight particle filtering," Advances in Sexual Medicine, vol. 4, no. ASM, Vol.4 No.3, July 2014, pp. 42-54, 2014.

[5] A. A. Gunawan and A. M. Arymurthy, "Visual tracking for abrupt motions of human spenn using smoothing stochastic approximate monte carlo," Procedia Computer Science, vol. 59, pp. 64 - 72, 2015, international Conference on Computer Science and Computational Intelligence (rCCSCr 2015).

[6] Aprinaldi, G. Jati, A. A. S. Gunawan, A. Bowolaksono, S.W.Lestari, and W. Jatrniko, "Human spenn tracking using particle swarm optimization combined with smoothing stochastic sampling on low frame rate video," in2015 International Symposium on Micro-NanoMechatronics and Human Science (MHS), Nov 2015, pp. 1-6.

[7] W.Blair, "Design of nearly constant velocity track filters for tracking maneuvering targets," inInformation Fusion, 2008 11th International Conference on, June 2008, pp. 1-7.

[8] --, "Design of nearly constant velocity track filters for brief maneu- vers," inInformation Fusion (FUSION), 2011 Proceedings of the 14th International Conference on, July 2011, pp. 1-8.

[9] Y. Bar-Shalom, X. R. Li, and T. Kirubarajan,Estimation with Applica- tions To Tracking and Navigation. A Wiley-Interscience Publication, 2001.

[10] J. Liu, Z. Lu, C. Leung, and Y. Sun, "Quantitative analysis of locomotive behavior of human spenn head and tail," in Experimental Robotics, ser. Springer Tracts in Advanced Robotics, J. P. Desai, G. Dudek, O. Khatib, and V. Kumar, Eds. Springer International Publishing, 2013, vol. 88, pp. 603-616. [Online].

Available: htrp:lldx.doLorgllO.1007/978-3-319-00065-7_41

VI. ACKNOWLEDGEMENT

This work is supported by "Hibah Publikasi Intemasional Terindeks untuk Tugas Akhir Mahasiswa"

No. 18471UN2.RI2/HKP.05.0012016 year 2016 entitled

"Pengembangan Swarm Mobile Robot untuk Patroli di Area Pabrik Industri" by the University Indonesia.

.

^,

.

^,

Frame #50 Frame #15

• 0

.

⁰

~

Frame #5

• 0 }

0

"'" \ ... ,..

,

.}

Frame #25

2) Tracking in Low Frame Rate Dataset: Low frame rate dataset make the sperm look like moving abruptly in video dataset. The proposed tracker can successfully through the whole sequence. Fig 6 show the performance of our tracker in handling abrupt motion of many sperm moving. It can be seen in frame #5 some sperms have inactive motions. And in frame #50, there are some sperms that move actively.

(6)

[II] M. A. Ma'sum, M. K. Arrofi, G. Jati,F. Arifin, M. N. Kurniawan, P. Mursanto, and W. Jatmiko, "Simulation of intelligent unmanned aerial vehicle (uav) for military surveillance," inAdvanced Computer Science and Information Systems (ICACSIS), 2013 International Conference on, Sept 2013, pp. 161-166.

[12] A. J. Haug,Bayesian Estimation and Tracking. A Wiley-lnterscience Publication, 2012.

[13] J. Mochnac, S. Marchevsky, andP. Kocan, "Bayesian filtering tech- niques: Kalman and extended kalman filter basics," inRadioelektronika, 2009. RADIOELEKTRONIKA '09. 19th International Conference, April 2009, pp. 119-122.

[14] M. Junger, T. Liebling, D. Naddef, and G. Nemhauser, 50 Years of Integer Programming 19582008. Springer, 2010.

[15] Student-Dave's-Tutorials. (2013) Multi-bug (object) tracking.

Tutorial. [Online]. Available: http://studentdavestutorials.weebly.coml multi-bugobject-tracking.html

[16] Kokopeli, Juny 2015. [Online]. Available: http:www.fertilityformen.

cominfospermvideo.php

[17] T. Ursell. (2013) Find image extrema. Matlab Exchange. [On- line). Available: http://www.mathworks.comlmatlabcentrallfileexchange/

41955-find-image-extrema/content/imextrema.ml

[18] W. Jatmiko, A. Nugraha, R. Effendi, W. Pambuko, R. Mardian, K. Sekiyama, and T. Fukuda, "Localizing multiple odor sources in a dynamic environment based on modified niche particle swarm optimization with flow of wind," WSEAS Transactions on Systems, vol. 8, no. 11, pp. 1187-1196, 2009, cited By 9. [Online]. Available:

https:llwww.scopus.comlinward/record.uri?eid=2-s2.D-74349089006&

partnerID=40&md5=Ib135100d6f29d2165c3607age097fa5

[19] Y. Okada,K.Dejima, and T. Ohishi, "Analysis and comparison of PM synchronous motor and induction motor type magnetic bearings," IEEE Trans. Ind. Appl., vol. 31, pp. 1047-1053, Sep.lOct. 1995.