27th Iranian Conference on Electrical Engineering (ICEE2019)
978-1-7281-1508-5/19/$31.00 ©2019 IEEE
Data augmentation of CT images of liver tumors to reconstruct super-resolution slices based on a multi-
frame approach
Moslem Farhadi
Department of Biomedical Engineering, Engineering Faculty,
Shahed University, Tehran, Iran [email protected]
Amir Hossein Foruzan*
Department of Biomedical Engineering, Engineering Faculty,
Shahed University, Tehran, Iran [email protected]
Abstract— Improvement of tumor images have applications in the extraction of features and image retrieval algorithms. The multi- frame super-resolution technique is a chief approach in high- resolution image reconstruction. However, the acquisition of several low-resolution frames is not practical in the medical domain. In this paper, we use volumetric CT data of the abdominal region to prepare new low-resolution images and employ them in a data augmentation approach to the super- resolution image reconstruction. We showed that our method improved the results of interpolation and conventional multi- frame approaches by 0.02 and 0.03 respectively using the SSIM index.
Keywords- Super-resolution, hepatic tumors, liver CT images, data augmentation, image reconstruction.
I. INTRODUCTION
Early and accurate detection of hepatic tumors helps physicians for better treatment planning and decreases the rate of mortality [1]. The conventional non-invasive procedure to search for abnormality in internal organs of the body is using imaging modalities. In its early stage, a tumor can be detected with much effort. High-resolution (HR) images are required for delineation and classification of tissue malignancies. A high- resolution image gives us more attributes of the tissue and discriminates different tumor types and determines the border of abnormality straightforwardly [2-5]. However, due to hardware limitations, cost issues, or other challenges, it is not an easy task to obtain HR data. Improving the resolution of an image can be performed via software solutions. The ensemble of techniques to obtain an HR image from a low-resolution (LR) input data is termed super-resolution (SR) algorithms.
To obtain an HR image, the LR image is interpolated, and high-frequency (HF) information is added to it. The major challenge in SR algorithms is to obtain HF information from an LR input. Based on the number of input images, SR techniques are classified into single-frame and multi-frame methods.
In a supervised single-frame approach (also called an example-based technique), a training dataset is used to build LR and HR dictionaries. For a test LR image, its patches are reconstructed using the LR dictionary. The reconstruction coefficients are used to build the corresponding HR patches [6]. In an unsupervised method, the image pyramid is employed to prepare several sample images with different
scales. Then, the HR data for the current level is obtained from the patches of the lower scales.
In the multi-frame method, a series of aliased LR frames are available, and it is assumed that each frame contains a part of HR data. A model is considered for the degradation of an HR image into the LR data. Since the model is an ill-posed problem, some constraints are included in the equation. In some cases, data augmentation techniques are employed to prepare multi-frame data as well.
II. PREVIOUS WORKS
The work presented in [7], was among the first researches on preparing an HR image using an ensemble of LR noisy images. They considered the SR problem as a restoration issue and proposed three approaches to obtain the solution: (1) the Maximum Likelihood (ML) estimation, (2) the Maximum a Posteriori (MAP) probability method, and (3) a set-theoretic approach [7]. They evaluated their method using synthetic data as well. In their subsequent papers, the researchers addressed the restoration of the HR image from purely translated LR data [8] and considered the robustness of the results to noise and motion parameters [9].
Yan et al. indicated that obtaining a set of LR CT images from a patient is not practical and it also involves risks to the patient [10]. Therefore, they considered the restoration of an HR CT image in the acquisition domain. Zhang et al. [11]
considered image super-resolution issue as a sparse- representation problem and used optimization techniques with constraints to obtain the HR image. To optimize the problem containing multiple variables, they updated each variable individually. They applied their method on five 2D medical images of both MR and CT data [11]. Li et al. [12] proposed a selection strategy for multi-frame images. They fused the selected LR images several times with multiple references to improve the result of the SR technique. In [13], a comprehensive review of SR algorithms is given.
In this paper, we confront the problem of multi-frame medical images. We propose a method to obtain several frames from a single acquisition. We employ the conventional method of multi-frame SR technique and use the MAP estimation to obtain HR results. As our minor novelty, we decide on the optimum parameters to achieve desirable outcomes in cases of
27th Iranian Conference on Electrical Engineering (ICEE2019) CT images of liver tumors. As far as we know, major SR
researches are in the field of brain MR images, and a few are in the field of CT hepatic tumors.
III. PROPOSED METHOD
The flowchart chart of the proposed algorithm is shown in Fig. 1. We use a volumetric CT data as input, augment data, and employ a multi-frame technique to obtain the HR image.
Figure 1. The flowchart of the proposed method.
We assume that an observed LR image ( ) is the result of warping, blurring, and subsampling of an HR image (1).
y = + , 1 ≤ ≤ .
(1) where is the high resolution image, is the transformation matrix, and is the blurring and down- sampling operators, and is the additive zero mean Gaussian noise. Concatenating individual images ( ), we represent (1) as in (2).
= + .
(2) In (2), , and are concatenated LR images, warping, and noise matrices, respectively. We obtain the Map estimation of the HR image by maximization of the conditional probability of knowing ( ( | )) as define in (3) [7].
= argmax ( | ) = argmax ( | ) ( ). (3) Representing the autocorrelation matrices of the noise and the HR images as and respectively, Equation (3) is rewritten as (4).
= argmax [ − ] [ − ] + . (4)
A mean square solution to (4) gives as in (5) to (7).
= .
(5)
= + . (6)
= . (7)
To produce new data from available CT images, we interpolate neighboring slices. Due to the slice thicknesses in CT data acquisition, each part of the tissue is scanned by a CT slice and its nearby slices as well. We exploit the overlap of CT slices to obtain new HF information from neighboring images.
To augment new data to slice , we obtain a new slice by
interpolation of the two neighboring slices and . The interpolation kernel is linear. The range of intensities of the interpolated and original images may be different. This issue makes the extracted HF data different from the two images and therefore the SR results are not acceptable. To resolve the problem, we can either normalize the two images or use histogram matching techniques. We checked the two solution and obtained better results by the histogram matching technique.
(a)
(b)
Figure 2. Variations of the PSNR against (a) PSF width and (b) noise level.
We use conventional rigid warping to obtain sequences of synthetic images ( , 1 ≤ ≤ ) from and . The new data is obtained from the observed LR image using (8).
y = , 1 ≤ ≤ , 1 ≤ ≤ . (8) In (8), and are the uniformly distributed random translation and rotation matrices in the range [−4, 4] and [−0.5, 0.5] respectively. Based on our experience, we assign non-integer translation parameters to improve the results.
The imaging parameters, including the Point Spread Function (PSF) and the noise level, were selected experimentally. We changed the PSF and noise level and evaluated the result using the PSNR measure. The best result was obtained with PSF = 0.5 and noise Std = 0.03 (Fig. 2).
A key point in the SR algorithm is to register LR data to compensate for any possible misalignments. For multi-frame data, typical registration algorithms such as optical flow are appropriate. In cases of single frame data including medical images, the registration parameters are available, and they are fed into the SR algorithm. Finally, we utilize the MAP technique to estimate the HR image.
IV. RESULTS AND DISCUSSIONS
We evaluated our algorithm using 23 multi-phase CT volumes with a dimension of 512×512×70 and element spacing 0.6836×0.6836×5 mm3. The data were obtained from the abdominal region of patients including one or more liver
27th Iranian Conference on Electrical Engineering (ICEE2019) tumors. We implemented our algorithm on an Intel coreTM-
i5with 4 GB DRAM without graphics acceleration. The code was implemented on MATLAB 2016 environment.
The evaluation measures are Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM) defined in (9) –(11).
= ∑ ∑ ( ( , ) − ( , )) /( ). (9)
= 20 log max( ) − 10 log . (10)
( , ) =
2 + 2 + + + + + .
(11)
In (9)-(11), and are image intensities, and are the sizes of the images, and are the mean and standard deviation of the corresponding images, respectively. The PSNR is the conventional similarity index while the SSIM is a recently introduced metric which locally measures the similarity between two images.
We performed three sets of experiments: (1) interpolation of a slice using the bicubic kernel, (2) reconstruction of SR slices by the conventional data augmentation technique, and (3) extraction of SR images by the proposed data augmentation method. In cases of data augmentation approaches, we used an identical number of images. In Fig. 3, we compare the typical results of the three approaches using the PSNR measure. We performed the experiments with non-contrast (NC), arterial (ART), and portal vein (PV) CT phases.
Figure 3. Comparison of the SR results by interpolation, the conventional data augmentation method, and the proposed algorithm.
The average PSNR (SSIM) metrics of the interpolation- based, conventional MAP super-resolution, and the proposed data augmentation techniques are 21.27 (0.64), 23.54 (0.65), and 23.74 (0.67), respectively.
Based on our knowledge, our paper is the first research on the super-resolution of liver tumor images. Our method can improve the outcomes of imaging modalities with traditional hardware. The output of our research can influence image retrieval algorithms as well.
Another important point about our research is the method of LR data preparation. In some multi-frame algorithms, the LR images are prepared from the original HR data. However,
the HR image is not available in real cases. In our algorithm, we prepare the LR images from the input LR data.
Another point is the dependence between the spacing of an input image and the output of the algorithm. For images with large voxel spacing, we found that the output of our method is not better than the conventional SR algorithms.
In Fig. 4, we show typical results of the SR reconstruction by the interpolation method and the proposed technique. As is shown in Fig. 4, the texture of our result is more similar to the real HR image. This point is of more importance when one uses an image to extract textural information.
The performance of our method depends on the augmented LR data and the outcomes of the registration technique as well.
When we use integer shifts to prepare LR data, no high- frequency information is added to the data. Therefore, our results are not improved. When non-integer random shifts are used to prepare augmented data; however, the PSNR of the SR image is enhanced well.
Phase Original HR image
Interpolation result
The Proposed method
Non-Contrast Arterial Portal-Vein
Figure 4. Typical SR results using interpolation-based technique and the proposed method. Dataset #101.
Multi-frame SR algorithms rely on the similarity of the image regions; however, the similarity of brain tissues are more than abdominal regions. To compensate for the lower similarity, we use the high-frequency information inherent in lateral slices. We are employing the histogram matching algorithm to balance the intensity variation of neighboring slices as well. Regarding the state-of-the-art SR techniques, we plan to apply our algorithm on MR brain images to compare our results fairly with other techniques.
V. CONCLUSION AND FUTURE WORKS
In this paper, we proposed an SR reconstruction algorithm based on a novel data augmentation technique. We optimized the parameters of our algorithm and used the MAP estimation of the results. Our results outperformed the interpolation
27th Iranian Conference on Electrical Engineering (ICEE2019) method and the conventional MAP estimation of the SR
technique.
In the future, we plan to utilize high-frequency information in multi-phase data to reconstruct tumor images. We decide to apply our method on MR brain images as well.
ACKNOWLEDGMENT
The authors would like to thank Prof. Yen-Wei Chen, Ritsumeikan University, Kansai, Japan, for the use of their images in this study.
REFERENCES
[1] Kharel R, Iyer V, Noronha V, Majumder PD, Biswas J. Role of high- resolution computerized tomography chest in identifying tubercular etiology in patients diagnosed as Eales’ disease. Journal of ophthalmic inflammation and infection. 2017 Dec;7(1):4.
[2] Greenspan H. Super-resolution in medical imaging. The Computer Journal. 2008 Feb 19;52(1):43-63.
[3] Ben-Ezra A, Greenspan H, Rubner Y. Regularized super-resolution of brain MRI. In Biomedical Imaging: From Nano to Macro, 2009.
ISBI'09. IEEE International Symposium on 2009 Jun 28 (pp. 254-257).
IEEE.
[4] Lu Y, Yang R, Zhang J, Zhang C. Super-resolution image reconstruction in parallel magnetic resonance imaging. InControl and Automation (ICCA), 2010 8th IEEE International Conference on 2010 Jun 9 (pp.
761-766). IEEE.
[5] Robinson MD, Chiu SJ, Lo J, Toth C, Izatt J, Farsiu S. New applications of super-resolution in medical imaging. Super-Resolution Imaging. 2010 Sep 28;2010:384-412.
[6] Romero, Jose E., Pierrick Coupé, and Jose V. Manjón. "Non-local MRI Library-Based Super-Resolution: Application to Hippocampus Subfield Segmentation." In International Workshop on Patch-based Techniques in Medical Imaging, pp. 68-75. Springer, Cham, 2016.
[7] M. Elad and A. Feuer, “Restoration of Single Super-Resolution Image From Several Blurred, Noisy and DownSampled Measured Images”, the IEEE Trans. on Image Processing, Vol. 6, no. 12, pp. 1646-58, December 1997.
[8] M. Elad and Y. Hel-Or, “A Fast Super-Resolution Reconstruction Algorithm for Pure Translational Motion and Common Space Invariant Blur,” the IEEE Trans. on Image Processing, Vol.10, No. 8, pp.1187-93, August 2001.
[9] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and Robust Multi-Frame Super-resolution”, IEEE Trans. On Image Processing, Vol.
13, No. 10, pp. 1327-1344, October 2004.
[10] Yan Z, Li J, Lu Y, Yan H, Zhao Y. Super-resolution in CT. International Journal of Imaging Systems and Technology. 2015 Mar;25(1):92-101.
[11] Zhang M, Desrosiers C, Qu Q, Guo F, Zhang C. Medical image super- resolution with non-local embedding sparse representation and improved IBP. In Acoustics, Speech, and Signal Processing (ICASSP), 2016 IEEE International Conference on 2016 Mar 20 (pp. 888-892). IEEE.
[12] Li, Xuelong, et al. "A multi-frame image super-resolution method."
Signal Processing 90.2 (2010): 405-414.
[13] Yue L, Shen H, Li J, Yuan Q, Zhang H, Zhang L. Image super- resolution: The techniques, applications, and future. Signal Processing.
2016 Nov 1;128:389-408.
