Big Multimodal Data-driven Medical Image Processing for the COVID-19 Diagnosis

Shengli Xie; Sergey Gorbachev; Zhaoshui He; Xiaozhao Fang; Zuyuan Yang; Guoxu Zhou

doi:10.57118/creosar/978-1-915740-01-4_2

Authors

Shengli Xie
Sergey Gorbachev
Zhaoshui He
Xiaozhao Fang
Zuyuan Yang
Guoxu Zhou

DOI:

https://doi.org/10.57118/creosar/978-1-915740-01-4_2

Keywords:

Information retrieval, hash codes, principal components analysis, residual preservation

Abstract

Diagnosis using medical images is extremely labor-intensive and could even be subjective. With the exciting progress of artificial intelligence (AI) in the last decade, it has been increasingly realistic and promising to develop automatic diagnoses from medical images. In this chapter, we give comprehensive advances on recent data-driven AI technologies for the COVID-19 diagnosis based on medical images. Moreover, rather than using a massive volume of data to train a network, it is also valuable to take multi-modal multi-view data into a unified learning framework to improve the accuracy of diagnosis. To this end, two categories of representation learning methods are proposed to deal with this variety and variability of big medical image data. One is an average approximate hashing (AAH) method for searching large-scale multimedia databases, which projects data into different semantic spaces but shares a unified hash code. The other focuses on nonnegative matrix factorization-based clustering models for multi-view data. Experiments justify the effectiveness and efficiency of the proposed methods.

References

WHO Coronavirus Disease (COVID-19) Dashboard. Accessed: Aug. 11, 2021.

J. Wang, Y. M. Bao, Y. F. Wen, H. B. Lu, H. Luo, Y. F. Xiang, et al., "Prior-attention residual learning for more discriminative COVID-19 screening in CT images," IEEE Transactions on Medical Imaging, vol. 39, pp. 2572–2583, 2020.

S. Minaee, R. Kafieh, M. Sonka, S. Yazdani, G. J. Soufi, "Deep-COVID: Predicting COVID-19 from chest X-ray images using deep transfer learning," Medical image analysis, vol. 65, pp. 101794, 2020.

Y. Oh, S. Park, J. C. Ye, "Deep learning COVID-19 features on CXR using limited training data sets," IEEE Transactions on Medical Imaging, vol. 39, pp. 2688–2700, 2020.

M. D. Zhang, R. H. Chu, C. Y. Dong, J. G. Wei, W. H. Lu, N. X. Xiong, "Residual learning diagnosis detection: An advanced residual learning diagnosis detection system for COVID-19 in industrial internet of things," IEEE Transactions on industrial informatics, vol. 17, pp. 6510–6518, 2021.

M. Raginsky, S. Lazebnik, "Locality-sensitive binary codes from shift-invariant kernels," in Advances in Neural information Processing Systems, British Columbia, Canada, 2009, pp. 1509–1517.

M. Datar, P. indyk, N. Immorlica, V. S. Mirrokni, "Locality-sensitive hashing scheme based on p-stable distributions," in Twentieth Symposium on Computational Geometry, New York, USA, 2004, pp.253-262.

B. Kulis, K. Grauman, "Kernelized locality-sensitive hashing for scalable image search," in IEEE international Conference on Computer Vision, Kyoto, Japan, 2010, pp. 2130-2137.

S. Bhattacharjee, J. Yuan, Y. Huang, J. Meng, L. Duan, "Queryadaptive multiview object instance search and localization using sketches," IEEE Trans. Multimedia, vol. 20, pp. 2761–2773, 2018.

C. Zhang, H. Fu, Q. Hu, X. Cao, Y. Xie, D. Tao, D. Xu, "Generalized latent multi-view subspace clustering," IEEE Trans. Pattern Analysis and Machine intelligence, vol. 42, pp. 86-99, 2018.

G. Y. Zhang, C. D. Wang, D. Huang, W. S. Zheng, Y. R. Zhou, "Tw-co-k-means: two-level weighted collaborative k-means for multi-view clustering," Knowledge-Based Systems, vol. 150, pp. 127–138, 2018.

F. Nie, G. Cai, J. Li, X. Li, "Auto-weighted multi-view learning for image clustering and semi-supervised classification," IEEE Trans. Image Processing, vol. 27, pp. 1501–1511, 2018.

X. Luo, X.-Y. Yin, L. Q. Nie, X. M. Song, Y. X. Wang, X.-S. Xu, "SDMCH: supervised discrete manifold-embedded cross-modal hashing," in Proceedings of the Twenty-Seventh international Joint Conference on Artificial intelligence, Stockholm, Sweden, 2018, pp. 2518–2524.

L. Liu, Z. J. Lin, L. Shao, F. M. Shen, G. G. Ding, J. G. Han, "Sequential discrete hashing for scalable cross-modality similarity retrieval," IEEE Transactions on Image Processing, vol. 26, pp. 107–118, 2017.

G. Irie, H. Arai, Y. Taniguchi, "Alternating co-quantization for cross-modal hashing," in 2015 IEEE international Conference on Computer Vision, Santiago, Chile, 2015, pp. 1886–1894.

B. T. Wu, Q. Yang, W.-S. Zheng, Y. Z. Wang, J. D. Wang, "Quantized correlation hashing for fast cross-modal search," in Proceedings of the Twenty-Fourth international Joint Conference on Artificial intelligence, Buenos Aires, Argentina, 2015, pp. 3946–3952.

Y. Wang, X. M. Lin, L. Wu, W. J. Zhang, Q. Zhang, "LBMCH: learning bridging mapping for cross-modal hashing," in Proceedings of the 38th international ACM SIGIR Conference on Research and Development in information Retrieval, Santiago, Chile, 2015, pp. 999–1002.

Y. X. Peng, H. Xin, Y. Z. Zhao, "An overview of cross-media retrieval: Concepts, methodologies, benchmarks and challenges," IEEE Transactions on Circuits Systems for Video Technology, vol. 28, pp. 2372–2385, 2018.

G. G. Ding, Y. C. Guo, J. L. Zhou, "Collective matrix factorization hashing for multimodal data," in IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014, pp. 2083-2090.

D. Mandai, S. Biswas, "Label consistent matrix factorization based hashing for cross-modal retrieval," in IEEE international Conference on Image Processing, Beijing, China, 2017, pp. 2901–2905.

C. Yan, J. L. Jiang, Z. H. Lai, Z. J. Hu, W. K. Wong, "Supervised discrete discriminant hashing for image retrieval," Pattern Recognition, vol. 78, pp. 79–90, 2018.

J. Tang, K. Wang, L. Shao, "Supervised matrix factorization hashing for cross-modal retrieval," IEEE Transactions on Image Processing, vol. 25, pp. 3157–3166, 2016.

X. Liu, A. Li, J. X. Du, S. J. Peng, W. T. Fan, "Efficient cross-modal retrieval via flexible supervised collective matrix factorization hashing," Multimedia Tools & Applications, vol. 77, pp. 28665–28683, 2018.

X. Xu, F. Shen, Y. Yang, H. T. Shen, X. Li, "Learning discriminative binary codes for large-scale cross-modal retrieval," IEEE Transactions on Image Processing, vol. 26, pp. 2494–2507, 2017.

D. Zhang, W. J. Li, "Large-scale supervised multimodal hashing with semantic correlation maximization," in Twenty-eighth AAAI Conference on Artificial intelligence, Québec, Canada, 2014, pp. 1.

Z. J. Lin, G. G. Ding, M. Q. Hu, J. M. Wang, "Semantics-preserving hashing for cross-view retrieval," in 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015, pp. 7–12.

A. Kumar, P. Rai, H. Daume, "Co-regularized multi-view spectral clustering," in Neural information Processing Systems, Granada, Spain, 2011, pp. 1413–1421.

S. Zeng, X. Wang, H. Cui, C. Zheng, D. Feng, "A unified collaborative multikernel fuzzy clustering for multiview data," IEEE Trans. Fuzzy Systems, vol. 26, pp. 1671–1687, 2018.

X. W. Liu, X. Z. Zhu, M. M. Li, L. Wang, C. Tang, J. P. Yin, et al., "Late fusion incomplete multi-view clustering," IEEE Trans. Pattern Analysis and Machine intelligence, vol. 41, pp. 2410–2423, 2019.

G. Y. Zhang, C. D. Wang, D. Huang, W. S. Zheng, "Multi-view collaborative locally adaptive clustering with minkowski metric," Expert Systems with Applications, vol. 86, pp. 307–320, 2017.

Z. Zhang, L. Liu, F. Shen, H. Shen, L. Shao, "Binary multi-view clustering," IEEE Trans. Pattern Analysis and Machine intelligence, vol. 41, pp. 1-9, 2019.

P. Zhou, Y. D. Shen, L. Du, F. Ye, X. Li, "Incremental multi-view spectral clustering," Knowledge-Based Systems, vol. 174, pp. 73–86, 2019.

Q. Yin, S. Wu, L. Wang, "Multiview clustering via unified and view-specific embeddings learning," IEEE Trans. Neural Networks and Learning Systems, vol. 29, pp. 5541–5553, 2018.

S. Huang, Z. Kang, I. W. Tsang, Z. Xu, "Auto-weighted multi-view clustering via kernelized graph learning," Pattern Recognition, vol. 88, pp. 174–184, 2019.

K. Chaudhuri, S. M. Kakade, K. Livescu, K. Sridharan, "Multi-view clustering via canonical correlation analysis," in int. Conf. Machine Learning, Montreal, Canada, 2009, pp. 129–136.

Z. C. Li, J. H. Tang, T. Mei, "Deep collaborative embedding for social image understanding," IEEE Trans. Pattern Analysis and Machine intelligence, vol. 41, pp. 2070–2083, 2019.

Y. Z. Jiang, Z. H. Deng, F.-L. Chung, S. T. Wang, "Realizing two-view tsk fuzzy classification system by using collaborative learning," IEEE Trans. Systems, Man, and Cybernetics: Systems, vol. 47, pp. 145–160, 2017.

Y. Wang, L. Wu, X. M. Lin, J. B. Gao, "Multiview spectral clustering via structured low-rank matrix factorization," IEEE Trans. Neural Networks and Learning Systems, vol. 29, pp. 4833–4843, 2018.

C. Q. Zhang, H. Z. Fu, Q. H. Hu, X. C. Cao, Y. Xie, D. C Tao, et al, "Generalized latent multi-view subspace clustering," IEEE Trans. Pattern Analysis and Machine intelligence, vol. 42, pp. 86–99, 2020.

F. Z. Zhuang, X. B. Li, X. Jin, D. P. Zhang, L. R. Qiu, Q. He, "Semantic feature learning for heterogeneous multitask classification via non-negative matrix factorization," IEEE Trans. Cybernetics, vol. 48, pp. 2284–2293, 2018.

D. D. Lee, H. S. Seung, "Learning the parts of objects by nonnegative matrix factorization," Nature, vol. 401, pp. 788–791, 1999.

J. L. Liu, C. Wang, J. Gao, J. W. Han, "Multi-view clustering via joint nonnegative matrix factorization," in int. Conf. Data Mining, Nevada, USA, 2013, pp. 252–260.

H. Wang, Y. Yang, T. R. Li, "Multi-view clustering via concept factorization with local manifold regularization," in int. Conf. Data Mining, Nevada, USA, 2016, pp. 1245–1250.

L. L. Zong, X. C. Zhang, L. Zhao, H. Yu, Q. L. Zhao, "Multi-view clustering via multi-manifold regularized non-negative matrix factorization," Neural Networks, vol. 88, pp. 74–89, 2017.

S. D. Huang, H. J. Wang, T. Li, T. R. Li, Z. L. Xu, "Robust graph regularized nonnegative matrix factorization for clustering," Data Mining and Knowledge Discovery, vol. 32, pp. 483-503, 1999.

X. M. Wang, T. Z. Zhang, X. B. Gao, "Multi-view clustering based on non-negative matrix factorization and pairwise measurements," IEEE Trans. Cybernetics, vol. 49, pp. 3333-3346, 2019.

M. Yin, J. B. Gao, S. L. Xie, Y. Guo, "Multi-view subspace clustering via tensorial t-product representation," IEEE Trans. Neural Networks and Learning Systems, vol. 30, pp. 851–864, 2019.

Z. J. Lin, Z. S. He, S. L. Xie, X. Wang, J. Tan, J. Lu, et al, "AANet: Adaptive attention network for COVID-19 detection from chest X-ray images," IEEE Transactions on Neural Networks and Learning Systems, vol. 32, pp. 4781–4792, 2021.

F. Chen, F. Wu, J. Xu, G. W. Gao, Q. Ge, X.-Y. Jing, "Adaptive deformable convolutional network," Neurocomputing, vol. 453, pp. 853–864, 2021.

K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 770–778.

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, "End-to-End object detection with transformers," in Proceedings of European Conference on Computer Vision, Sec, Glasgow, 2020, pp. 213–229.

A.Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N . Gomez, . Kaiser, et al, "Attention is all you need," in Advances in Neural information Processing Systems, Long Beach, CA, USA, 2017, pp. 5998–6008.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. H. Zhai, T. Unterthiner, et al, "An image is worth 16x16 words: Transformers for image recognition at scale," in Proceedings of international Conference on Learning Representations, Vienna, Austria, 2021, pp. 1–21.

J. Nie, Y. W. Pang, S. J. Zhao, J. G. Han, X. L. Li, "Efficient selective context network for accurate object detection," IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, pp. 3456–3468, 2021.

Y. T. Hu, J. Li, Y. F. Huang, X. B. Gao, "Channel-wise and spatial feature modulation network for single image super-resolution," IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, pp. 3911–3927, 2020.

K. Simonyan, A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proceedings of the 3rd international Conference on Learning Representations, San Diego, CA, USA, 2015, pp. 1–14.

K. Zhang, X. H. Liu, J. Shen, Z. H. Li, Y. Sang, X. W. Wu, et al, "Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography," Cell, vol. 181, pp. 1423–1433, 2020.

X. Y. Mei, H.-C. Lee, K.-Y. Diao, M. Q. Huang, B. Lin, C. Y. Liu, et al, "Artificial intelligence–enabled rapid diagnosis of patients with COVID-19," Nature Medicine, vol. 26, pp. 1224–1228, 2020.

C. Jin, W. X. Chen, Y. K. Cao, Z. W. Xu, Z. M. Tan, X. Zhang, et al, "Development and evaluation of an artificial intelligence system for COVID-19 diagnosis," Nature Communications, vol. 11, pp. 1–14, 2020.

L. Li, L. X. Qin, Z. G. Xu, Y. B. Yin, X. Wang, B. Kong, et al, "Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: Evaluation of the diagnostic accuracy," Radiology, vol. 296, pp. E65–E71, 2020.

L. D. Wang, Z. Q. Lin, A. Wong, "COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images," Scientific Reports, vol. 10, pp. 1–12, 2020.

A. Shamsi, H. Asgharnezhad, S. S. Jokandan, A. Khosravi, P. M. Kebria, D. Nahavandi, et al, "An uncertainty-aware transfer learning-based framework for COVID-19 diagnosis," IEEE Transactions on Neural Networks and Learning Systems, vol. 32, pp. 1408–1417, 2021.

B. Wu, Y. Z. Wang, "Neighborhood-preserving hashing for large-scale cross-modal search," in Proceedings of the 2016 ACM Conference on Multimedia Conference, Wörthersee, Austria, 2016, pp. 352–356.

M. Zheng, J. J. Bu, C. Chen, C. Wang, L. J. Zhang, G. Qiu, et al, "Graph regularized sparse coding for image representation," IEEE Transactions on Image Processing, vol. 20, pp. 1327–1336, 2011.

H. Zou, T. Hastie, R. Tibshirani, "Sparse principal component analysis," Journal of Computational & Graphical Statistics, vol. 15, pp. 265–286, 2006.

J. Franklin, "The elements of statistical learning: data mining, inference and prediction," Publications of the American Statistical Association, vol. 99, pp. 567–567, 2010.

G. F. Lu, Y. Wang, J. Zou, Z. Wang, "Matrix exponential based discriminant locality preserving projections for feature extraction," Neural Networks, vol. 97, pp. 127–136, 2017.

X. Z. Fang, K. H. Jiang, N. Han, S. H. Teng, G. X. Zhou, S. L. Xie, "Average approximate hashing-based double projections learning for cross-modal retrieval," IEEE Transactions on cybernetics, DOI 10.1109/TCYB.2021.3081615, 2021.

S. P. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, "Distributed optimization and statistical learning via the alternating direction method of multipliers," Foundations and Trends in Machine Learning, vol. 3, pp. 1–122, 2011.

D. Wang, X. B. Gao, X. M. Wang, L. H. He, "Semantic topic multimodal hashing for cross-media retrieval," in Proceedings of the Twenty-Fourth international Joint Conference on Artificial intelligence, Buenos Aires, Argentina, 2015, pp. 3890–3896.

L. Xin, Z. K. Hu, H. B. Ling, Y. M. Cheung, "Mtfh: A matrix tri-factorization hashing framework for efficient cross-modal retrieval," IEEE Transactions on Pattern Analysis and Machine intelligence, vol. 43, pp. 964–981, 2021.

Z. J. Lin, G. G. Ding, J. G. Han, J. M. Wang, "Cross-view retrieval via probability-based semantics-preserving hashing," IEEE Transactions on Cybernetics, vol. 47, pp. 4342–4355, 2017.

K. Simonyan, A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014. Available: https://arxiv.org/pdf/1409.1556.pdf

D. Mandal, K. N. Chaudhury, S. Biswas, "Generalized semantic preserving hashing for cross-modal retrieval," IEEE Transactions on Image Processing, vol. 28(1):102–112, 2019.

J. L. Zhou, G. G. Ding, Y. C. Guo, "Latent semantic sparse hashing for cross-modal similarity search," in Proceedings of the 37th ACM SIGIR Conference on Research and Development in information Retrieval, Queensland, Australia, 2014, pp. 415–424.

Z. Y. Yang, N. Y. Liang, W. Yan, Z. N. Li, S. L. Xie, "Uniform distribution non-negative matrix factorization for multiview clustering," IEEE Trans. Cybernetics, vol. 51, pp. 3249–3262, 2021.

C. Ding, T. Li, W. Peng, H. Park, "Orthogonal nonnegative matrix tri-factorizations for clustering," in int. Conf. Knowledge Discovery and Data Mining, Philadelphia, USA, 2006, pp. 126–135.

Z. G. Yang, Q. Li, W. Y. Liu, J. M. Lv, "Shared multi-view data representation for multi-domain event detection," IEEE Trans. Pattern Analysis and Machine intelligence, vol. 42, pp. 1243-1256, 2019.

N. Y. Liang, Z. Y. Yang, Z. N. Li, W. J. Sun, S. L. Xie, "Multi-view clustering by non-negative matrix factorization with co-orthogonal constraints," Knowledge-Based Systems, vol. 194, pp. 105582, 2020.

H. Zhao, Z. Ding, Y. Fu, "Multi-view clustering via deep matrix factorization," in AAAI Conf. Artificial intelligence, California, USA, 2017, pp. 2921–2927.

N. Rai, S. Negi, S. Chaudhury, O. Deshmukh, "Partial multi-view clustering using graph regularized NMF," in int. Conf. Pattern Recognition, Cancun, Mexico, 2016, pp. 2192–2197.

N. Y. Liang, Z. Y. Yang, Z. N. Li, S. L. Xie, C. Y. Su, "Semi-supervised multi-view clustering with graph-regularized partially shared non-negative matrix factorization," Knowledge-Based Systems, vol. 190, pp. 105185, 2020.

J. Y. Peng, P. Luo, Z. Y. Guan, J. P. Fan, "Graph-regularized multi-view semantic subspace learning," int. Journal of Machine Learning and Cybernetics, vol. 10, pp. 879–895, 2019.

S. Li, H. F. Liu, Z. Q. Tao, Y. Fu, "Multi-view graph learning with adaptive label propagation," in int. Conf. Big Data, Boston, MA, USA, 2017, pp. 110–115.

N. Y. Liang, Z. Y. Yang, Z. N. Li, S. L. Xie, W. J. Sun, "Semi-supervised multi-view learning by using label propagation based non-negative matrix factorization," Knowledge-Based Systems, vol. 228, pp. 107244, 2021.

A. P. Singh, G. J. Gordon, "Relational learning via collective matrix factorization," in int. Conf. Knowledge Discovery and Data Mining, Las Vegas, USA, 2008, pp. 650–658.