x
Volume 45 Issue 3
May  2022
Turn off MathJax
Article Contents
Article Navigation >  Journal of Molecular Imaging  > 2022  >  45(3): 459-464
WANG Zongmin, FU Lin, GAO Yunyue. Research progress of multimodal medical imaging based on deep learning[J]. Journal of Molecular Imaging, 2022, 45(3): 459-464. doi: 10.12122/j.issn.1674-4500.2022.03.30
Citation: WANG Zongmin, FU Lin, GAO Yunyue. Research progress of multimodal medical imaging based on deep learning[J]. Journal of Molecular Imaging, 2022, 45(3): 459-464. doi: 10.12122/j.issn.1674-4500.2022.03.30
shu

Research progress of multimodal medical imaging based on deep learning

doi: 10.12122/j.issn.1674-4500.2022.03.30

  • Department of Ultrasound Diagnosis, Affiliated Hospital of Inner Mongolia Medical University, Hohhot 010000, China

  • Received Date: 2022-02-24
    Available Online: 2022-06-21
  • Publish Date: 2022-05-20
    Abstract
  • With the rapid development of deep learning, big data and other technologies, artificial intelligence is the most promising technology in the field of medicine. In view of the key role of medical imaging in the diagnosis and timely treatment of diseases, the combination of medical imaging and artificial intelligence is becoming an essential interdisciplinary research direction. In clinical practice, doctors often need to refer to multimodal image data for comprehensive analysis and judgment in order to diagnose diseases more accurately and comprehensively. This paper first introduces the basic concept and working principle of multi-mode deep learning, and the representative research results of deep learning technology applied to multi-mode medical image-assisted diagnosis are summarized. Finally, the technical challenges of multi-mode deep learning technology in the medical image field are analyzed, and the application prospect of multi-mode deep learning technology is forecasted.

     

    • FullText(HTML)
  • loading
    • References(50)
  • [1]
    Chan HP, Samala RK, Hadjiiski LM, et al. Deep learning in medical image analysis [M]//Advances in Experimental Medicine and Biology. Cham: Springer International Publishing, 2020: 3-21.
    [2]
    李锡荣. 多模态深度学习及其在眼科人工智能的应用展望[J]. 协和医学杂志, 2021, 12(5): 602-7. https://www.cnki.com.cn/Article/CJFDTOTAL-XHYX202105002.htm
    [3]
    李伊宁, 王弘熠, 王天任, 等. 基于深度学习的多模态融合的临床应用[J]. 医学理论与实践, 2021, 34(10): 1654-5, 1662. https://www.cnki.com.cn/Article/CJFDTOTAL-YXLL202110013.htm
    [4]
    Yeh YR, Lin TC, Chung YY, et al. A novel multiple kernel learning framework for heterogeneous feature fusion and variable selection [J]. IEEE Trans Multimed, 2012, 14(3): 563-74. doi: 10.1109/TMM.2012.2188783
    [5]
    刘锦辉, 冷晓玲. 多模态超声联合深度学习对乳腺癌新辅助化疗疗效及侵袭性评价的研究进展[J]. 分子影像学杂志, 2021, 44(6): 1034-40. doi: 10.12122/j.issn.1674-4500.2021.06.30
    [6]
    Cun YL, Boser B, Denker J, et al. Handwritten digit recognition with a backpropogation network[C]. Advances in Neural Information Processing Systems. 1989: 396-404.
    [7]
    Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks[J]. Commun ACM, 2017, 60 (6): 84-90. doi: 10.1145/3065386
    [8]
    Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2015-04-10]. https://arxiv.org/abs/1409.1556
    [9]
    Szegedy C, Liu W, Jia YQ, et al. Going deeper with convolutions[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA. IEEE, 2015: 1-9.
    [10]
    He KM, Zhang XY, Ren SQ, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA. IEEE, 2016: 770-8.
    [11]
    Huang G, Liu Z, van der Maaten L, et al. Densely connected convolutional networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA. IEEE, 2017: 2261-9.
    [12]
    Ronneberger O, Fischer P, Brox T, et al. U-Net: Convolutional net works for biomedical image segmentation[C]. The 18th International Conference on Medical Image Computing and Computer Assisted Interventions(MICCAI). Munich: Springer, 2015: 234-41.
    [13]
    王彤, 何萍, 苏畅, 等. 计算机辅助多模态融合超声诊断乳腺良恶性肿瘤[J]. 中国医学影像技术, 2021, 37(8): 1210-3. https://www.cnki.com.cn/Article/CJFDTOTAL-ZYXX202108027.htm
    [14]
    Huang RB, Lin ZH, Dou HR, et al. AW3M: an auto-weighting and recovery framework for breast cancer diagnosis using multi-modal ultrasound[J]. Med Image Anal, 2021, 72: 102137. doi: 10.1016/j.media.2021.102137
    [15]
    Cai YL, Landis M, Laidley DT, et al. Multi-modal vertebrae recognition using Transformed Deep Convolution Network[J]. Comput Med Imaging Graph, 2016, 51: 11-9. doi: 10.1016/j.compmedimag.2016.02.002
    [16]
    钟霁媛, 陈思翰, 王晗. 基于深度神经网络多模态融合的颞叶内侧癫痫鉴别[J]. 现代计算机, 2019(19): 13-7, 40. doi: 10.3969/j.issn.1007-1423.2019.19.003
    [17]
    赵鑫, 强彦, 葛磊. 基于双模态深度自编码的孤立性肺结节诊断方法[J]. 计算机科学, 2017, 44(8): 312-7. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJA201708054.htm
    [18]
    Kitajima K, Matsuo H, Kono A, et al. Deep learning with deep convolutional neural network using FDG-PET/CT for malignant pleural mesothelioma diagnosis[J]. Oncotarget, 2021, 12(12): 1187-96. doi: 10.18632/oncotarget.27979
    [19]
    Xu LN, Tetteh G, Lipkova J, et al. Automated whole-body bone lesion detection for multiple myeloma on 68Ga-pentixafor PET/CT imaging using deep learning methods[J]. Contrast Media & Mol Imaging, 2018, 2018: 2391925.
    [20]
    Havaei M, Davy A, Warde-Farley D, et al. Brain tumor segmentation with deep neural networks[J]. Med Image Anal, 2017, 35: 18-31. doi: 10.1016/j.media.2016.05.004
    [21]
    Ma SQ, Tang JJ, Guo F. Multi-task deep supervision on attention R2U-net for brain tumor segmentation[J]. Front Oncol, 2021, 11: 704850. doi: 10.3389/fonc.2021.704850
    [22]
    Budak Ü, Guo YH, Tanyildizi E, et al. Cascaded deep convolutional encoder-decoder neural networks for efficient liver tumor segmentation[J]. Med Hypotheses, 2020, 134: 109431. doi: 10.1016/j.mehy.2019.109431
    [23]
    Meng L, Zhang QQ, Bu SH. Two-stage liver and tumor segmentation algorithm based on convolutional neural network[J]. Diagnostics, 2021, 11(10): 1806. doi: 10.3390/diagnostics11101806
    [24]
    Wardhana G, Naghibi H, Sirmacek B, et al. Toward reliable automatic liver and tumor segmentation using convolutional neural network based on 2.5D models[J]. Int J Comput Assist Radiol Surg, 2021, 16(1): 41-51. doi: 10.1007/s11548-020-02292-y
    [25]
    Momin S, Lei Y, Tian Z, et al. Lung tumor segmentation in 4D CT images using motion convolutional neural networks[J]. Med Phys, 2021, 48(11): 7141-53. doi: 10.1002/mp.15204
    [26]
    Yin XX, Jian YX, Zhang Y, et al. Automatic breast tissue segmentation in MRIs with morphology snake and deep denoiser training via extended Stein's unbiased risk estimator[J]. Heal Inf Sci Syst, 2021, 9(1): 1-21. doi: 10.1007/s13755-020-00123-7
    [27]
    Baccouche A, Garcia-Zapirain B, Castillo Olea C, et al. Connected-UNets: a deep learning architecture for breast mass segmentation[J]. Npj Breast Cancer, 2021, 7: 151. doi: 10.1038/s41523-021-00358-x
    [28]
    Qi YX, Li JY, Chen H, et al. Computer-aided diagnosis and regional segmentation of nasopharyngeal carcinoma based on multi-modality medical images[J]. Int J Comput Assist Radiol Surg, 2021, 16(6): 871-82. doi: 10.1007/s11548-021-02351-y
    [29]
    凌彤, 杨琬琪, 杨明. 利用多模态U形网络的CT图像前列腺分割[J]. 智能系统学报, 2018, 13(6): 981-8. https://www.cnki.com.cn/Article/CJFDTOTAL-ZNXT201806018.htm
    [30]
    董阳, 潘海为, 崔倩娜, 等. 面向多模态磁共振脑瘤图像的小样本分割方法[J]. 计算机应用, 2021, 41(4): 1049-54. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJY202104020.htm
    [31]
    Pei LM, Vidyaratne L, Rahman MM, et al. Context aware deep learning for brain tumor segmentation, subtype classification, and survival prediction using radiology images[J]. Sci Rep, 2020, 10: 19726. doi: 10.1038/s41598-020-74419-9
    [32]
    Vesal S, Gu MX, Kosti R, et al. Adapt everywhere: unsupervised adaptation of point-clouds and entropy minimization for multi-modal cardiac image segmentation[J]. IEEE Trans Med Imaging, 2021, 40(7): 1838-51. doi: 10.1109/TMI.2021.3066683
    [33]
    Montaña-Brown N, Ramalhinho J, Allam M, et al. Vessel segmentation for automatic registration of untracked laparoscopic ultrasound to CT of the liver[J]. Int J Comput Assist Radiol Surg, 2021, 16(7): 1151-60. doi: 10.1007/s11548-021-02400-6
    [34]
    Zhong ZS, Kim Y, Plichta K, et al. Simultaneous cosegmentation of tumors in PET-CT images using deep fully convolutional networks [J]. Med Phys, 2019, 46(2): 619-33. doi: 10.1002/mp.13331
    [35]
    Guo Z, Li X, Huang H, et al. Deep learning-based image segmentation on multimodal medical imaging[J]. IEEE Trans Radiat Plasma Med Sci, 2019, 3(2): 162-9. doi: 10.1109/TRPMS.2018.2890359
    [36]
    Rahaman MA, Chen JY, Fu ZN, et al. Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness[C]//2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society. November 1-5, 2021, Mexico. IEEE, 2021: 3267-72.
    [37]
    韩坤, 潘海为, 张伟, 等. 基于多模态医学图像的Alzheimer病分类方法[J]. 清华大学学报: 自然科学版, 2020, 60(8): 664-71, 682. https://www.cnki.com.cn/Article/CJFDTOTAL-QHXB202008006.htm
    [38]
    Mitrea D, Badea R, Mitrea P, et al. Hepatocellular carcinoma automatic diagnosis within CEUS and B-mode ultrasound images using advanced machine learning methods[J]. Sensors, 2021, 21(6): 2202. doi: 10.3390/s21062202
    [39]
    Zhang Q, Xiong JY, Cai YH, et al. Multimodal feature learning and fusion on B-mode ultrasonography and sonoelastography using point-wise gated deep networks for prostate cancer diagnosis[J]. Biomedizinische Tech, 2020, 65(1): 87-98. doi: 10.1515/bmt-2018-0136
    [40]
    柯艺雅, 周小波. 基于深度学习的多模态骨癌影像分类诊断系统研究[J]. 信息与电脑, 2021(6): 136-8. doi: 10.3969/j.issn.1003-9767.2021.06.040
    [41]
    代广喆. 基于深度学习的PET/CT肾癌分类方法研究[D]. 哈尔滨: 哈尔滨工业大学, 2020.
    [42]
    Chang YJ, Huang TY, Liu YJ, et al. Classification of parotid gland tumors by using multimodal MRI and deep learning[J]. NMR Biomed, 2021, 34(1): e4408.
    [43]
    杨舜翔. 基于多模态医学影像的疾病早筛方法研究[D]. 济南: 山东大学, 2021.
    [44]
    Capobianco N, Meignan M, Cottereau AS, et al. Deep-learning 18F-FDG uptake classification enables total metabolic tumor volume estimation in diffuse large B-cell lymphoma[J]. J Nucl Med, 2021, 62(1): 30-6. doi: 10.2967/jnumed.120.242412
    [45]
    Zhang L, Lu L, Summers RM, et al. Convolutional invasion and expansion networks for tumor growth prediction[J]. IEEE Trans Med Imaging, 2017, 37(2): 638-48.
    [46]
    Nie D, Lu JF, Zhang H, et al. Multi-channel 3D deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages[J]. Sci Rep, 2019, 9: 1103. doi: 10.1038/s41598-018-37387-9
    [47]
    Teo KY, Daescu O, Cederberg K, et al. Correlation of histopathology and multi-modal magnetic resonance imaging in childhood osteosarcoma: predicting tumor response to chemotherapy[J]. PLoS One, 2022, 17(2): e0259564. doi: 10.1371/journal.pone.0259564
    [48]
    Wu XL, Li MY, Cui XW, et al. Deep multimodal learning for lymph node metastasis prediction of primary thyroid cancer[J]. Phys Med Biol, 2022, 67(3): 035008. doi: 10.1088/1361-6560/ac4c47
    [49]
    Le WT, Vorontsov E, Romero FP, et al. Cross-institutional outcome prediction for head and neck cancer patients using self-attention neural networks[J]. Sci Rep, 2022, 12: 3183. doi: 10.1038/s41598-022-07034-5
    [50]
    Li HM, Boimel P, Janopaul-Naylor J, et al. Deep convolutional neural networks for imaging data based survival analysis of rectal cancer[C]//2019 IEEE 16th International Symposium on Biomedical Imaging. Venice, Italy. IEEE, 2019: 846-9.
    • Relative Articles
    • Supplements(0)
    • Cited By
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索
    Get Citation
    PDF
    XML

    Article Metrics

    Article views (1919) PDF downloads(330) Cited by()
    Proportional views
    Related
    • Copyright © 2013《Journal of Molecular Imaging》 Editorial Office ICP备00000000号
    • Key tip: Adobe Reader6 The full text may be displayed abnormally，Please upgrade to Adobe Reader7Or higher .
    • Address: 1838 North Guangzhou Dadao, Guangzhou City, Guangdong Province, China Tel: 020-61648176 E-mail: 

    Supported by: Beijing Renhe Information Technology Co. Ltd  support: info@rhhz.net

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return