留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码
x

基于深度学习的多模态医学影像研究进展

王宗敏,  福林,  高云玥

downloadPDF
文章导航 > 分子影像学杂志  > 2022 >  45(3): 459-464
王宗敏, 福林, 高云玥. 基于深度学习的多模态医学影像研究进展[J]. 分子影像学杂志, 2022, 45(3): 459-464. doi: 10.12122/j.issn.1674-4500.2022.03.30
引用本文: 王宗敏, 福林, 高云玥. 基于深度学习的多模态医学影像研究进展[J]. 分子影像学杂志, 2022, 45(3): 459-464. doi: 10.12122/j.issn.1674-4500.2022.03.30
shu
WANG Zongmin, FU Lin, GAO Yunyue. Research progress of multimodal medical imaging based on deep learning[J]. Journal of Molecular Imaging, 2022, 45(3): 459-464. doi: 10.12122/j.issn.1674-4500.2022.03.30
Citation: WANG Zongmin, FU Lin, GAO Yunyue. Research progress of multimodal medical imaging based on deep learning[J]. Journal of Molecular Imaging, 2022, 45(3): 459-464. doi: 10.12122/j.issn.1674-4500.2022.03.30
shu

基于深度学习的多模态医学影像研究进展

doi: 10.12122/j.issn.1674-4500.2022.03.30

  • 内蒙古医科大学附属医院超声诊断科，内蒙古 呼和浩特 010000

基金项目: 

内蒙古医科大学科技百万工程联合项目 YKD2018KJBW(LH)036

详细信息
    作者简介:

    王宗敏，在读硕士研究生，E-mail: 1559768938@qq.com

    通讯作者:

    福林，主任医师，E-mail: fulin200907@163.com

Research progress of multimodal medical imaging based on deep learning

  • Department of Ultrasound Diagnosis, Affiliated Hospital of Inner Mongolia Medical University, Hohhot 010000, China

    摘要
  • 摘要: 凭借深度学习及大数据等技术的飞速发展，人工智能是医学领域最具发展前景的技术，鉴于医学影像对疾病的诊断与及时治疗的关键作用，医学影像与人工智能的结合正成为重要的交叉学科研究方向。在临床实践中，医生为了更精确全面的诊断疾病，往往需要同时参考多模态的影像数据进行综合分析和判断。本文首先介绍了多模态深度学习的基本概念和工作原理，对深度学习技术应用于多模态医学影像辅助诊断的代表性研究成果做出综述，分析了多模态深度学习在医学影像领域的技术挑战，并对该技术的应用前景作出展望。

     

    Abstract: With the rapid development of deep learning, big data and other technologies, artificial intelligence is the most promising technology in the field of medicine. In view of the key role of medical imaging in the diagnosis and timely treatment of diseases, the combination of medical imaging and artificial intelligence is becoming an essential interdisciplinary research direction. In clinical practice, doctors often need to refer to multimodal image data for comprehensive analysis and judgment in order to diagnose diseases more accurately and comprehensively. This paper first introduces the basic concept and working principle of multi-mode deep learning, and the representative research results of deep learning technology applied to multi-mode medical image-assisted diagnosis are summarized. Finally, the technical challenges of multi-mode deep learning technology in the medical image field are analyzed, and the application prospect of multi-mode deep learning technology is forecasted.

     

    • HTML全文
    • 参考文献(50)
  • [1] Chan HP, Samala RK, Hadjiiski LM, et al. Deep learning in medical image analysis [M]//Advances in Experimental Medicine and Biology. Cham: Springer International Publishing, 2020: 3-21.
    [2] 李锡荣. 多模态深度学习及其在眼科人工智能的应用展望[J]. 协和医学杂志, 2021, 12(5): 602-7. https://www.cnki.com.cn/Article/CJFDTOTAL-XHYX202105002.htm
    [3] 李伊宁, 王弘熠, 王天任, 等. 基于深度学习的多模态融合的临床应用[J]. 医学理论与实践, 2021, 34(10): 1654-5, 1662. https://www.cnki.com.cn/Article/CJFDTOTAL-YXLL202110013.htm
    [4] Yeh YR, Lin TC, Chung YY, et al. A novel multiple kernel learning framework for heterogeneous feature fusion and variable selection [J]. IEEE Trans Multimed, 2012, 14(3): 563-74. doi: 10.1109/TMM.2012.2188783
    [5] 刘锦辉, 冷晓玲. 多模态超声联合深度学习对乳腺癌新辅助化疗疗效及侵袭性评价的研究进展[J]. 分子影像学杂志, 2021, 44(6): 1034-40. doi: 10.12122/j.issn.1674-4500.2021.06.30
    [6] Cun YL, Boser B, Denker J, et al. Handwritten digit recognition with a backpropogation network[C]. Advances in Neural Information Processing Systems. 1989: 396-404.
    [7] Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks[J]. Commun ACM, 2017, 60 (6): 84-90. doi: 10.1145/3065386
    [8] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2015-04-10]. https://arxiv.org/abs/1409.1556
    [9] Szegedy C, Liu W, Jia YQ, et al. Going deeper with convolutions[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA. IEEE, 2015: 1-9.
    [10] He KM, Zhang XY, Ren SQ, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA. IEEE, 2016: 770-8.
    [11] Huang G, Liu Z, van der Maaten L, et al. Densely connected convolutional networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA. IEEE, 2017: 2261-9.
    [12] Ronneberger O, Fischer P, Brox T, et al. U-Net: Convolutional net works for biomedical image segmentation[C]. The 18th International Conference on Medical Image Computing and Computer Assisted Interventions(MICCAI). Munich: Springer, 2015: 234-41.
    [13] 王彤, 何萍, 苏畅, 等. 计算机辅助多模态融合超声诊断乳腺良恶性肿瘤[J]. 中国医学影像技术, 2021, 37(8): 1210-3. https://www.cnki.com.cn/Article/CJFDTOTAL-ZYXX202108027.htm
    [14] Huang RB, Lin ZH, Dou HR, et al. AW3M: an auto-weighting and recovery framework for breast cancer diagnosis using multi-modal ultrasound[J]. Med Image Anal, 2021, 72: 102137. doi: 10.1016/j.media.2021.102137
    [15] Cai YL, Landis M, Laidley DT, et al. Multi-modal vertebrae recognition using Transformed Deep Convolution Network[J]. Comput Med Imaging Graph, 2016, 51: 11-9. doi: 10.1016/j.compmedimag.2016.02.002
    [16] 钟霁媛, 陈思翰, 王晗. 基于深度神经网络多模态融合的颞叶内侧癫痫鉴别[J]. 现代计算机, 2019(19): 13-7, 40. doi: 10.3969/j.issn.1007-1423.2019.19.003
    [17] 赵鑫, 强彦, 葛磊. 基于双模态深度自编码的孤立性肺结节诊断方法[J]. 计算机科学, 2017, 44(8): 312-7. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJA201708054.htm
    [18] Kitajima K, Matsuo H, Kono A, et al. Deep learning with deep convolutional neural network using FDG-PET/CT for malignant pleural mesothelioma diagnosis[J]. Oncotarget, 2021, 12(12): 1187-96. doi: 10.18632/oncotarget.27979
    [19] Xu LN, Tetteh G, Lipkova J, et al. Automated whole-body bone lesion detection for multiple myeloma on 68Ga-pentixafor PET/CT imaging using deep learning methods[J]. Contrast Media & Mol Imaging, 2018, 2018: 2391925.
    [20] Havaei M, Davy A, Warde-Farley D, et al. Brain tumor segmentation with deep neural networks[J]. Med Image Anal, 2017, 35: 18-31. doi: 10.1016/j.media.2016.05.004
    [21] Ma SQ, Tang JJ, Guo F. Multi-task deep supervision on attention R2U-net for brain tumor segmentation[J]. Front Oncol, 2021, 11: 704850. doi: 10.3389/fonc.2021.704850
    [22] Budak Ü, Guo YH, Tanyildizi E, et al. Cascaded deep convolutional encoder-decoder neural networks for efficient liver tumor segmentation[J]. Med Hypotheses, 2020, 134: 109431. doi: 10.1016/j.mehy.2019.109431
    [23] Meng L, Zhang QQ, Bu SH. Two-stage liver and tumor segmentation algorithm based on convolutional neural network[J]. Diagnostics, 2021, 11(10): 1806. doi: 10.3390/diagnostics11101806
    [24] Wardhana G, Naghibi H, Sirmacek B, et al. Toward reliable automatic liver and tumor segmentation using convolutional neural network based on 2.5D models[J]. Int J Comput Assist Radiol Surg, 2021, 16(1): 41-51. doi: 10.1007/s11548-020-02292-y
    [25] Momin S, Lei Y, Tian Z, et al. Lung tumor segmentation in 4D CT images using motion convolutional neural networks[J]. Med Phys, 2021, 48(11): 7141-53. doi: 10.1002/mp.15204
    [26] Yin XX, Jian YX, Zhang Y, et al. Automatic breast tissue segmentation in MRIs with morphology snake and deep denoiser training via extended Stein's unbiased risk estimator[J]. Heal Inf Sci Syst, 2021, 9(1): 1-21. doi: 10.1007/s13755-020-00123-7
    [27] Baccouche A, Garcia-Zapirain B, Castillo Olea C, et al. Connected-UNets: a deep learning architecture for breast mass segmentation[J]. Npj Breast Cancer, 2021, 7: 151. doi: 10.1038/s41523-021-00358-x
    [28] Qi YX, Li JY, Chen H, et al. Computer-aided diagnosis and regional segmentation of nasopharyngeal carcinoma based on multi-modality medical images[J]. Int J Comput Assist Radiol Surg, 2021, 16(6): 871-82. doi: 10.1007/s11548-021-02351-y
    [29] 凌彤, 杨琬琪, 杨明. 利用多模态U形网络的CT图像前列腺分割[J]. 智能系统学报, 2018, 13(6): 981-8. https://www.cnki.com.cn/Article/CJFDTOTAL-ZNXT201806018.htm
    [30] 董阳, 潘海为, 崔倩娜, 等. 面向多模态磁共振脑瘤图像的小样本分割方法[J]. 计算机应用, 2021, 41(4): 1049-54. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJY202104020.htm
    [31] Pei LM, Vidyaratne L, Rahman MM, et al. Context aware deep learning for brain tumor segmentation, subtype classification, and survival prediction using radiology images[J]. Sci Rep, 2020, 10: 19726. doi: 10.1038/s41598-020-74419-9
    [32] Vesal S, Gu MX, Kosti R, et al. Adapt everywhere: unsupervised adaptation of point-clouds and entropy minimization for multi-modal cardiac image segmentation[J]. IEEE Trans Med Imaging, 2021, 40(7): 1838-51. doi: 10.1109/TMI.2021.3066683
    [33] Montaña-Brown N, Ramalhinho J, Allam M, et al. Vessel segmentation for automatic registration of untracked laparoscopic ultrasound to CT of the liver[J]. Int J Comput Assist Radiol Surg, 2021, 16(7): 1151-60. doi: 10.1007/s11548-021-02400-6
    [34] Zhong ZS, Kim Y, Plichta K, et al. Simultaneous cosegmentation of tumors in PET-CT images using deep fully convolutional networks [J]. Med Phys, 2019, 46(2): 619-33. doi: 10.1002/mp.13331
    [35] Guo Z, Li X, Huang H, et al. Deep learning-based image segmentation on multimodal medical imaging[J]. IEEE Trans Radiat Plasma Med Sci, 2019, 3(2): 162-9. doi: 10.1109/TRPMS.2018.2890359
    [36] Rahaman MA, Chen JY, Fu ZN, et al. Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness[C]//2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society. November 1-5, 2021, Mexico. IEEE, 2021: 3267-72.
    [37] 韩坤, 潘海为, 张伟, 等. 基于多模态医学图像的Alzheimer病分类方法[J]. 清华大学学报: 自然科学版, 2020, 60(8): 664-71, 682. https://www.cnki.com.cn/Article/CJFDTOTAL-QHXB202008006.htm
    [38] Mitrea D, Badea R, Mitrea P, et al. Hepatocellular carcinoma automatic diagnosis within CEUS and B-mode ultrasound images using advanced machine learning methods[J]. Sensors, 2021, 21(6): 2202. doi: 10.3390/s21062202
    [39] Zhang Q, Xiong JY, Cai YH, et al. Multimodal feature learning and fusion on B-mode ultrasonography and sonoelastography using point-wise gated deep networks for prostate cancer diagnosis[J]. Biomedizinische Tech, 2020, 65(1): 87-98. doi: 10.1515/bmt-2018-0136
    [40] 柯艺雅, 周小波. 基于深度学习的多模态骨癌影像分类诊断系统研究[J]. 信息与电脑, 2021(6): 136-8. doi: 10.3969/j.issn.1003-9767.2021.06.040
    [41] 代广喆. 基于深度学习的PET/CT肾癌分类方法研究[D]. 哈尔滨: 哈尔滨工业大学, 2020.
    [42] Chang YJ, Huang TY, Liu YJ, et al. Classification of parotid gland tumors by using multimodal MRI and deep learning[J]. NMR Biomed, 2021, 34(1): e4408.
    [43] 杨舜翔. 基于多模态医学影像的疾病早筛方法研究[D]. 济南: 山东大学, 2021.
    [44] Capobianco N, Meignan M, Cottereau AS, et al. Deep-learning 18F-FDG uptake classification enables total metabolic tumor volume estimation in diffuse large B-cell lymphoma[J]. J Nucl Med, 2021, 62(1): 30-6. doi: 10.2967/jnumed.120.242412
    [45] Zhang L, Lu L, Summers RM, et al. Convolutional invasion and expansion networks for tumor growth prediction[J]. IEEE Trans Med Imaging, 2017, 37(2): 638-48.
    [46] Nie D, Lu JF, Zhang H, et al. Multi-channel 3D deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages[J]. Sci Rep, 2019, 9: 1103. doi: 10.1038/s41598-018-37387-9
    [47] Teo KY, Daescu O, Cederberg K, et al. Correlation of histopathology and multi-modal magnetic resonance imaging in childhood osteosarcoma: predicting tumor response to chemotherapy[J]. PLoS One, 2022, 17(2): e0259564. doi: 10.1371/journal.pone.0259564
    [48] Wu XL, Li MY, Cui XW, et al. Deep multimodal learning for lymph node metastasis prediction of primary thyroid cancer[J]. Phys Med Biol, 2022, 67(3): 035008. doi: 10.1088/1361-6560/ac4c47
    [49] Le WT, Vorontsov E, Romero FP, et al. Cross-institutional outcome prediction for head and neck cancer patients using self-attention neural networks[J]. Sci Rep, 2022, 12: 3183. doi: 10.1038/s41598-022-07034-5
    [50] Li HM, Boimel P, Janopaul-Naylor J, et al. Deep convolutional neural networks for imaging data based survival analysis of rectal cancer[C]//2019 IEEE 16th International Symposium on Biomedical Imaging. Venice, Italy. IEEE, 2019: 846-9.
    • 相关文章
    • 施引文献
  • 资源附件(0)
  • 加载中
WeChat 点击查看大图
计量
  • 文章访问数:  1741
  • HTML全文浏览量:  295
  • PDF下载量:  304
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-02-24
  • 网络出版日期:  2022-06-21
  • 刊出日期:  2022-05-20

目录

    /

    返回文章
    返回
    • 版权所有© 2013《分子影像学杂志》编辑部 ICP备00000000号
    • 重点提示: Adobe Reader6可能造成全文显示不正常，请升级为Adobe Reader7或更高。
    • 地址: 广东省广州市广州大道北1838号 Tel: 020-61648176 E-mail: 

    本系统由 北京仁和汇智信息技术有限公司 开发  技术支持：info@rhhz.net

    x 关闭 永久关闭

    关于《分子影像学杂志》变更刊期通知

    各位专家、作者、读者：

    为了缩短出版时滞，促进科研成果的快速传播，我刊自2024年1月起，刊期由双月刊变更为月刊。本刊主要栏目有：基础研究、临床研究、技术方法、综述等。

    感谢各位专家、作者、读者长期以来对我刊的支持与厚爱！

    南方医科大学学报编辑部

    《分子影像学杂志》

    2023年12月27日