Cross Encoder-Decoder Transformer with Global-Local Visual Extractor for Medical Image Captioning

Hojun Lee, Hyunjun Cho, Jieun Park, Jinyeong Chae, Jihie Kim

Research output: Contribution to journalArticlepeer-review

10 Scopus citations

Abstract

Transformer-based approaches have shown good results in image captioning tasks. How-ever, current approaches have a limitation in generating text from global features of an entire image. Therefore, we propose novel methods for generating better image captioning as follows: (1) The Global-Local Visual Extractor (GLVE) to capture both global features and local features. (2) The Cross Encoder-Decoder Transformer (CEDT) for injecting multiple-level encoder features into the decoding process. GLVE extracts not only global visual features that can be obtained from an entire image, such as size of organ or bone structure, but also local visual features that can be generated from a local region, such as lesion area. Given an image, CEDT can create a detailed description of the overall features by injecting both low-level and high-level encoder outputs into the decoder. Each method contributes to performance improvement and generates a description such as organ size and bone structure. The proposed model was evaluated on the IU X-ray dataset and achieved better performance than the transformer-based baseline results, by 5.6% in BLEU score, by 0.56% in METEOR, and by 1.98% in ROUGE-L.

Original languageEnglish
Article number1429
JournalSensors
Volume22
Issue number4
DOIs
StatePublished - 1 Feb 2022

Keywords

  • Deep learning
  • Medical image captioning
  • Transformer

Fingerprint

Dive into the research topics of 'Cross Encoder-Decoder Transformer with Global-Local Visual Extractor for Medical Image Captioning'. Together they form a unique fingerprint.

Cite this