TY - JOUR AU - Li, Christy Y. AU - Liang, Xiaodan AU - Hu, Zhiting AU - Xing, Eric P. PY - 2019/07/17 Y2 - 2024/03/29 TI - Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation JF - Proceedings of the AAAI Conference on Artificial Intelligence JA - AAAI VL - 33 IS - 01 SE - AAAI Technical Track: Natural Language Processing DO - 10.1609/aaai.v33i01.33016666 UR - https://ojs.aaai.org/index.php/AAAI/article/view/4637 SP - 6666-6673 AB - <p>Generating long and semantic-coherent reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel <em>Knowledge-driven Encode, Retrieve, Paraphrase</em> (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical report generation. Specifically, KERP decomposes medical report generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an <em>Encode</em> module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a <em>Retrieve</em> module that retrieves text templates based on the detected abnormalities; and lastly, a <em>Paraphrase</em> module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit—Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.</p> ER -