Breast Screening: On the Use of Multi-Modality in Medical Imaging Diagnosis

乳腺癌筛查：医学影像诊断中多模态技术的应用

ABSTRACT

摘要

This paper describes the field research, design and comparative deployment of a multimodal medical imaging user interface for breast screening. The main contributions described here are threefold: 1) The design of an advanced visual interface for multimodal diagnosis of breast cancer (Breast Screening); 2) Insights from the field comparison of Single-Modality vs Multi-Modality screening of breast cancer diagnosis with 31 clinicians and 566 images; and 3) The visu aliz ation of the two main types of breast lesions in the following image modalities: (i) MammoGraphy (MG) in both Cr a nio caudal (CC) and Medio lateral oblique (MLO) views; (ii) UltraSound (US); and (iii) Magnetic Resonance Imaging (MRI). We summarize our work with recommendations from the radiologists for guiding the future design of medical imaging interfaces.

本文介绍了针对乳腺筛查的多模态医学影像用户界面的实地研究、设计及对比部署。主要贡献包含三方面：1) 乳腺癌多模态诊断高级可视化界面设计（乳腺筛查）；2) 通过31名临床医师对566张影像的单模态与多模态乳腺癌诊断对比获得的实地研究洞见；3) 以下影像模态中两种主要乳腺病变的可视化呈现：(i) 乳腺X线摄影（MG）的头尾位（CC）与内外斜位（MLO）视图；(ii) 超声（US）；(iii) 磁共振成像（MRI）。我们结合放射科医师提出的建议对工作进行了总结，以指导未来医学影像界面的设计。

CCS CONCEPTS

CCS概念

• Human-centered computing $\longrightarrow$ User studies; Usability testing; Interaction techniques; User centered design; User interface design.

• 以人为中心的计算 (Human-centered computing) $\longrightarrow$ 用户研究；可用性测试；交互技术；以用户为中心的设计；用户界面设计。

KEYWORDS

关键词

human-computer interaction, user-centered design, multimodality, healthcare systems, medical imaging, breast cancer, annotations

人机交互、以用户为中心的设计、多模态、医疗系统、医学影像、乳腺癌、标注

ACM Reference Format:

ACM 参考文献格式:

Francisco Maria Calisto, Nuno Nunes, and Jacinto C. Nascimento. 2020. Breast Screening: On the Use of Multi-Modality in Medical Imaging Diagnosis. In International Conference on Advanced Visual Interfaces (AVI ’20), September 28-October 2, 2020, Salerno, Italy. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3399715.3399744

Francisco Maria Calisto、Nuno Nunes 和 Jacinto C. Nascimento。2020。乳腺筛查：多模态在医学影像诊断中的应用。见：国际高级视觉界面会议 (AVI '20)，2020年9月28日-10月2日，意大利萨勒诺。ACM，美国纽约州纽约市，5页。https://doi.org/10.1145/3399715.3399744

1 INTRODUCTION

1 引言

Breast cancer is the most common cancer in women worldwide [12]. Screening plays a fundamental role in the reduction of patient mortality rate. The most widely employed image modality for breast screening is MammoGraphy (MG). However, high-risk or dense breast patients require UltraSound (US) or Magnetic Resonance Imaging (MRI) for proper examination [18]. Therefore, it is quite rare to conduct screening using a Single-Modality.

乳腺癌是全球女性最常见的癌症 [12]。筛查在降低患者死亡率方面发挥着重要作用。目前乳腺筛查最广泛采用的成像方式是乳腺X线摄影 (MG)。但对于高风险或致密型乳腺患者，需通过超声 (US) 或磁共振成像 (MRI) 进行准确检查 [18]。因此，采用单一模态进行筛查的情况较为罕见。

In this paper, we describe the design and comparative testing of Breast Screening integrating information from several and different image modalities. We tested the design of Breast Screening with 31 clinicians noting that the time spent per each image on a Multi-Modality strategy is reduced when compared with the SingleModality scenario. In addition, the lesion classification (e.g., Breast Imaging Reporting and Data System - BIRADS [29]) is also reduced from our Multi-Modality proposed approach.

本文介绍了整合多种不同影像模态信息的乳腺筛查设计方案及对比测试结果。我们邀请31名临床医生对乳腺筛查方案进行测试，发现与单一模态策略相比，多模态策略下每幅图像的阅片时间有所减少。此外，通过我们提出的多模态方法，病灶分类（如乳腺影像报告和数据系统(BIRADS) [29]）所需时间也得到降低。

1.1 Breast Screening Challenges

1.1 乳腺筛查的挑战

Overall the system involves the following functionalities: (1) an interface for identifying (and annotating ground truth) of two types of lesions (i.e., masses and calcification s) across image modalities; (2) support for categorization of the breast tissues (dense vs nondense); (3) a classification (and recommendation) schema for lesion severity using BIRADS [1, 29]; (4) prompt access to clinical covariables, such as personal and familiar records; and (5) proper visualization s for a follow-up diagnosis of the patients.

该系统整体包含以下功能：(1) 跨影像模态识别（及标注真实值）两种病灶类型（即肿块和钙化）的交互界面；(2) 支持乳腺组织分类（致密型与非致密型）；(3) 基于BIRADS [1,29]的病灶严重程度分类（及推荐）方案；(4) 快速调取临床协变量（如个人及家族病史记录）；(5) 为患者随访诊断提供的规范化可视化功能。

1.2 Design Process

1.2 设计流程

The following topics summarize the process we conducted: (1) findings from a formative study with 31 clinicians, comprising Radiology Room (RR) observations and interviews, which are relevant for both Health Informatics (HI) and Human-Computer Interaction (HCI) fields of research. This leads us to explore the design goals (see Section 3); (2) findings from an evaluation study [6] of Breast Screening, a prototype we developed for the generation of a breast dataset with expert annotations (see Section 4); and (3) design recommendations for the use of visualization s to support medical imaging diagnosis (see Sections 4 and 5).

我们开展的过程可概括为以下主题：(1) 对31名临床医生进行的形成性研究结果，包括放射科(RR)观察和访谈，这些发现与健康信息学(HI)和人机交互(HCI)研究领域均相关。这引导我们探索设计目标(见第3节)；(2) 对我们开发的乳腺筛查原型[6]的评估研究结果，该原型用于生成带有专家标注的乳腺数据集(见第4节)；(3) 支持医学影像诊断的可视化应用设计建议(见第4节和第5节)。

1.3 Contributions

1.3 贡献

In Breast Screening we provide several new insights, following novel interaction and visualization paradigms [23] in the context of breast cancer screening: (i) multimodal interaction; (ii) indistinct visualization of cluttered lesions; (iii) big data management platform; and $(i v)$ clinicians’ multi-screen, multi-environment interaction.

在乳腺筛查中，我们基于乳腺癌筛查场景下的新型交互与可视化范式[23]，提出了多项创新见解：(i) 多模态交互；(ii) 杂乱病灶的模糊可视化；(iii) 大数据管理平台；以及$(iv)$临床医生的多屏幕、多环境交互。

2 RELATED WORK

2 相关工作

This section addresses related work in the HCI field, describing several medical imaging applications. Our approach covers the limitations of the works following described. More specifically, we are able to deal with non-homogeneous data. Comprising multimodal images [38], classification (i.e., BIRADS scores) and annotations (i.e., delineation of the lesion contours).

本节探讨了人机交互 (HCI) 领域的相关工作，描述了几种医学影像应用。我们的方法解决了后续所述工作的局限性，具体而言，能够处理非均匀数据，包括多模态图像 [38]、分类 (如 BIRADS 评分) 和标注 (如病灶轮廓勾画)。

2.1 Data Visualization

2.1 数据可视化

To our knowledge, few papers [15, 16, 24] have focused purely on supporting the image search user experience through novel UIs. These authors described several techniques for presenting all images within a collection in a short time. Moreover, authors asked users to think and perform browsing an image gallery and selecting an image from the gallery. These studies, showed us refinement techniques as complements in image systems with relevant user feedback. However, the presented works are limited to non-clinical users, making it impossible to do a generalization to our research.

据我们所知，仅有少数论文 [15, 16, 24] 专注于通过新颖的用户界面 (UI) 来提升图像搜索体验。这些作者提出了多种在短时间内展示整个图像集的技术，并要求用户通过浏览图库并从中选择图像来完成思考与操作。这些研究向我们展示了结合用户反馈的图像系统优化技术，但其研究对象仅限于非临床用户，因此无法直接推广到我们的研究领域。

2.2 Clinical Workflow

2.2 临床工作流程

In medical imaging, diagnostic tools enable clinicians to manage patient data, better attend to ongoing tasks and view critical information. For the diagnostic, understanding the clinical workflow is of chief importance while introducing novel tools and interaction techniques. Other authors [11, 30] present many considerations for collaborative healthcare technology design and discuss the implications of their findings on the current clinical workflow for the development of more effective care interventions. Supported by the above literature, our goal is to introduce a new tool with several novel interaction techniques, which will improve the final medical imaging diagnosis.

在医学影像领域，诊断工具能帮助临床医生管理患者数据、更高效处理当前任务并查看关键信息。引入新型工具和交互技术时，理解临床工作流程至关重要。其他研究者 [11, 30] 提出了协同医疗技术设计的诸多考量，并探讨了其发现对现有临床工作流程的影响，以开发更有效的护理干预措施。基于上述文献支持，我们的目标是推出一款具备多项创新交互技术的新工具，从而提升医学影像诊断的最终效果。

2.3 Medical Imaging

2.3 医学影像

From current medical imaging technologies, several issues were identified in the HCI design [5, 7, 13]. Some works [2, 25] show the current medical imaging identification techniques for other clinical domains, where most of available systems fail to address the visual nature of the task. In these two works [2, 25], the authors create a visual approach to support the Mental Model development of the user. Medical imaging technologies are used to support physicians on the examination, diagnosis, and (in some cases) report [37]. Others [10, 28, 33], study the effectiveness and performance of medical imaging systems, demonstrating how to design a user study for medical imaging experts. Further, van Schooten et al. [33] measured user performance in terms of time taken and error rate, while interacting with the provided system. Executing it with several medical users, in this work, the authors show an experiment where their users have similar characteristics as ours.

从现有医学成像技术中，研究人员在HCI(人机交互)设计领域发现了若干问题[5,7,13]。部分研究[2,25]展示了针对其他临床领域的医学影像识别技术现状，指出现有系统大多未能解决任务的视觉特性。在这两项研究[2,25]中，作者创建了可视化方法来支持用户心智模型( Mental Model )的构建。医学成像技术主要用于辅助医生进行检查、诊断及(部分情况下)报告撰写[37]。另有研究[10,28,33]聚焦医学影像系统的效能与性能评估，阐述了如何为医学影像专家设计用户研究。此外，van Schooten等人[33]通过测量用户与系统交互时耗费的时间及错误率来评估用户表现。该研究通过对多名医疗用户进行测试，展示了实验对象特征与本研究用户群体具有相似性。

2.4 Diagnostic Systems

2.4 诊断系统

Medical imaging has also been extensively studied under the topic of Computer-Aided Diagnosis $(C A D x)_{*}$ , which refers to systems that assist radiologists in image interpretation [4, 22]. Wilcox et al. [35] propose a design for in-room, patient-centric information displays, based on iterative design with clinicians. However, these systems are not contemplating the design of an advanced visual interface for multimodal diagnosis on breast cancer disease. In the above works, we still lack on empirical studies regarding how clinicians can contribute with information contextual iz ation about their clinical workflow, and general medical imaging diagnosis. Having said that, we also want to add contribution with a study of how medical imaging technologies can play a role in this contextual iz ation.

计算机辅助诊断 (CADx) 领域对医学影像进行了广泛研究 [4, 22]。Wilcox 等人 [35] 提出了一种基于临床医生迭代设计的病房内以患者为中心的信息显示方案。然而这些系统并未考虑为乳腺癌多模态诊断设计高级可视化界面。上述研究中，我们仍缺乏关于临床医生如何通过临床工作流信息情境化和常规医学影像诊断做出贡献的实证研究。有鉴于此，我们还希望通过研究医学影像技术如何促进这种情境化来做出额外贡献。

3 DESIGN OF BREAST SCREENING

3 乳腺筛查设计

The design of Breast Screening started with a qualitative study to understand radiology practices and workflow in the context of breast screening. Our study involved 31 clinicians, recruited on a volunteer basis from a large range of clinical scenarios (distinct health institutions in Portugal): 8 clinicians from Hospital Fernando Fonseca; 12 clinicians from IPO-Lisboa; 1 clinician from Hospital de Santa Maria; 8 clinicians from IPO-Coimbra; 1 clinician from Madeira Medical Center; and 1 clinician from SAMS. Clinicians’ experience ranged from 5 - 30 years of medical practice. The recruited specialists are in advanced career positions and were observed and interviewed in a semi-structured fashion. Each session took approximately 30 minutes.

乳腺筛查的设计始于一项定性研究，旨在理解乳腺筛查背景下的放射学实践和工作流程。我们的研究涉及31名临床医生，他们来自葡萄牙各类医疗机构（不同健康机构）并自愿参与：8名来自Hospital Fernando Fonseca；12名来自IPO-Lisboa；1名来自Hospital de Santa Maria；8名来自IPO-Coimbra；1名来自Madeira Medical Center；以及1名来自SAMS。这些临床医生的从业经验在5至30年之间。招募的专家均处于职业发展的高级阶段，并以半结构化方式接受观察和访谈。每次会话时长约30分钟。

3.1 Standard Clinical Environments

3.1 标准临床环境

Breast Screening works with the standard formats supported by medical imaging [21], including the MG, US and MRI modalities. These modalities are available in a standard Digital Imaging and Communications in Medicine (DICOM) format and supported in Single-Modality by existing systems [12]. Moreover, most systems are general purpose and do not adapt to specific clinical domains (e.g., breast screening). Therefore they do not provide adequate support to the different clinical workflows [7].

乳腺筛查支持医学影像的标准格式[21]，包括MG（乳腺X线摄影）、US（超声）和MRI（磁共振成像）模态。这些模态以标准的DICOM（医学数字成像与通信）格式提供，并被现有系统以单模态形式支持[12]。此外，大多数系统是通用的，并未针对特定临床领域（如乳腺筛查）进行适配，因此无法为不同的临床工作流程提供充分支持[7]。

3.2 Design Goals

3.2 设计目标

Combining the clinical context and the technical design challenges lead to a set of design issues, including: medical imaging structure trade-offs, RR temporal awareness, image segmentation [20], and radiologists system trust. Based on these, we define five design goals:

结合临床背景与技术设计挑战，我们总结出以下关键设计问题：医学影像结构权衡、RR时序感知、图像分割[20]以及放射科医生系统信任度。基于这些问题，我们确立了五项设计目标：

4 BREAST SCREENING

4 乳腺筛查

To validate the proposed design goals, we created Breast Screening, as a Medical Imaging visualization proof-of-concept to be evaluated in a realistic clinical scenario. In our design explorations, we sought to integrate several image modalities and visualization to support insight.

为验证所提出的设计目标，我们开发了乳腺筛查系统作为医学影像可视化概念验证，并在真实临床场景中进行评估。在设计探索过程中，我们尝试整合多种影像模态和可视化技术以支持诊断洞察。

Figure 1: Single-Modality (left) and Multi-Modality (right) Views. The UI components are as follows: 4. List of Patient Views; and 4.5. Study List Tabs; as well as 5. Medical Imaging Diagnosis Views; 5.1. Viewports; 5.2. Toolbars; and 5.3. Modality Selection.

图 1: 单模态(左)与多模态(右)视图。界面组件包括: 4. 患者视图列表; 4.5. 研究列表标签页; 5. 医学影像诊断视图; 5.1. 视窗; 5.2. 工具栏; 5.3. 模态选择。

4.1 User Interface

4.1 用户界面

The User Interface (UI) consists of two main components: 4. List of Patient Views; and 5. Medical Imaging Diagnosis Views. These two main components (Figure 1) are also divided into several sections: 4.5. Study List Tabs; 5.1. Viewports; 5.2.