Situativity, functionality and trust: Results of a scenario-based interview study on the explainability of AI in medicine




explainability, XAI, AI in healthcare, embodied AI, voice dialog system


A central requirement for the use of artificial intelligence (AI) in medicine is its explainability, i. e., the provision of addressee-oriented information about its functioning. This leads to the question of how socially adequate explainability can be designed. To identify evaluation factors, we interviewed healthcare stakeholders about two scenarios: diagnostics and documentation. The scenarios vary the influence that an AI system has on decision-making through the interaction design and the amount of data processed. We present key evaluation factors for explainability at the interactional and procedural levels. Explainability must not interfere situationally in the doctor-patient conversation and question the professional role. At the same time, explainability functionally legitimizes an AI system as a second opinion and is central to building trust. A virtual embodiment of the AI system is advantageous for language-based explanations


Aminololama-Shakeri, Shadi; López, Javier (2019): The doctor-patient relationship with artificial intelligence. In: American Journal of Roentgenology 212 (2), S. 308–310. DOI:

Barredo-Arrieta, Alejandro et al. (2020): Explainable artificial intelligence (XAI). Concepts, taxonomies, opportunities and challenges toward responsible AI. In: Information Fusion 58, S. 82–115. DOI:

Barter, Christine; Renold, Emma (1999): The use of vignettes in qualitative research. In: Social Research Update 25. Online verfügbar unter, zuletzt geprüft am 11. 01. 2024.

Becker, Aliza (2019): Artificial intelligence in medicine. What is it doing for us today? In: Health Policy and Technology 8 (2), S. 198–205. DOI:

Bossen, Claus; Pine, Kathleen (2023): Batman and Robin in healthcare knowledge work. Human-AI collaboration by clinical documentation integrity specialists. In: ACM Transactions on Computer-Human Interaction 30 (2), S. 1–29. DOI:

Čartolovni, Anto; Tomičić, Ana; Lazić Mosler, Elvira (2022): Ethical, legal, and social considerations of AI‑based medical decision-support tools. A scoping review. In: International Journal of Medical Informatics 161, S. 104738. DOI:

Dickel, Sascha (2021): Wenn die Technik sprechen lernt. Künstliche Kommunikation als kulturelle Herausforderung mediatisierter Gesellschaften. In: TATuP – Zeitschrift für Technikfolgenabschätzung in Theorie und Praxis 30 (3), S. 23–29. DOI:

Feldhus, Nils; Wang, Qianli; Anikina, Tatiana; Chopra, Sahil; Oguz, Cennet; Möller, Sebastian (2023): InterroLang. Exploring NLP models and datasets through dialogue-based explanations. In: Houda Bouamor, Juan Pino und Kalika Bali (Hg.): Findings of the Association for Computational Linguistics: EMNLP 2023. Singapur: Association for Computational Linguistics, S. 5399–5421. DOI:

Gupta, Akshit; Basu, Debadeep; Ghantasala, Ramya; Qiu, Sihang; Gadiraju, Ujwal (2022): To trust or not to trust. How a conversational interface affects trust in a decision support system. In: Frédérique Laforest et al. (Hg.): Proceedings of the ACM Web Conference 2022. New York, NY: Association for Computing Machinery, S. 3531–3540. DOI:

HEG-KI – Hochrangige Expertengruppe für KI (2019): Ethik-Leitlinien für eine vertrauenswürdige KI. Brüssel: Europäische Kommission.

Heil, Reinhard et al. (2021): Artificial intelligence in human genomics and biomedicine. Dynamics, potentials and challenges. In: TATuP – Zeitschrift für Technikfolgenabschätzung in Theorie und Praxis 30 (3), S. 30–36. DOI:

Hillmann, Stefan; Möller, Sebastian; Michael, Thilo (2021): Towards speech-based interactive post hoc explanations in explainable AI. In: Astrid Carolus, Carolin Wienrich und Ingo Siegert (Hg.): Proceedings of the 1st AI‑Debate Workshop. Magdeburg: Universität Magdeburg, pp. 13–15.

Hughes, Rhidian; Huby, Meg (2004): The construction and interpretation of vignettes in social research. In: Social Work and Social Sciences Review 11 (1), S. 36–51. DOI:

Jenkins, Nicholas; Bloor, Michael; Fischer, Jan; Berney, Lee; Neale, Joanne (2010): Putting it in context. The use of vignettes in qualitative interviewing. In: Qualitative Research 10 (2), S. 175–198. DOI:

Kinar, Yaron et al. (2017): Performance analysis of a machine learning flagging system used to identify a group of individuals at a high risk for colorectal cancer. In: PLoS One 12 (2), S. e0171759. DOI:

Kloker, Anika; Fleiß, Jürgen; Koeth, Christoph; Kloiber, Thomas; Ratheiser, Patrick; Thalmann, Stefan (2022): Caution or trust in AI? How to design XAI in sensitive use cases? In: AMCIS 2022 Proceedings 16.

Kosow, Hannah; Gaßner, Robert (2008): Methoden der Zukunfts- und Szenarioanalyse. Überblick, Bewertung und Auswahlkriterien. Berlin: Institut für Zukunftsstudien und Technologiebewertung. Online verfügbar unter, zuletzt geprüft am 15. 01. 2024.

Mahmood, Amama; Huang, Chien-Ming (2022): Effects of rhetorical strategies and skin tones on agent persuasiveness in assisted decision-making. In: Carlos Martinho, João Dias, Joana Campos und Dirk Heylen (Hg.): Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents. New York, NY: Association for Computing Machinery, S. 1–8. DOI:

Markus, Aniek; Kors, Jan; Rijnbeek, Peter (2021): The role of explainability in creating trustworthy artificial intelligence for health care. A comprehensive survey of the terminology, design choices, and evaluation strategies. In: Journal of Biomedical Informatics 113, S. 103655. DOI:

Meuser, Michael; Sackmann, Reinhold (1992): Zur Einführung. Deutungsmusteransatz und empirische Wissenssoziologie. In: Michael Meuser und Reinhold Sackmann (Hg.): Analyse sozialer Deutungsmuster. Beiträge zur empirischen Wissenssoziologie. Pfaffenweiler: Centaurus-Verlagsgesellschaft, S. 9–37.

Miller, Tim (2019): Explanation in artificial intelligence. Insights from the social sciences. In: Artificial Intelligence 267, S. 1–38. DOI:

Panch, Trishan; Mattie, Heather; Atun, Rifat (2019): Artificial intelligence and algorithmic bias. Implications for health systems. In: Journal of Global Health 9 (2), S. 010318. DOI:

Pentzold, Christian; Bischof, Andreas; Heise, Nele (2018): Einleitung. Theoriegenerierendes empirisches Forschen in medienbezogenen Lebenswelten. In: Christian Pentzold, Andreas Bischof und Nele Heise (Hg.): Praxis grounded theory. Theoriegenerierendes empirisches Forschen in medienbezogenen Lebenswelten. Wiesbaden: Springer, S. 1–24. DOI:

Powell, John (2019): Trust me, I’m a chatbot. How artificial intelligence in health care fails the Turing Test. In: Journal of Medical Internet Research 21 (10), S. e16222. DOI:

Samhammer, David et al. (2023): Klinische Entscheidungsfindung mit Künstlicher Intelligenz. Ein interdisziplinärer Governance-Ansatz. Heidelberg: Springer. DOI:

Scharowski, Nicolas; Perrig, Sebastian; Svab, Melanie; Opwis, Klaus; Brühlmann, Florian (2023): Exploring the effects of human-centered AI explanations on trust and reliance. In: Frontiers in Computer Science 5, S. 1–15. DOI:



How to Cite

Marquardt M, Graf P, Jansen E, Hillmann S, Voigt-Antons J-N. Situativity, functionality and trust: Results of a scenario-based interview study on the explainability of AI in medicine. TATuP [Internet]. 2024 Mar. 15 [cited 2024 Apr. 22];33(1):41-7. Available from: