Search

Benchmarking Large Language Models for Clinical Data Retrieval via FHIR: A Prompt-and-Feedback Baseline for Tool-Augmented Agentic Systems

Contribution type: article

Title: Benchmarking Large Language Models for Clinical Data Retrieval via FHIR: A Prompt-and-Feedback Baseline for Tool-Augmented Agentic Systems

Authors:

Johannes Schmidt, Nordakademie, Germany
Arne Ewald, Nordakademie, Germany

Keywords: Large Language Models (LLMs), Fast Healthcare Interoperability Resources (FHIR), Natural Language Processing (NLP), Healthcare Interoperability, Retrieval-Augmented Generation (RAG), Agentic Workflows

Abstract:

This paper evaluates the ability of Large Language Models (LLMs) to generate syntactically and semantically correct FHIR REST queries from natural language for retrieving medical data from Clinical Data Repositories (CDRs). The goal is to explore natural language interfaces that can improve clinical data access and interoperability across healthcare systems. Six experiments were conducted with nine LLMs, comparing baseline prompting against structured prompts, few-shot examples, and feedback-loops using HTTP error codes or messages. Results show that even without external tools, several models achieve high syntactic validity, with accuracy further improved by prompt-engineering and simple feedback mechanisms. However, semantic correctness remains challenging, in particular for medical codes, date logic, and site-specific conventions. Error analyses demonstrate where Retrieval-Augmented Generation (RAG), terminology services, and agentic repair could provide immediate gains, making this work a valuable prompt-centric baseline for the next generation of tool-augmented clinical query systems.

Publication Date: November 16, 2025

Presented during:

Dates: November 16, 2025 to November 20, 2025

Location: Nice / Saint-Laurent-du-Var, France

Venue:

Novotel Nice Airport Cap 3000

40 Avenue de Verdun
06700 SAINT LAURENT DU VAR
France

Hotel website

Copyright (c) DTR Society, 2025

Contact Us

Proposals

The submission system is currently being prepared. We invite you to subscribe to the newsletter so you will be the first to know when it goes online.