Medical RAG Pipeline Selection with Retrieval Quality Prediction and LLM Safety Explanations

Abstract

Medical retrieval-augmented generation (RAG) systems must match retrieval strategy to question structure, estimate whether the retrieved evidence is strong enough to support an answer, and communicate clearly when the evidence is insufficient. This paper presents RQ-SafeSelector, an evidence-aware pipeline-selection framework evaluated on the 420-question RAGCare-QA benchmark across six medical specialties, three complexity levels, and three annotated RAG pipeline classes. The framework compares sparse TF-IDF, multi-vector lexical–character retrieval, and graph-enhanced hybrid retrieval; predicts retrieval quality from out-of-fold scores; and routes each question to Basic RAG, Multi-vector RAG, or Graph-enhanced RAG with a structured safety explanation and abstention gate. Graph-enhanced Hybrid achieved the strongest overall Recall@3 (0.336) and nDCG@5 (0.339), while Multi-vector Hybrid obtained the highest Recall@5 (0.548). RQ-SafeSelector reached 0.933 accuracy and 0.833 macro-F1 for pipeline selection. The calibrated retrieval-quality predictor achieved ROC-AUC 0.686 with expected calibration error 0.036. Applying the quality gate reduced the audit-defined unsupported-answer risk to 0.000 while answering 30.7% of questions, with Success@3 of 0.527 among answered cases. The findings support evaluation of medical RAG as a routed, evidence-aware, and selectively answering system rather than as a single retriever followed by an unconstrained generator.