An Evidence-Grounded Multimodal Medical-Imaging Benchmark
Building a 513-question medical-imaging benchmark for answers and supporting evidence
Rushendra Sidiboma, Parth Patwa, and Sanat Mishra
BioStack Platforms
Most medical-imaging benchmarks stop at the answer. That bar is too low.
Each item in our benchmark gives the model one question, five answer choices, and 7 to 12 labeled medical images. The model must choose one answer and identify one to three images that support it.
Answer accuracy measures whether the answer is correct. Joint accuracy requires both the answer and its supporting images to be correct on the same question.
The gap was large. Across five complete frontier-model runs, answer accuracy ranged from 33.9% to 41.9%. Joint accuracy ranged from just 11.7% to 25.0%.
A correct answer does not prove image grounding
Medical-imaging questions are easy to leak clues into. Reports can state the finding. Stems can name the modality. Answer choices can contain patterns that make one option look plausible before the images are opened.
That does not mean every correct answer is a guess. It means answer accuracy alone cannot tell us what made the answer possible.
A stronger benchmark asks two questions: Did the model choose the reference answer? Can it point to the images that support that answer?
Evidence selection cannot reveal a model's internal reasoning or prove what caused an answer. It does raise the behavioral bar. The model must commit to both the finding and the images that show it.
A correct answer without acceptable evidence remains partial, not complete.
The data behind the benchmark
We started with our proprietary dataset of 186 de-identified CT, MRI, and PET/CT cases paired with radiologist reports. The collection contained 134,828 DICOM instances across 2,166 series. DICOM is the standard format used to store and exchange medical images.
The final benchmark draws from 135 of those cases. The reports anchored the clinical findings and served as private supervision during construction. The evaluated models never saw them.
We indexed and rendered every study. DICOM metadata narrowed the likely series, which we divided into contact sheets of 12 labeled slices. Two multimodal reviewers scanned the sheets independently without seeing the target finding and nominated candidate slices.
We mapped their nominations back to full-resolution images, added nearby slices, and combined them into a pool of up to 15 candidates. Both reviewers then checked that pool against the report-backed finding and rated the visual support as directly visible, possibly visible, or unassessable. We retained targets with at least one direct or possible judgment, and selected up to three of the strongest slices as private reference evidence. Contact sheets were used only for retrieval, never as model input or final evidence.
For each question, we paired these evidence images with comparison images from the same case. The resulting image-level labels are report-derived and LLM-assisted. They were not independently adjudicated by radiologists.
One answer. Supporting images. Both must be right.
Each item gives the model one question, five answer choices, and 7 to 12 images. Every image carries an opaque ID such as img01. The ID reveals nothing clinical. The model sees no report, answer key, or evidence label.
Here is one complete item.
Which incidental nonspinal process is demonstrated on the available images?
A Pansinusitis-related inflammatory process
B Otitis-related inflammatory process
C Sialadenitis-related inflammatory process
D Tonsillitis-related inflammatory process
E Mastoiditis-related inflammatory process
Green ✓ = acceptable evidence. Red × = comparison image that does not establish the answer.
The borders are shown only for explanation. Evaluated models received the images without them.
Reference answer: A. Pansinusitis-related inflammatory process
Reference evidence: img04, img12
One failed response (Grok 4.5):
Why joint scoring matters. Grok 4.5 diagnosed mastoiditis and selected img05 and img09. Both are high axial brain slices showing neither the paranasal sinuses nor mastoid air cells: zero acceptable hits. The reference is pansinusitis, supported by img04 and img12. Answer, evidence, and joint all fail.
Across the benchmark, the model may select one to three images. One acceptable hit is required when the reference set contains one or two images; two hits are required when it contains three or more. At least half of the selected images must be acceptable. We also accept same-series slices within two positions of a reference slice because adjacent slices can show the same finding. Joint accuracy passes only when answer and evidence both pass.
The answer choices leaked shortcuts
This was the failure that mattered. A model could often narrow the answer before looking at the images.
The correct answer stood out
Our audit of an earlier version found two recurring cues. The correct option often had the maximum token length. In plain English, it was the longest choice. It also often had the highest semantic similarity to the question.
That made the task easier for the wrong reason. A model could favor the longest or most question-like choice without resolving the image. Strong answer accuracy could therefore reflect option design as much as visual reasoning.
We needed the images to carry the deciding signal.
We rebuilt every item as a balanced contrast set
For each finding, we generated multiple five-choice sets. Each held the clinical category fixed and varied one image-resolvable axis, such as location, extent, pattern, or anatomic relationship.
We matched choices in grammar, specificity, abstraction, prevalence, and length. Checks rejected length outliers, giveaway wording, unmatched qualifiers, and copies of the private target.
One provider generated candidates; another validated them. Two text-only providers saw three shuffled versions without images, report, or key. We rewrote any set the text exposed.
Worked example: how the pansinusitis item was built
| 1. Ground |
|
| 2. Isolate |
|
| 3. Contrast |
|
| 4. Stress-test |
|
We audited the corpus, not just individual questions
A final pass looked for shortcuts that only appear at scale: gold-associated words, answer-length outliers, semantic centrality, overlap with the private target, and text-only solvability. Flagged items were regenerated. We also balanced the correct labels across A through E, so the answer position carried no useful signal.
The finished choices were almost identical in average length: 3.092 words for correct answers and 3.114 for distractors. Correct labels were balanced at 102, 104, 104, 101, and 102 across A through E.
Giveaway flags fell from 424 to zero. Exact private-target copies fell from 160 to one. Blind text-only gates selected the intended answer in 203 of 3,069 shuffled trials (6.6%), a construction diagnostic rather than a model score. Target-model failures never affected membership or answers.
The final benchmark
We began with the radiology reports, not model failures. We extracted distinct, positive visual findings, each tied to one exact report sentence. A case could contribute more than one target when its report contained several distinct findings. LLM reviewers then searched the full study in two stages: first describing images without seeing the target, then checking shortlisted full-resolution slices against the report-backed finding. For each target with usable image support, a question generator wrote one neutral finding-identification question and an initial five-choice answer set. This produced 533 candidate questions from 139 cases.
We then applied a rule fixed before evaluation. We removed 20 candidates that required an exact measurement or threshold when the rendered images exposed no physical scale. This left 513 questions from 135 cases. No target-model result affected inclusion.
With membership fixed, we rebuilt the final MCQs using the contrast-set process above. For each finding, one provider generated several complete five-choice sets. Each set held the clinical category fixed, varied one image-resolvable feature, and contained five parallel values before the report-supported value was mapped to a balanced answer position. A second provider checked the choices against the private report, while two blind text-only providers tested three shuffled versions without images, report, or answer key. We retained the hardest valid set. Of the 513 final items, 510 use newly generated contrast sets and three use predefined fallback sets chosen before model evaluation.
The final wording pass removed explicit CT, MR, MRI, and PET names from 403 stems; the other 110 required no change. It did not change membership, labels, images, evidence, or splits. The benchmark contains 329 development, 96 validation, and 88 test items. Splits are deterministic and case-disjoint. We report aggregate performance across all 513 questions. The split names do not imply model training.
How we queried the models
Every model received the same 513 items in one structured multimodal request per question. The model saw one question, five answer choices, and 7 to 12 labeled images, then returned one answer and one to three evidence image IDs. We used a short, fixed prompt with no model-specific tuning:
We supplied no radiology report, patient metadata, target statement, or evidence labels. We used no few-shot examples, chain-of-thought instructions, or model-specific prompt variants. Each image followed its corresponding image-ID label. Questions, images, and scoring rules were fixed before evaluation.
The best joint score was 25.0%
We evaluated five complete frontier-model runs from four providers on the same 513 questions.
| Model | Answer accuracy | Evidence accuracy | Joint accuracy |
|---|---|---|---|
| Claude Fable 5 | 41.9% | 51.7% | 25.0% |
| Claude Opus 5 | 38.2% | 48.7% | 20.9% |
| GPT-5.6 SOL | 41.5% | 39.4% | 17.0% |
| Gemini 3.1 Pro Preview | 34.9% | 33.3% | 11.7% |
| Grok 4.5 | 33.9% | 47.0% | 17.3% |
Answer accuracy ranged from 33.9% to 41.9%. Joint accuracy ranged from 11.7% to 25.0%. The best rounded joint result was roughly one quarter of the benchmark.
The three scores show different failure patterns. Claude Fable 5 had the highest evidence accuracy at 51.7%, but only 25.0% joint accuracy. GPT-5.6 SOL nearly matched the top answer score at 41.5%, yet reached 17.0% joint accuracy.
The averages hid different strengths
Fable led overall, yet the other four models still showed distinct strengths on individual cases.
Clinical anatomy. On a head CT question about the caudalmost prominent ventricular component, GPT-5.6 SOL alone returned the third ventricle with acceptable evidence. Fable, Opus, Gemini, and Grok all chose the fourth ventricle.
Anatomic localization. On a lower-extremity CT question about bilateral subcutaneous edema, Gemini and Grok localized it to both feet and selected acceptable images. Fable, Opus, and GPT-5.6 SOL all chose both calves.
Answer versus evidence. On a chest CT question about an air-containing lung abnormality, Opus and Grok got both the bullous finding and the supporting images right. GPT-5.6 SOL and Gemini chose the right finding but missed the evidence. Fable chose a pneumatocele.
Each model looked expert on some cases. None looked excellent across the benchmark. The aggregate leader still missed questions that lower-scoring models solved, and many correct answers came with weak image localization. A single average hides that unevenness. Joint accuracy exposes it.
From evaluation to post-training
A leaderboard number cannot tell a team what to improve. A useful evaluation should separate the failures.
At item level, this benchmark separates joint success, answer-only success, evidence-only success, and failure on both. Answer-only success can indicate a localization or selectivity problem. Evidence-only success can indicate difficulty resolving the clinical contrast. Failure on both can indicate a broader task gap. These signals can guide different data and post-training decisions.
BioStack builds real benchmarks, evaluations and reinforcement-learning environments for healthcare and lifescience agents. We use sealed evaluation sets to identify concrete failure modes, design targeted post-training work, and measure changes on held-out tasks. That last step matters. Improvement should be measured, not assumed.
The Evidence-Grounded Benchmark is one example of the evaluation layer in that workflow. Its value is the failure signal it exposes.
If you are building a medical multimodal model, we can design a private, real-world benchmark around the failures that matter, turn those findings into targeted post-training work, and measure the result on held-out tasks.
For private evaluations, datasets, RL environments, post-training pilots, or research collaborations, contact founders@getbiostack.com.