Only 17 of 700 AI Attempts Found Every Required Finding
Part 2
Rushendra Sidiboma, Parth Patwa, and Sanat Mishra
BioStack Platforms
Earlier, we built an evidence-grounded QA benchmark that asked models to show the images behind an answer.
Read Part 1: We Made Medical AI Show Its Work.
Even there, the images had already been selected for the model. This time, we removed that shortcut.
A full CT, MRI, or PET/CT study can contain dozens of series and thousands of images. Before a model can interpret a finding, it has to choose where to look, narrow the search, inspect the right slices, and decide when it has seen enough.
We built a medical-imaging navigation environment around that job. Each episode gives the model access to one full rendered study, a neutral review request, and 12 turns. The radiology report, required findings, and private scoring data stay hidden.
We evaluated GPT-5.6 Sol, Gemini 3.1 Pro Preview, Claude Fable 5 at low effort, and Grok 4.6 once on each of the same 175 cases. No model recovered every required finding in more than six cases. The best complete-case accuracy was 3.43%.
Most benchmarks begin after the search is over
Selected-image benchmarks are useful for measuring recognition. They ask whether a model can understand the images placed in front of it. The search problem has already been solved.
A full study adds another layer. The relevant finding may sit in one reconstruction among many and span only a few slices. A model can describe a chosen image well and still fail when it has to find that image for itself.
Reading an image is one skill. Finding it inside a study is another.
Built from full medical-imaging studies
We built the environment from complete CT, MRI, and PET/CT studies from 186 de-identified cases. After expanding the multi-frame scans, it exposes 172,353 individually addressable images across 2,166 series.
We began by indexing every study and rendering its DICOM images into a consistent catalog. Multi-frame files needed extra work: 1,171 DICOM instances contained 39,410 embedded frames, and we exposed every frame with zero omissions. We then grouped images by series and generated contact sheets covering 25 slices at a time. The model uses those sheets to narrow the search, then opens individual slices at full resolution before citing them. Gold findings drawn from the radiologist-authored report stay behind a private scoring boundary. The finished environment wraps the study in a reset and action interface, records every step, and checkpoints after each completed turn.
The environment starts with the whole study
At the beginning of an episode, the model receives limited public context and a catalog of series. It can survey the study, use contact sheets to narrow the range, open selected slices at full resolution, and submit up to eight free-text findings.
Browse: List the study series and available image ranges.
Narrow: Use compact contact sheets to choose where to look.
Inspect: Open selected slices at full resolution.
Submit: Return up to eight free-text findings with citations.
Every submitted finding includes confidence and citations. A citation is valid only if the model opened that slice at full resolution earlier in the episode. Contact sheets help with retrieval, but they cannot be cited as final evidence.
There are no answer choices and no hidden answer slot. The model has to decide what matters and write the finding in its own words.
The workflow has four stages, even when the study contains thousands of images.
Every turn leaves a usable trace
We save the ordered images shown to the model, the raw response, the parsed action, the tool result, opened-slice identities, errors, and the final submission. The environment checkpoints after every completed turn, so an interrupted rollout can resume without losing finished work.
We keep the raw trace immutable and store rewards in separate, versioned overlays. A customer can rescore a rollout later without altering the model's original behavior.
Bring your own inference
The model runner and the environment are separate. Customers can drive the reset and action interface from their own policy workers, or register an OpenAI-compatible multimodal endpoint and let the managed service collect rollouts. They can also train offline from exported rollouts.
The environment owns study state, navigation tools, the turn budget, replay, and rollout capture. A separate private verifier owns the scoring contract. The customer keeps control of the model, sampling settings, and post-training loop.
A real episode: one injury inside 1,122 images
In one de-identified CT angiography study, the model had two series, 46 possible contact sheets, and 1,122 full-resolution images available. The traumatic hand injury occupied a small part of that search space.
The model began with an arterial-phase contact sheet covering slices 476 to 500. It opened selected slices at full resolution, widened the range, sampled the venous series, and returned to the arterial series for its final citations.
The contact sheet narrowed the search. Final citations came from full-resolution slices opened later.
Search footprint: 2 of 46 contact sheets and 15 of 1,122 full-resolution images.
Episode length: 8 of 12 available turns.
How the model got there
Turn 1: Opened the arterial contact sheet for slices 476 to 500 and found the likely region without loading the whole study.
Turns 2 to 3: Opened selected arterial slices, widened the range, and chose candidate citations for the traumatic-injury conclusion.
Turns 4 to 5: Compared the same region in the venous acquisition.
Turns 6 to 7: Returned to arterial slices at full resolution and finalized the cited images.
Turn 8: Submitted two conclusions within the 12-turn budget.
The episode earned full reward
"Severe mangling and partial traumatic amputation of the right thumb with extensive soft-tissue loss and comminuted fractures of the first metacarpal and phalanges."
This case had one report-backed gold finding: fractures involving the right-hand phalanges. The quoted conclusion recovered it, and all three cited slices had been opened at full resolution. The episode recovered its gold target, received a strict terminal reward of 1.0, and counted as a complete-case pass.
The conclusion matters, but the full trajectory is what makes the episode useful for post-training. It shows where the model searched, what it opened, what it skipped, and where its policy can improve.
A final answer can be right or wrong. A rollout shows what to improve.
How the reward works
Browsing never tells the model whether it is getting warmer. Navigation actions receive no hidden clinical reward. The score arrives only after the model submits its conclusions.
The source of truth is the radiologist-authored report paired with each study. We turn its documented findings into private gold targets that the evaluated model never sees. After submission, two blinded model families translate the model's free-text conclusions into fixed clinical attributes without seeing the report or gold targets. A third blinded judge resolves disagreements. Deterministic rules then compare the translated clinical concept, polarity, anatomy, and required qualifiers with the report-backed gold.
A gold finding receives strict credit only when the clinical match passes and the submission cites a full-resolution slice opened earlier in the episode. The terminal reward starts from the share of gold findings recovered, with deductions for explicit contradictions. This ties the reward to both the clinical conclusion and the model's actual navigation.
The plots below report strict accuracy. Required-finding recovery counts exact gold-target matches with valid opened-slice provenance. Complete-case accuracy counts a case only when every required finding passes.
Four frontier models. 175 cases each.
The result covers 175 cases and 241 report-backed gold findings. Each model made one attempt per case, for 700 full-study attempts in total. All runs used the 12-turn navigation contract and the same strict target definitions and scoring rules.
Only 17 of 700 attempts recovered every required finding
No model crossed 3.5% complete-case accuracy. Claude recovered the most individual required findings: 9 of 241. Grok recovered every required finding in 6 of 175 cases, the highest complete-case count. The strongest target-recovery result and the strongest complete-case result came from different models.
The message is simple: strong performance on selected images does not yet translate into reliable full-study review. This environment makes that gap visible and trainable.
Every rollout reveals what to improve
A score says that a model failed. The rollout shows what kind of failure it was. Did the model search the wrong series? Did it reach the right region but miss the decisive slices? Did it open useful images and still write the wrong conclusion? Did it lose a qualifier such as laterality or severity? Or did the episode fail because of an invalid action, malformed submission, or citation to an image it never opened? Each case points to a different training signal.
Strong trajectories can become supervised examples. Better and worse paths on the same study can form preference pairs. Terminal rewards can support reinforcement learning. The same trace can be rescored when a customer adds workflow-specific reward components.
The goal is not just to grade the model. It is to produce the signal needed to improve it.
What customers can do with the environment
Customers can run hosted or local LLMs through an OpenAI-compatible API, collect every observation and action as a replayable rollout, attach private reward after submission, and export the result into supervised, preference, or reinforcement-learning workflows. The next checkpoint can run against the same environment contract, so customers can compare behavior as well as final answers.
Build the environment around the behavior you want to improve
Customers building frontier models need environments that make the target behavior observable, collect replayable trajectories, and turn outcomes into useful reward.
BioStack builds private RL environments for high-value healthcare workflows. We can connect your inference stack, adapt the navigation and reward contract to your use case, and deliver the rollouts and evaluation needed for post-training.
If you are post-training a frontier multimodal model for healthcare, contact founders@getbiostack.com.
If you are building a medical multimodal model, we can design a private, real-world benchmark around the failures that matter, turn those findings into targeted post-training work, and measure the result on held-out tasks.
For private evaluations, datasets, RL environments, post-training pilots, or research collaborations, contact founders@getbiostack.com.