An SDTM mapping specification describes how raw study data should become standardized datasets. Writing one means working across the data, study documents, and the SDTM standard. A lab result may need a unit conversion. A variable may depend on several source files. Some decisions can only be resolved by reading the protocol.
We built specbench to test how well AI systems can do this work. Each task gives a system a synthetic study folder and asks it to write the mapping specification. We then compare that specification with a known reference.
The benchmark
The benchmark contains 40 synthetic study packages. Each includes clinical data extracts, randomization and lab transfers, and the documents needed to interpret them: a protocol, annotated CRF, completion guidelines, and vendor transfer specifications.
The request identifies the datasets to produce, but does not list their output columns. The system has to work out which fields belong in the specification, how to combine data from different sources, and which mapping rules the documents require.
This matters because a task that lists every output column has already done part of the work. We started with the public SDTM cases in yamaa, where all three systems scored in the 90s. We kept the rule-based grading approach and built larger tasks that leave those decisions to the system.
Scoring
Each study package is generated alongside its reference specification. The same code produces the data, documents, and expected mapping rules, so the reference answers are known in advance. No language model is used to generate them.
We distinguish between rules a system can infer from the supplied materials and decisions that require a sponsor’s preference. The accuracy scores below cover the inferable rules.
A model judge assesses whether the written specification satisfies each rule. Separate code checks measure variable coverage, source-field mappings, and whether quoted citations appear in the cited sources.
Results
For this first evaluation, we tested all three systems on six study packages: two small, two medium, and two large. Together they contain 116 datasets and 2,137 inferable mapping rules.
Astraea got 94% of those rules right, compared with 87% for Claude Code and 84% for Codex. Astraea and Claude Code both used Claude Opus 5.5; Codex used GPT-6 Astra. All three received the same instruction text, and Claude Opus 5.5 judged their specifications using the same prompt.
Astraea led at all three study sizes. It also included more of the expected variables and provided a verified quotation on every variable row.
The omissions help explain the differences. Claude Code left out visit variables in the exposure dataset and study-day variables in medical history. Codex omitted many of the supplemental qualifiers in the reference specification and asked which ones the sponsor wanted.
Built-in checks
Astraea checks the specification while it is being written. Every raw field must either be mapped or explicitly excluded with a reason before the run can finish. Missing required variables also prevent completion.
The system can look up the SDTM implementation guide and controlled terminology as it writes. For example, it used the CDISC unit ug for a dose collected in mcg.
Grounding
Each row identifies its source fields and quotes supporting material from the study documents or the standard. A field that does not exist in the extract, or a quotation that cannot be found in its cited source, is rejected.
In this evaluation, all 2,210 of Astraea’s variable rows included a verified citation. It also recorded reasons for excluding 103 raw fields. That gives a reviewer a record of both the mappings and the decisions to leave data out.
Discussion
Astraea and Claude Code used the same model, but Astraea produced more accurate and complete specifications. This suggests that the workflow around the model matters. The comparison does not tell us how much each individual check contributed.
We evaluated six synthetic study packages, with mapping accuracy assessed by a model judge. These results do not establish performance on real studies. Citation checks also have a limit: they confirm that a quotation exists, not that it supports the mapping. The specifications still need study-specific validation and scientific review.
Specbench helps us see where a specification needs work: incorrect rules, missing fields, and decisions without supporting evidence.
Methods
The full benchmark contains 40 packages and 15,709 rules; 9% of the rules are tagged as sponsor choices. Packages span three sizes, from 8 datasets and 70 raw fields to 32 datasets and 190 raw fields. The six-package evaluation contains 2,137 inferable rules and 228 sponsor-choice rules.
Each package contains one CSV per eCRF form in a Rave-style structure, an IRT randomization transfer, and a central lab transfer using the vendor’s identifiers and units. The task specifies dataset classes, structures, and keys without supplying output-column lists.
The generator combines 18 form modules using a shared study context for subjects, arms, visits, dates, and discontinuations. Each module produces data, documentation, and reference mappings. A package is rejected if a raw field has no reference mapping or a variable is absent from the SDTMIG 3.4 catalog.
Astraea ran its sdtm-spec workflow on claude-opus-5-5. Claude Code used claude-opus-5-5 with a 1M-token context; Codex used gpt-6-astra. The judge used claude-opus-5-5 in two passes. Metrics other than rule verdicts were computed by code against the generated reference.
The original article reports 95% confidence intervals over packages: 93–96% for Astraea, 86–88% for Claude Code, and 82–86% for Codex. Inferable-rule scores by size were 95% / 87% / 85% on small packages, 95% / 88% / 84% on medium packages, and 93% / 87% / 84% on large packages, in the same system order.
The benchmark grew out of yamaa. Its reference prompts were split into individual rules, then tagged as inferable or sponsor-dependent. The generated study packages replaced the original tasks while retaining that grading approach.
For pharmaceutical teams, traceability matters alongside accuracy. Linking mapping rules to source documents and accounting for every raw field gives reviewers a clearer record of how a specification was produced. This can help teams investigate discrepancies, assess the impact of changes, and assemble evidence for validation and audit review. These checks support that process; they do not replace study-specific validation or the team’s scientific review and approval.