Points show saved historical results. A connecting line requires complete, identical declared evaluation conditions. Model and date may vary; the test itself may not.
Loading the curated observations… If data cannot load, use the JSON download and methodology below.
All selected groups appear below, including unscored and missing records. Chart markers link to their table rows. Scroll the table horizontally on a small screen.
Saved results by run, strategy and token budget
Run / declared model
Strategy / budget
Diagnostic credit
Critical retained
Coverage
Verdicts
02 / Provenance
Run details
Model and provider names are saved declarations, not authenticated execution. Source revisions, fixture hashes and option fingerprints describe the declared conditions.
The arithmetic mean of saved automatic diagnosis credits among scored answers, including partial credit. This is not binary accuracy. A skipped or missing answer is unscored, never a zero. Each row reports its own denominator; groups are not pooled into an overall score.
Critical evidence retained
Critical log items retained / critical items in the original input. This measures evidence availability after reduction, not whether a response cites evidence correctly. It is a separate dimension; even an unscored model call can have a measured retained-input count.
Strategies and budgets
raw uses the unreduced input; error_window selects context around errors. 8k is the declared token budget and unrestricted has no budget cap. Token estimates use the saved estimator, not a claim of exact tokenizer counts. Retaining evidence alone does not guarantee diagnosis.
Versions are boundaries
These pilots preserve scoring-v3, diag-v1 and response-v1 results without rescoring. New engine or scoring versions do not rewrite old observations. A changed source revision, fixture pool, size, seed, version or execution option breaks comparability.
Exact requirements for a connecting line
The page computes comparability from source revision; scenario pool, sizes, fixture hashes and seed; dataset, prompt, response and scoring versions; token estimator; generation, strategy-option and provider-extra-argument fingerprints; provider; effort; repetitions; strategy; budget; benchmark and metric. Every field must be known and match. A curator-provided cohort label cannot override this rule. Unknown values never compare as known. Only scored observations on distinct dates can be connected.
Coverage is deliberately small: no confidence intervals, general performance rankings or release-over-time conclusions are supported. More observations under the same declared evaluation conditions are needed to assess a trend.
04 / Keep it reproducible
Add the next observation
Publish reviewed aggregate records with their original scoring and declared conditions. Keep private prompts, logs, outputs and operational details out of the publication.