Research notebook / ChroSift

Less log.
Enough evidence?

Does reducing failure logs preserve useful diagnosis and evidence? A public history of small, documented experiments.

01 / Observations

History, with context

How to read this ↗

Points show saved historical results. A connecting line requires complete, identical declared evaluation conditions. Model and date may vary; the test itself may not.

Loading the curated observations… If data cannot load, use the JSON download and methodology below.

02 / Provenance

Run details

Model and provider names are saved declarations, not authenticated execution. Source revisions, fixture hashes and option fingerprints describe the declared conditions.

Complete run metadata is available in the curated JSON.

03 / Methodology

What the numbers mean

Diagnostic credit · 0–1

The arithmetic mean of saved automatic diagnosis credits among scored answers, including partial credit. This is not binary accuracy. A skipped or missing answer is unscored, never a zero. Each row reports its own denominator; groups are not pooled into an overall score.

Critical evidence retained

Critical log items retained / critical items in the original input. This measures evidence availability after reduction, not whether a response cites evidence correctly. It is a separate dimension; even an unscored model call can have a measured retained-input count.

Strategies and budgets

raw uses the unreduced input; error_window selects context around errors. 8k is the declared token budget and unrestricted has no budget cap. Token estimates use the saved estimator, not a claim of exact tokenizer counts. Retaining evidence alone does not guarantee diagnosis.

Versions are boundaries

These pilots preserve scoring-v3, diag-v1 and response-v1 results without rescoring. New engine or scoring versions do not rewrite old observations. A changed source revision, fixture pool, size, seed, version or execution option breaks comparability.

Exact requirements for a connecting line

The page computes comparability from source revision; scenario pool, sizes, fixture hashes and seed; dataset, prompt, response and scoring versions; token estimator; generation, strategy-option and provider-extra-argument fingerprints; provider; effort; repetitions; strategy; budget; benchmark and metric. Every field must be known and match. A curator-provided cohort label cannot override this rule. Unknown values never compare as known. Only scored observations on distinct dates can be connected.

Coverage is deliberately small: no confidence intervals, general performance rankings or release-over-time conclusions are supported. More observations under the same declared evaluation conditions are needed to assess a trend.

04 / Keep it reproducible

Add the next observation

Publish reviewed aggregate records with their original scoring and declared conditions. Keep private prompts, logs, outputs and operational details out of the publication.

Read the maintenance guide ↗