# Evaluation (R5)

**Question set:** 30 fixed questions — 24 answerable and 6 unanswerable (2
near-miss, 2 out-of-corpus, 1 under-specified, 1 ambiguous). **Label
provenance:** every answerable gold span was hand-copied from the ingested chunk
text (5 papers, 260 chunks) and is re-verified as a normalised substring of the
named `(doc_id, chunk_index, page_no)` before measurement; no label is
model-generated. **Judge provenance:** `eval/judge.yaml` pins
`gemini-3.8-flash`, a different exact id from the generator; the eval probes it
and recorded `judge_model_unavailable` (HTTP 503/timeouts, see
`eval/judge-status.json`). The generator was not substituted as judge, so
citation precision and faithfulness come from a deterministic rule-based
checker and are labelled **rule-based, not a model judge**. No Cohen's κ is
computed: the gold file has 30 question labels and zero human per-claim support
labels. The set is 30 rather than 100 because 100 spans could not be labelled
honestly in one session; see `eval/README.md` for the reduction and the guard
that enforces it.

The runs in "Three historical runs" were generated on the build machine
(Windows 11, ONNX CPU) against the real corpus:
`fastembed:BAAI/bge-small-en-v1.5` embeddings and
`fastembed:Xenova/ms-marco-MiniLM-L-6-v2` reranker. That historical comparison
used the retired keyless endpoint. The current keyed run is described in
"Keyed run: the schema-stage cause". Calibration: `eval/calibration.json`,
`gold-human-v1` (fitted on all 30 gold questions — see the in-sample caveat
below), threshold `0.6361` on the probability scale (≈ −0.353 on the raw
reranker score), fit-set `f1a55cdd…0479`.

Reproduce:

```sh
bash scripts/live.sh eval                      # production path → eval-report.json
bash scripts/live.sh eval --force-generation   # diagnostic only → eval-report-forced.json
bash scripts/live.sh eval --ablate vector-only  # weakened path → eval-report-ablation.json
```

`scripts/live.sh` parses `P:/portfolio/.secrets/model.env`, exports only
`DOCQA_MODEL_API_KEY` / `DOCQA_JUDGE_API_KEY` and the endpoint settings, and
refuses to run without a key; no value is echoed. Without a model endpoint the
same harness uses the labelled extractive fallback (that is how the committed
`eval-report-ablation.json` was produced in R2).

The command exits **6** when any gate fails. The current keyed run exits **0**:
all five production gates pass. It took **710.3 s (11.84 min)** wall clock,
measured 30 questions, and made **26 model requests with 0 retries, 0
rate-limit failures and 0 server errors**. The numbers are reported, not tuned
away.

## Keyed run: the schema-stage cause

W2-a reported seven `schema` failures (B1, B3, B4, B5, D1, D2, R2). They were
not schema failures. The instrumented run records, per failing question,
`http_status`, `finish_reason`, `structured_mode`, the parse errors and
bounded excerpts of the wire body; in that run all seven came back HTTP 200
with `finish_reason: "tool_calls"`, `structured_mode: "tool"` and zero parse
errors, so the forced `submit_answer` call was honoured and the JSON parsed.
Six were `quote_mismatch` and one `bad_quote` (a quote over the 240-character
cap); the claims died in the citation contract, not the schema. The eval now
labels those two causes separately: `schema` means the payload failed the
answer schema, `validation` means a parsed claim was dropped by the citation
contract.

The mismatches have one measured cause: the chunks are raw PyMuPDF extraction
of two-column pages. A left-column line and a right-column line are stored on
the same extracted line, and hyphenated words are split across line breaks, so
a sentence quoted in natural reading order is not contiguous in the chunk.
Concretely, the model returned `Training of BERTBASE was performed on 4\nCloud
TPUs ...`, while the stored chunk inserts `Label = IsNext` between those two
lines; it returned `setting (Section 3.2)` where the stored chunk contains
`setting (sec- [70] tion 3.2)`. Telling the model that passages are interleaved
extraction and that a quote must not contain a newline or cross the column gap
recovered three of the original seven (B4, D1, R2) and both failures the
instrumented run added (S3, D3); four (B1, B3, B5, D2) still return a quote
that spans the gap. The ADR 0004 JSON-only fallback is implemented as exactly
one retry after a schema parse failure, recorded as
`structured_mode: "json-fallback"`, and it fires only on such a failure; it
fired **zero** times here because there were no parse failures, so the lift is
not a silent mode switch. Measured: the answer rate moved from 11/24 (45.8%)
to 15/24 (62.5%) [42.7%, 78.8%], and all five gates pass; the interval's lower
bound is below the 50% threshold, so the point estimate passes, not the whole
interval. T2, a single generation-stage failure in W2-a, also answered; with
one sample that movement is not separable from provider variance. The root
cause is not eliminated: de-interleaving the columns requires a re-ingest that
changes chunk text, retrieval and the gold chunk ids, which this card forbids.

Every failing row now carries a `failure` object: `reason` (`schema_parse_error`,
`quote_mismatch`, `bad_quote`, `unknown_citation`, `uncited`,
`claim_validation_failed`, `gold_chunk_absent` or `model_call_error`), `stage`,
`structured_mode`, `http_status`, `finish_reason`, `parse_errors`,
`response_excerpt` and `raw_text_excerpt` (both capped at 200 characters),
plus `claims_returned`, `claims_kept`, `dropped_uncited`,
`dropped_unknown_citation`, `dropped_quote_mismatch`, `dropped_bad_quote`,
`schema_failures` and `repair_turns`. `metrics.stage_failure_reasons` counts
those reasons and `metrics.structured_mode_counts` counts the mode that
produced each final payload (`tool`, `json`, `json-fallback`, `extractive`).

## Three historical runs

All three use the same 30 questions, corpus, embedder and reranker. The
historical keyless endpoint allowed about one request per 30 s, so the runs were
serialised.

| run | UTC window | threshold | purpose | wall clock | model requests | answer rate (24) | unanswerable refusal (6) | citation precision* | faithfulness* | gates | exit |
|---|---|---|---:|---|---:|---:|---|---|---|---|---:|
| (a) baseline | 17:43:50–17:46:19 | `0.2552` (provisional) | gate run before the refit | 148.1 s | 5 | 2/24 (8.3%) [2.3%, 25.8%] | 6/6 (100%) [61.0%, 100%] | 2/2 (100%) [34.2%, 100%] | 2/2 (100%) [34.2%, 100%] | 4 PASS / 1 FAIL | 6 |
| (b) forced diagnostic | 17:52:55–18:08:12 | `0.6361` bypassed | measure the model's own answer rate | 916.4 s | 30 | 8/24 (33.3%) [18.0%, 53.3%] | 5/6 (83.3%) [43.6%, 97.0%] | 7/10 (70.0%) [39.7%, 89.2%] | 7/10 (70.0%) [39.7%, 89.2%] | 3 FAIL (diagnostic only) | 0 by design |
| (c) final production | 18:09:38–18:22:37 | `0.6361` | gate run after the refit | 778.6 s | 26 | 8/24 (33.3%) [18.0%, 53.3%] | 6/6 (100%) [61.0%, 100%] | 7/9 (77.8%) [45.3%, 93.7%] | 7/9 (77.8%) [45.3%, 93.7%] | 2 PASS / 3 FAIL | 6 |

\* rule-based, not a model judge. Run (b) is **not a gate run**: it bypasses
only the calibrated low-score refusal (`DOCQA_FORCE_GENERATION=1` /
`--force-generation`), the citation validator still runs, and the report records
`run.gate_run: false`. Its report is `eval-report-forced.json`; run (a) is
`eval-report-baseline.json`; run (c) is `eval-report.json`.

## What the forced diagnostic showed

Run (b) answers the question this card existed to ask: **is the free model
unable to answer, or is the refusal threshold doing the refusing?** Both were
true, in different proportions.

- With the threshold bypassed, the model still produced a contract-valid answer
  for only **8/24 answerable questions (33.3%, [18.0%, 53.3%])**. The threshold
  is no longer the binding constraint after the refit: it admits all 24
  answerable questions (in-sample), and run (c) reaches the same 8/24.
- Of the 16 answerable questions that did not answer, **13 model replies did make
  a claim, but the quote was not a verbatim substring of the cited chunk**, so
  the validator correctly dropped it (B1–B5, S3, S4, D1–D3, D5, R2, R5). This
  is the dominant failure, and it is a model/protocol failure, not a retrieval
  failure: 11 of those 13 had the gold chunk in the top-8.
- Two replies (S2, D4) declined with `insufficient_context`; one call (S1)
  returned no `submit_answer` tool call.
- Five questions (S1, S2, S4, D4, D5) had already lost the gold chunk at
  retrieval, so they could not have been answered from the context.

The unanswerable side moved the other way. With the new threshold, run (c)
admits two unanswerable questions (U1, U6) to the model instead of refusing
them at `low_score`, but the model or validator then refuses both — U1 with
`insufficient_context`, U6 with `validation_failed` — so the refusal rate stays
6/6. The in-sample calibration alone refuses 4/6 (U2–U5); the other two are
refused downstream. This is stated as measured rather than presented as a
calibration win.

## Metrics

All rates carry a Wilson 95% interval and their denominator `n`.

| metric | definition |
|---|---|
| retrieval recall | the gold chunk is in the returned top-k for an answerable question |
| answer rate | an answerable question returns ≥1 validated claim |
| unanswerable refusal rate | an unanswerable question returns `insufficient_evidence=true` |
| citation precision (rule-based) | a citation is correct when it names the gold chunk |
| faithfulness (rule-based) | every citation on a claim names the gold chunk and its verbatim quote contains, or is contained in, the gold quote |
| `stage_failed` | earliest stage that lost the span/answer: `retrieval`, `fusion`, `rerank`, `generation`, `schema` (the model's payload failed the answer schema) or `validation` (a claim was dropped by the citation contract) |

Rules are enforced per metric: retrieval recall ≥ 75%, answer rate ≥ 50%,
unanswerable refusal ≥ 80%, citation precision ≥ 80%, faithfulness ≥ 80%. An
empty denominator fails a gate; a gate never passes vacuously.

## Ablation table

Retrieval rows use the same 24 answerable questions. The answer rows come from
the historical run (c); the current keyed run's answer rows are in
"Keyed run: the schema-stage cause". The forced row is a labelled diagnostic and
is not a production result.

| configuration | n | metric | result | 95% Wilson CI |
|---|---:|---|---:|---|
| vector-only top-8 | 24 | recall@8 | 17/24 (70.8%) | [50.8%, 85.1%] |
| + lexical / RRF top-8 | 24 | recall@8 | 17/24 (70.8%) | [50.8%, 85.1%] |
| + rerank top-8 | 24 | recall@8 | 18/24 (75.0%) | [55.1%, 88.0%] |
| + calibrated refusal + live generator (full path) | 24 | answer rate | 8/24 (33.3%) | [18.0%, 53.3%] |
| + calibrated refusal + live generator (full path) | 6 | unanswerable refusal | 6/6 (100%) | [61.0%, 100.0%] |
| + calibrated refusal + live generator (full path) | 9 | citation precision* | 7/9 (77.8%) | [45.3%, 93.7%] |
| + calibrated refusal + live generator (full path) | 9 | faithfulness* | 7/9 (77.8%) | [45.3%, 93.7%] |
| forced generation (diagnostic, not a gate run) | 24 | answer rate | 8/24 (33.3%) | [18.0%, 53.3%] |

\* rule-based, not a model judge.

Stage attribution over all 30 questions in run (c): `retrieval=5`, `fusion=0`,
`rerank=1`, `generation=0`, `schema=11`. The five retrieval failures are S1, S2,
S4, D4 and D5 (the gold chunk is absent from the reranked top-8); R4 is the
single rerank loss — the gold chunk was not in context, yet the model answered
with two citations to other chunks, which is why precision is 7/9 rather than
8/8. The eleven rows labelled `schema` in that run were B1–B5, S3, D1–D3, R2
and R5: the model ran, but every claim it returned failed the citation
contract. They were citation-contract drops, not schema parse failures — the
distinction the current report makes with `validation`. The `generation=0`
count differs from run (a)'s `generation=13` because the refitted threshold no
longer refuses answerable questions before generation; the refusals that remain
in run (c) are four unanswerable questions at `low_score`.

## The weakened retriever must fail a gate

`--ablate vector-only` lowers the retriever to vector-only, top-3. It measures
the same 24 answerable questions:

| configuration | metric | result | 95% Wilson CI | gate |
|---|---|---:|---|---|
| production (hybrid+RRF+rerank, top-8) | recall@8 | 18/24 (75.0%) | [55.1%, 88.0%] | **PASS** (≥75%) |
| weakened (vector-only, top-3) | recall@3 | 15/24 (62.5%) | [42.7%, 78.8%] | **FAIL** (<75%) |

The ablation also fails the unanswerable-refusal gate: vector retrieval carries
no rerank score, so the calibrated threshold cannot fire and only 1/6
unanswerable questions is refused ([3.0%, 56.4%] vs the 80% gate). In the R2
extractive run its answer rate rose to 24/24 ([86.2%, 100%]) and citation
precision fell to 15/85 (17.6%, [11.0%, 27.1%]) — answering more often while
grounding less. Those answer-path rows are labelled with their generator in
`eval-report-ablation.json`; the retrieval gate is the controlled comparison
because it is generator-independent and changes for that reason alone. The
ablation artifact predates the gold refit, so its `refusal_threshold` field
still records `0.2552`; no retrieval number depends on that field.

## Reading the current keyed result honestly

- **Retrieval passes at the threshold, barely.** 18/24 = 75.0% is exactly the
  gate. The interval reaches down to 55.1%, and the vector-only and hybrid rows
  are identical at n=24, so this set does not establish that RRF or rerank is
  reliably better. Rerank recovers one question (R4); fusion recovers none.
- **The answer-rate gate passes at 15/24 (62.5%, [42.7%, 78.8%]).** Nine
  answerable questions do not answer: five lost the gold chunk at retrieval
  (S1, S2, S4, D4, D5) and four returned a quote that spans the interleaved
  columns (B1, B3, B5, D2). The interval's lower bound is below the 50%
  threshold, so the point estimate passes, not the whole interval.
- **Citation precision and faithfulness pass at 13/15 (86.7%) and 12/15
  (80.0%).** Faithfulness sits exactly on the 80% gate and its interval reaches
  down to 54.8%, so this is a point-estimate pass. R4's two claims still cite
  non-gold chunks; they pass the verbatim-quote contract but do not answer the
  question from the gold passage. The gate is not lowered.
- **The refusal gate passes, but not because the calibration is perfect.** All
  six unanswerable questions are refused: U2–U5 at `low_score`, U1 with
  `insufficient_context`, U6 with `validation_failed`. The in-sample calibration
  itself refuses only 4/6; U1 and U6 reach the model and are refused there. The
  held-out folds refused 2/3 unanswerables each.
- **The calibration's reported fit is in-sample.** The final artifact is fitted
  on all 30 gold questions — the same questions run (c) evaluates. Its
  balanced-accuracy figure is therefore optimistic, and run (c) is not a clean
  held-out test. The honest estimate is the id-parity split: fit on odd ids and
  evaluate even ids gives BA 0.833 (TPR 1.0, TNR 0.667; 10/10 answerable
  admitted, 2/3 unanswerable refused), and the reverse split gives the same BA
  (14/14, 2/3). Each fold has only three unanswerable questions, so those
  numbers are a direction, not a tight estimate. Details are in
  `eval/calibration.json` and below.
- **Ordering is not statistically established.** CIs overlap; n=24 is a sanity
  check. The harness reports the numbers it can defend and refuses to claim
  more.

## Calibration provenance and the in-sample caveat

`scripts/fit_calibration_gold.py` fits a question-level Platt scaling on the
top reranked score of each of the 30 hand-authored gold questions
(`label = 1` for the 24 answerable, `0` for the 6 unanswerable), then chooses
the threshold that maximises balanced accuracy (mean of TPR and TNR). Fitting
on the decision statistic the refusal gate actually uses is deliberate: the
gate asks "should this question be answered?", not "is this chunk relevant?".

| fit | n | a | b | threshold | TPR | TNR | balanced accuracy |
|---|---:|---:|---:|---:|---:|---:|---:|
| odd ids → even ids | 17 fit / 13 eval | 0.2627 | 0.8584 | 0.6921 | 1.000 | 0.667 | 0.833 |
| even ids → odd ids | 13 fit / 17 eval | 0.2659 | 0.8185 | 0.6461 | 1.000 | 0.667 | 0.833 |
| **all 30 (committed artifact)** | **30** | **0.3376** | **0.6775** | **0.6361** | **1.000** | **0.667** | **0.833** |

The committed row is **in-sample**: `eval/calibration_fit_set.json` is the same
30 questions as `eval/gold.jsonl`, and the threshold was refitted on all of
them after the held-out check. The held-out rows are the out-of-sample
evidence; both directions agree, which is why the final threshold is not
visibly overfitted, but with three negatives per fold the agreement is weak
evidence. The score cut implied by the final threshold (p = 0.6361) is ≈ −0.353
on the reranker's raw logit scale; note that `dummy:lexical-overlap` has a
different scale, so this artifact must not be used with the dummy reranker.

## Run cost and rate limiting

The historical keyless endpoint answered about once per 30 s per IP; the three
historical runs below were serialised at 31 s. The current keyed free tier
answers one request at a time: the client holds one request in flight per
endpoint, spaces request **starts** by at least `DOCQA_MODEL_MIN_INTERVAL_S`
(20 s), and retries a 429 or 503 twice (5 s, then 10 s) before raising
`RateLimitedError`. The current keyed production run took 710.3 s with 26
requests and no retries or rate-limit failures.

| historical run | requests | retries | rate-limited | minimum spacing floor | measured wall clock |
|---|---:|---:|---:|---:|---:|
| (a) baseline | 5 | 0 | 0 | 4 × 31 = 124 s | 148.1 s |
| (b) forced diagnostic | 30 | 0 | 0 | 29 × 31 = 899 s | 916.4 s |
| (c) final production | 26 | 0 | 0 | 25 × 31 = 775 s | 778.6 s |

Run (c)'s wall clock is almost entirely the rate-limit floor; local ONNX and
retrieval work adds ~4 s across 30 questions because it is dominated by the
31 s waits. The report records the budget in `run`:
`model_calls {requests, retries, rate_limited}`, `model_min_interval_s`,
`wall_clock_s`, and `blocked` (set after three consecutive 429s). There were 0
retries and 0 rate-limit failures in all three runs, and `blocked` is `null` in
every report.

## Files

| file | contents |
|---|---|
| `eval/gold.jsonl` | 30 fixed questions; span-level gold for the 24 answerable ones |
| `eval/README.md` | size rationale, label provenance, judge provenance, row format |
| `eval/calibration.json` | coefficients, threshold, fit-set hash, held-out history, in-sample caveat |
| `eval/calibration_fit_set.json` | 30 question-level fit rows (`question_id`, `query`, `chunk_id`, `label`, `rerank_score`, `split`) |
| `scripts/fit_calibration_gold.py` | human-gold fitter (production calibration) |
| `scripts/fit_calibration.py` | provisional synthetic fitter, kept for comparison; overwrites the production artifact |
| `eval-report.json` | final production run (c): per-question rows, aggregates, retrieval ablation, gates |
| `eval-report-baseline.json` | run (a): gate run with the provisional `0.2552` threshold |
| `eval-report-forced.json` | run (b): forced-generation diagnostic, `gate_run: false` |
| `eval-report-ablation.json` | vector-only/top-3 weakened run (R2 extractive generator), same schema |
| `src/docqa/eval.py` | loader, gold/corpus verification, Wilson intervals, stage attribution, gates, runner |
| `src/docqa/calibration_fit.py` | shared Platt fit, balanced accuracy, threshold choice |
| `tests/test_eval.py` | golden tests for intervals, gold shape, stage attribution, and gate failure |