Fact-Faithful Editing with Ensemble Reranking: A Pre-Registered Study

1Introduction

Machine-writing pipelines increasingly separate drafting from editing: a generator writes a draft from a brief, and an editor rewrites that draft into publishable prose. For such an editor to be useful it must satisfy two constraints at once. It must be fact-faithful: the rewrite may neither contradict the draft nor introduce facts the draft does not support. And it must be length-preserving: editing is not summarisation, and a rewrite that quietly deletes a third of the draft has failed even if every surviving sentence is entailed.

These two constraints are in tension, and the tension is structural. Human-written training targets routinely contain facts absent from the draft; in this corpus the human originals invented numbers in 51.1% of items relative to their drafts. Cleaning the targets with rule-based and NLI-based filters removes those inventions, but the surviving targets are on average shorter relative to their drafts, and the editor learns to delete. In my runs, fact-faithful filtering moved the median output/draft ratio from about 0.98 for the un-fine-tuned base model to about 0.64 after fine-tuning. Nine pre-registered attempts to fix this on the target side, the prompt, the learning rate, the schedule, or the decoder failed. The repair that worked was to rewrite the targets themselves to be length-preserving (the "v4lp" data).

The study is explicitly pre-registered. All of it ran as numbered phases, and for every hypothesis the statement and decision rule were committed to the project git repository before any GPU or API work for that phase began; each phase closes with a machine-written summary and a result commit. A hypothesis counts as supported only if its pre-registered rule passes — by default, the lower bound of the item-cluster bootstrap 95% confidence interval on the paired difference must exceed zero. Of the registered hypotheses, twenty-three listed in Section 6 were not supported. I report them with the same detail as the successes, because several of the failures are the most informative results in the study.

1.1Contributions

1. Diagnosis of filter-induced length collapse (phases 3–5, 7). Under a fixed generation configuration, the un-fine-tuned base model has median output/draft ratio 0.981; fine-tuning on fact-filtered targets drives the median to about 0.64 monotonically over training (checkpoints 0.981 → 0.780 → 0.639 → 0.643 → 0.645). The deletion behaviour is learned from the targets, not inherited from the base model. The dose–response passed its pre-registered checks on 9B; its fine-grained form failed to replicate on 4B, so the dynamics are scale-sensitive even though the collapse itself is not.
2. Length-preserving targets (v4lp, phases 14–17). Rewriting each filtered target with an external rewriter under an explicit length contract, and keeping the 807 of 2,020 rewrites that satisfy both the fact rules and the contract, restores the median ratio to about 0.94 (9B: 0.938–0.945; 4B: 0.944–0.949) on fresh seeds and against a matched control trained on the un-rewritten targets over the identical 807 ids (control medians 0.70–0.73; paired median differences +0.202 and +0.231, both past the pre-registered ≥ +0.1 bar), while meeting a faithfulness non-degradation bar (pooled difference +2.9 points, CI [−3.25, 9.08], bar ≤ +5.0).
3. Decode-time ensemble reranking ("E-pick best-of-N", phases 21–24, 26, 28). Sample N candidates per item and select the candidate with the lowest mean contradiction score under an ensemble E = {D, R} of two NLI models (DeBERTa-v3-large NLI and RoBERTa-large NLI); the selected text is scored by a held-out third NLI model, B (BART-large NLI), never used for selection or training. Over 720 paired items, the contradicted share falls from 497/720 to 473/720 at best-of-4 (Δ3.3 pts, CI [0.6, 6.1], p = 0.024), to 446/720 at best-of-8 (Δ7.1, CI [4.0, 10.1], p ≈ 0), and to 441/720 at best-of-16 (Δ7.8, CI [4.6, 11.1], p ≈ 0). All three pre-registered hypotheses passed; best-of-16 adds only +0.7 over best-of-8.
4. A negative-results ledger (Section 6). Twenty-three pre-registered hypotheses were not supported, spanning target-side filters, schedules, optimisation (including a DPO effect that passed once, Δ6.2 CI [1.2, 11.2], and failed both replications), decoding (length-control prompts, chunked inference, EOS-forbidding, temperature, a "preserve facts" instruction), picker construction (single-family picking; a third picker family, twice; claim-decomposed scoring), repair (a checker-located verify-then-revise pass that changed nothing), evaluation (a blind LLM-judge panel), and model scale (27B: better fit on every recorded loss, no detectable contradiction reduction).
5. Honest accounting of what remains unsolved. After the strongest intervention, about 62% of items still register at least one contradiction under the held-out scorer; the blind-judge disagreement leaves open that part of the measured gain is NLI-family-specific; and the family-breaking judge study plus the author's hand labels were still pending at draft time.

1.2What this paper is not

This is not a system paper chasing a leaderboard. It is a single-editor, single-corpus, pre-registered study whose evidence base is 120 locked evaluation items pooled across at most six tuning lanes (720 paired items), with uncertainty reported as item-cluster bootstrap intervals and exact McNemar tests. I make no claim that the two surviving results transfer beyond this setting, and the editor they produce is not usable unattended: after every intervention tested, a majority of its outputs still contain at least one machine-flagged contradiction.

2Related work

This study touches four literatures. Citations are limited to papers verified during the project's literature review (project file soho-ev2-lit-h32/LITERATURE_MAP.md, 58+ verified entries with link checks logged); nothing is cited that was not checked.

Measuring factual consistency. NLI-based consistency checking over source–generation pairs is the dominant automatic instrument: FactCC (Kryściński et al., 2020), FEQA (Durmus et al., 2020), QAGS (Wang et al., 2020), and the SummaC benchmark (Laban et al., 2022), whose sliding-window sentence-pair convolution is the direct ancestor of my indicator. Maynez et al. (2020) established the hallucination taxonomy and, importantly here, the large disagreement between automatic checkers and human judgement — exactly the family-specificity threat that my held-out-scorer design and the pending family-break study address. TRUE (Honovich et al., 2022) meta-evaluates 11 checkers; AlignScore (Zha et al., 2023) trains a unified alignment function on 4.7M examples from seven tasks and was, on that basis, the one non-stock-encoder candidate for a third picker family — a bet this paper's H32c result closes as a null.

Claim decomposition. FActScore (Min et al., 2023) decomposes generations into atomic claims scored against a source; SAFE (Wei et al., 2024) and VERISCORE (Song et al., 2024) refine verifiable-claim scoring. My indicator (flag if any sentence-level pair exceeds 0.5) is coarser; claim-level decomposition inside the picker was the H32a experiment, and Section 5.8 reports it as a null: claim-mean scoring flipped most picks and made the held-out result slightly worse.

Best-of-N reranking and entailment rewards. Verifier-based reranking dates to Cobbe et al. (2021); learning from human preferences (Stiennon et al., 2020; Ouyang et al., 2022) and evidence-conditioned variants (Nakano et al., 2021) frame the training-side alternative, with DPO (Rafailov et al., 2023) the practical tool this study used and reports as a replication failure. Entailment as a training or selection signal is well precedented (Lee et al., 2022; Roit et al., 2023; Zhang et al., 2019). The failure modes are equally documented: reward-model overoptimization grows with the selection budget (Gao et al., 2023), and repeated-sampling gains saturate as the verifier's precision binds (Brown et al., 2024) — the phenomenon my best-of-8 → best-of-16 plateau (+0.7 pts) exhibits directly.

Verify-then-revise. A second pass that checks and repairs text is broadly supported: post-hoc correction with verifiers (Peng et al., 2023), tool-interactive critique (CRITIC, Gou et al., 2023), self-refinement (Madaan et al., 2023), chain-of-verification (Dhuliawala et al., 2023), targeted factual error correction (Cao et al., 2020), collaborative editing (PEER, Schick et al., 2022), and instruction-tuned editors (CoEdIT, Raheja et al., 2023). Tyen et al. (2024) add the caveat that models often cannot locate their own errors but can fix them once located — which favoured the design where the checker locates the sentence and the editor only rewrites it. That was the H32b experiment, and Section 5.9 reports what happened when the 9B editor was handed the located errors: it returned near-copies, and the arm changed nothing (Δ0.0, p = 1.0).

Two further literatures bear on specific failures: fine-grained correctional feedback aligning better than whole-response preferences (RLHF-V, Yu et al., 2023 — consistent with my DPO replication failures), and the finding that fine-tuning on knowledge the model does not hold encourages hallucination (Gekhman et al., 2024 — a caution for any plan to fix contradictions by adding rewritten targets). Parameter-efficient fine-tuning (LoRA, Hu et al., 2022; QLoRA, Dettmers et al., 2023) is used unchanged from common practice; LoRA capacity (rank 4/32/64) proved immaterial to the length pathology, which argues the pathology is data-side. Judge-instrument biases (Zheng et al., 2023; Liu et al., 2023; Dubois et al., 2024; Kim et al., 2023; Gupta et al., 2024) frame the blind-panel null of Section 6.

3Method

3.1Task

An editor model receives a machine draft (written from a brief by Qwen3.5-4B) and must output a passage that (i) keeps every fact of the draft, introducing nothing unsupported, and (ii) matches the draft's length. The instruction used throughout training and generation is a single system prompt:

You are an editor. Rewrite the draft so it reads like a careful human writer: keep every fact from the draft, match its length within 20%, use natural varied prose. Output only the rewritten text.

The user message is Draft:\n<draft>. Length behaviour is measured as the output/draft word-count ratio; the "within 20%" phrasing is a training-time contract, not an evaluation threshold.

3.2Training data construction

Seed corpus. The starting pool is 6,186 (draft, human passage) pairs over 3,260 distinct source documents from five sources: wikipedia-2020 (2,522), crs (1,655), epmc (1,511), postnotes (474), nasa (24). The human passages are the original targets; the drafts were generated by Qwen3.5-4B. The provenance of the human passages and the draft-generation settings predates this project and is not documented in the project records.

Fact-faithfulness rebuild (phase 1). Targets may only use facts present in the draft. The rebuild ran seven registered stages: S1 a regex audit extracting numbers, dates, and names present in the target but absent from the draft, plus a locked-eval document-leak guard (result: 0 of 6,186 pairs dropped; document overlap with the locked evaluation set is zero); S2 per-target-sentence NLI against sliding 300-word draft windows (stride 200; premise = window, hypothesis = sentence; hypotheses capped at 256 tokens so premises always retain at least 253 tokens under max_length 512); S3 classify (clean / rewrite / drop); S4 LLM rewrite of flagged targets (glm-5.3, temperature 0.2, up to 6 attempts; 4,682 calls); S5 NLI re-verification with the identical script; S6 acceptance (zero invented numbers and dates, no sentence with contradiction ≥ 0.5, length ratio of rewrite to original in [0.5, 1.2]); S7 report. The accepted set (v2) is 2,956 of 6,186 pairs, with the invented-fact item share reduced from 78.4% to 0% by the regex rules. A subsequent claim-level check (S4c: each factual claim in the target must be supported by the draft; up to two rewrite rounds) produced the v3 set of 2,020 pairs used for the main retraining phases.

Length-preserving targets (v4lp). The v3 targets are fact-clean but short. The v4lp construction rewrites each v3 target with an external rewriter (glm-5.3, temperature 0.6) under a length-preservation contract — keep the facts, keep the wording close, do not shorten, 90–110% of the draft's words — producing 2,020 rewrites of which 807 pass the acceptance rules (no regex-invented number/date/name, no DeBERTa contradiction ≥ 0.5 against the draft, entailment share at or above the v3 targets' median, word ratio 0.90–1.15). Training on v4lp uses the identical ids, order, and drafts as its control (ctlv4: the unchanged v3 targets restricted to the same 807 ids), which isolates the target text as the only difference.

3.3Editor models and training

Editors are Qwen3.5-9B and Qwen3.5-4B fine-tuned with QLoRA: 4-bit NF4 quantisation with double quantisation, bf16 compute, LoRA rank 32, α 64, dropout 0.05 on q/k/v/o/gate/up/down projections, learning rate 1e-4 with 3% warmup and cosine decay, one epoch, per-device batch 2 with gradient accumulation 8, max_len 2048, dynamic padding with a length-grouped sampler, gradient checkpointing. Standard runs use 124 steps (2,020-pair v3 data) or 50 steps (807-pair v4lp/ctlv4 data), with intermediate checkpoints where a phase requires them. Preference optimisation (DPO, β = 0.1, lr 1e-5, 50 steps × 8 pairs) runs from the v4lp step-50 adapters; the ensemble-pair variant ("T-Edpo", H27) uses the same trainer with 532 best-vs-worst candidate pairs per seed selected by the ensemble mean score. The 27B scale-up (H31) retrains the identical two-stage recipe (same data, LoRA config, step counts, seeds 7/8) on Qwen3.5-27B, 4-bit sharded across both RTX 3090s via device_map = auto with per-device microbatch 1 under a pre-registered OOM ladder; no hyperparameter, datum, or decision rule changed.

Hardware: 2 × NVIDIA RTX 3090 (24 GB), one job per GPU with GPU-UUID checks; the 27B chain spans both. Software: Python 3.12.14, torch 2.12.1+cu130, transformers 5.5.0, peft 0.21.0, trl 0.24.0, bitsandbytes 0.50.2; full versions snapshotted in the project env/ directory.

3.4Decoding defaults and best-of-N

All evaluation generations use the training prompt, temperature 0.7, top-p 0.95, max_new_tokens 1100, batch 8, 4-bit loading, and generation seed equal to the training seed. Best-of-N arms sample N candidates per item at these settings with generation seeds (train seed + 1000k) for candidate k = 0…N−1; candidate 0 reproduces the lane's seeded generation (reproduction-checked at analysis time). A lane's "greedy" baseline is the single sampled pass at temperature 0.7 with the lane seed — the system's standard decoding, not argmax.

3.5Checkers, contradiction indicator, and E-pick

Three NLI models define the automatic faithfulness signal:

• D = MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli (revision b3546ea6…);
• R = FacebookAI/roberta-large-mnli (revision 2a8f12d2…);
• B = facebook/bart-large-mnli (revision d7645e12…) — held out: B is never used for candidate selection, training, or any decision that could contaminate it, and serves as the headline scorer for the reranking results.

An item is contradicted under a checker iff at least one sentence of at least 3 words reaches contradiction probability ≥ 0.5 against any sliding premise window of the draft; the item's scalar score under a checker is its maximum sentence-level contradiction probability.

E-pick (ensemble pick) selects, per item, the best-of-N candidate with the lowest mean of the (D, R) max-contradiction scores; ties break to the lower max(D, R), then to the lower candidate index. E-pick never consults B and never sees the greedy baseline as a candidate. Per-candidate D and R scorings are computed once and pinned by sha256; composites and their scorings are re-used (never recomputed) by later phases, with byte-identity assertions where rebuilt. A precursor variant (phase 18a) selected by D alone under a lexicographic rule; its results are same-family (select and score both under D) and are reported as a precursor, not headline evidence.

3.6Judges

LLM judges are used for pairwise quality gates and the blind-panel check, never for the headline contradiction metric. Pairwise judging is length-matched (the longer text trimmed to the shorter at the nearest sentence boundary), run in both orders, scored 1 / 0.5 / 0; judges are validity-gated (each must prefer a human passage over the draft on the locked set; measured human-vs-draft preference 0.929 for GLM in phase 0, 0.916 GLM / 1.0 Grok in phase 3). The panel evolved by registered decision: GLM (glm-5.3, temperature 0) throughout; Grok (grok-4.7) added after author approval with family exclusion (Grok never judges Grok-generated text); Astra reduced to tiebreaker-only; an external OpenRouter panel used for the blind study of Section 6.4. A calibration exercise (phase 27) added a fourth local judge (JEV) with a confidence cascade: JEV flagged 4.8% of 1,567 units, agreed with 43% of low-confidence escalations, and the cascade would have saved about $53 of $81.6 in API cost — descriptive tooling only.

4Evaluation protocol

4.1Locked evaluation set

All system-level numbers are computed on a locked evaluation set of 120 items drawn from 74 distinct documents (the document is the bootstrap cluster), never used for training. The editor inputs are the untouched Qwen3.5-4B drafts for the 120 locked ids; reference outputs are external rewrites written from the brief (not the draft). A leakage audit verified that 0 of the 6,186 training pairs share a document with the locked set. One caveat is recorded: the frozen dev set had been read by automation before the audit, which the document-level zero-overlap result mitigates but does not erase.

4.2Pooling

Most headline comparisons pool seed-matched lanes: each lane is one trained editor evaluated once on the 120 items, and a paired comparison pools the lanes item by item. Pooling six preference-tuning lanes (seeds 3–8) yields 720 paired items; two-lane comparisons yield 240. All E-pick results use the six-lane pool over lanes trained from the v4lp step-50 adapters (phases 18c–20); the greedy baseline of each lane is its own temperature-0.7 single pass, so every pooled delta is paired at the item level within lanes.

4.3Statistics

Paired differences carry item-cluster bootstrap 95% confidence intervals (clusters = documents or seed-lane items; 10,000 resamples, numpy seed 18; early judge-interval analyses used 2,000 resamples, seed 0, and are labelled as such) and exact McNemar tests on the discordant counts. Length is the median output/draft word ratio per system (paired medians for paired contrasts). Repetition loops count outputs with more than 20% duplicate sentences. Invented facts are regex-extracted numbers/dates/names present in the output but absent from the draft.

4.4Decision rules and pre-registration discipline

The unit of research is the phase. Before any GPU or API work for a phase begins, its hypothesis (numbered S… or H…) and its decision rule are committed to git in the project research log; the rule states the metric, the pool, and the pass condition. The default pass condition is supported iff the pre-registered CI lower bound is greater than zero; several phases instead use eligibility rules that must hold on both seeds (H15's paired-median rule, H16's per-seed ratio gates, H17's non-degradation bar), stated verbatim where the phase is discussed. Descriptive arms registered as never gating are labelled as such. Each phase closes with a machine-written SUMMARY.md and stats.json and a result commit; all numbers in this paper are transcribed from those artifacts. Reproduction checks (recomputed counts vs published counts of re-used artifacts) are run by the analyzers; no mismatch occurred in the headline phases. The two documented pre-registration slips are disclosed in Section 7.3.

5Results

5.1Filtering causes length collapse, and fine-tuning learns it

Editors fine-tuned on the filtered v2/v3 targets produce median output/draft ratios of about 0.63–0.69 (v2 reference lanes: 0.6451 seed 1, 0.6421 seed 2), while NLI-measured faithfulness improved over the pre-rebuild baselines (phase 3: 31–36 contradicted items per 120, versus 49–55 for the pre-rebuild SFT baselines). The base model without any adapter, decoded under the identical configuration, has median ratio 0.981: the deletion behaviour is not a property of the base model or the decoding configuration.

The dose–response experiment (phase 5, supported) pins the collapse on fine-tuning itself. Checkpoints of a single seed-1 9B run:

Both pre-registered checks passed (a monotone-enough decline from base through step 62; a plateau after step 62). NLI contradiction shares move non-monotonically over the same checkpoints (26.7 / 35.8 / 23.3 / 30.0 / 32.5%), so the trade is length against judged human-likeness (GLM vs-draft rises from 0.555 at base to about 0.81 by step 62), not length against faithfulness. The same dose–response failed to replicate on 4B (phase 7: the fine-grained check failed on both seeds), so the fine-grained dynamics are scale-sensitive even though the collapse itself appears on both sizes.

5.2Length-preserving targets (v4lp) repair the collapse

The v4lp repair was evaluated under three pre-registered hypotheses after an initial mixed result (H14: seed 1 eligible, median 0.938 and GLM vs-draft 0.857; seed 2 failed its NLI bar — reported as not supported under its own rule, which is why the repair's evidence rests on H15–H17 instead).

H15 (matched control; supported). The control ctlv4 trains on the unchanged v3 targets over the identical 807 ids, order, and drafts, for the identical 50 steps. If the length gain came from the selection of items or the shortened schedule rather than the rewritten targets, ctlv4 would match v4lp. It does not; both seeds pass the rule (paired median difference ≥ +0.1 and control median < 0.75):

The v4lp editors eliminate sub-half-length outputs (0.0% on both seeds, versus 15.8% for the controls) and repetition loops (0 versus 7–10), and cut invented facts by two orders of magnitude relative to the reference lanes.

H16 (4B replication; supported). On the 4B base, v4lp training gives median ratios 0.949 (seed 1) and 0.944 (seed 2); both seeds passed the pre-registered gates. The repair is not a 9B artefact.

H17 (fresh seeds, faithfulness non-degradation; supported). Because rewriting targets could in principle reintroduce content problems, H17 pre-registered a one-sided bar: pooled over fresh seeds 3–4, the paired NLI-contradiction difference (v4lp − ctlv4) must be ≤ +5.0 points. Result: 78 vs 71 contradicted items per 240; difference +2.92 points, 95% CI [−3.25, 9.08], exact McNemar p = 0.427. The bar is met on the pre-registered point estimate; the CI's upper end exceeding the bar is a power statement, not a gate failure.

Together H15–H17 establish that the length collapse induced by fact-faithful filtering is repaired by making the targets themselves length-preserving, on both model scales and fresh seeds, without a pre-registered faithfulness regression under the NLI check used at the time. The NLI-family caveat applies and is discussed in Section 7.

5.3Decode-time ensemble reranking (E-pick best-of-N)

The second result is a decode-time intervention on the finished v4lp-derived editors. Six preference-tuning lanes (DPO, seeds 3–8, all initialised from v4lp step-50 adapters) provide the pooled baseline: their temperature-0.7 single passes ("greedy", arm A) contradict 497/720 items under the held-out scorer B (69.0%). E-pick selects among N sampled candidates by the mean (D, R) contradiction score; B scores the selected text. No training is involved.

All deltas are greedy minus E-pick under B, pooled over the six lanes, item-cluster bootstrap (10,000 resamples, seed 18) and exact McNemar. Per-lane E-pick bo8 counts (contradicted /120 under B): 83→73, 85→71, 79→76, 83→78, 81→69, 86→79 — the improvement is consistent in direction across all six lanes rather than driven by outliers. Median output/draft ratios are unchanged by picking (0.918–0.943 greedy versus 0.921–0.939 E-pick bo8 per lane): the reranker does not buy faithfulness by shortening.

Scaling from best-of-4 to best-of-8 roughly doubles the gain (3.3 → 7.1 points); best-of-16 adds only +0.7 on top of best-of-8 (441 vs 446), so the candidate-quality distribution saturates well before N = 16.

Two descriptive rows from the same analyses bound the circularity concern (Section 7.1). Under the picker-family scorer D, the same E-pick bo8 composites show Δ16.5 points (CI [13.8, 19.3]); under R, Δ14.4 (CI [11.8, 17.1]) — larger than the held-out B estimate of 7.1, as expected when selection correlates with the scorer's family. The held-out estimate is the headline; the blind-judge evidence bearing on it is in Sections 6.4 and 7.

5.4The precursor chain, and why it is not the headline

The study's most eye-catching numbers came first and are circular; I report them with that label. On the two v4lp lanes (240 items), with candidates selected by the DeBERTa checker D and scored by the same D: the v4lp baseline contradicted 78/240 (32.5%); DPO training (H18c) took the pooled greedy to 26.2% (a Δ of 6.2 points, CI [1.2, 11.2], p = 0.028 — the initial success that later failed replication, Section 6.2); and DPO plus best-of-8 reranking reached 9.6% (H18e, descriptive). Picker and scorer share the checker, so the reduction is partly mechanical: contradictions as the selector's own instrument measures them can be resampled away. The honest form of that chain is the held-out design of Section 5.3, where the same best-of-8 machinery moves a disjoint scorer by 7.1 points from a 69.0% baseline — and even that remains within the NLI paradigm (Section 7.1).

5.5Scaling the editor to 27B (H31): not supported

After the thirty-phase core, the one lever never moved was model scale. H31 (pre-registered 23 October 2026) retrained the identical recipe — v4lp SFT then LoRA-DPO, same data, same LoRA config, same 50-step schedules — on Qwen3.5-27B for seeds 7 and 8, and compared the 27B greedies to the seed-matched 9B greedies under B on the same 120 locked items per lane (240 paired items).

Per-lane, the 27B is numerically better by 4 and 3 items of 120 under B — but the pre-registered pooled test says this is noise:

The success rule (pooled paired delta, 95% CI lower bound > 0) is not met: H31 verdict: not supported. The most instructive secondary is the training-loss table (same data, same seeds, hence same validation splits):

The 27B fits the training distribution better on every lane and loss — and contradicts no less under the held-out scorer. Scale, in this pipeline, converts into fit, not into factual consistency of the edits. The standing conclusion is unchanged: decode-time selection (E-pick bo8 at 9B: 7.9 pts, CI [2.5, 13.3], p = 0.0054 on these very seeds) remains the only robust lever found; three times the parameters at lower training loss bought nothing a pre-registered test could detect.

5.6A third picker family, twice a null: ELECTRA (H28) and AlignScore (H32c)

H28 tested adding an ELECTRA-large-mnli checker to the picker (E3 = {D, R, X}, scored by B): 441 vs 446 of 720 (Δ0.7, CI [0.0, 1.4], p = 0.125) — a trivial difference, decisively below the bar. The registered interpretation was that a third within-NLI-encoder family adds nothing.

H32c (pre-registered 24 October 2026, executed the same day) tested the concept properly with a family that is not a stock encoder NLI: AlignScore-base (official checkpoint, sha-pinned; a RoBERTa alignment function trained on 4.7M examples across seven tasks), inserted into the picker as E2b = mean{D, AlignScore} (primary, maximal family distance) and E2a = mean{D, R, AlignScore} (descriptive). The family-distance manipulation check confirmed the premise — candidate-level Pearson correlations with the standing members were D–A 0.268 and R–A 0.388 (against D–R 0.428), so AlignScore genuinely is distant. Result under the held-out scorer B on the 720 pool:

H32c verdict: not supported. Replacing R with AlignScore is directionally worse (the picker flips 34 items out and 52 in, net −18); adding it on top is neutral. The family distance holds, but the distant member lacks the per-sentence precision R contributes. Two checkers from disjoint NLI families appear both necessary and sufficient for the transfer measured in Section 5.3 — and the registered QAGS follow-up (a QA-paradigm checker) was not triggered.

5.7Absolute levels and what remains unsolved

The reranking gains are relative. After E-pick bo8, 446/720 items (62.0%) still carry at least one B-flagged contradiction; after bo16, 441/720 (61.3%). Checker families also disagree on levels: the same greedy baselines flag 203/720 under D and 399/720 under R, versus 497/720 under B — level comparisons are meaningful only within a scorer, which is why all gated deltas are paired and within-scorer. I conclude that ensemble reranking reliably reduces NLI-measured contradiction, and that it does not come close to solving it.

5.8H32a: claim-decomposed E-pick — not supported

Design as pre-registered (24 October 2026, before any H32 GPU work; pre-registration commit e71815b). The per-candidate picker score becomes the mean over atomic claims — each candidate's sentences split into single-fact fragments by a frozen deterministic rule (split on semicolons, colons, dashes, and coordinating conjunctions; fragments under 3 words merged back) — with each claim scored by D and R against the same sliding draft windows, instead of the max over sentence-pair probabilities. Selection still by lowest mean{D, R}; candidate set = the existing best-of-8 candidates, reused read-only. Arms: X = standing sentence-level E-pick bo8 composites (reused with their existing B items); T = claim-level composites on the same candidates. Pools: 720 paired items primary, the H31 240-pool secondary. Test: pooled paired delta under B, item-cluster bootstrap (10,000, seed 18), exact McNemar; supported iff the CI lower bound > 0. Pre-registered downgrade gate: if the median claim count of T-selected texts differs from X-selected by more than 20% relative, the primary is downgraded to descriptive.

Result (written 25 October 2026, 00:08 AWST, by the pre-registered analyzer; reported as-is, no re-spin). The claim-count gate passed comfortably — median claims of T-selected vs X-selected texts differed by at most 3.5% per lane (for example 29.5 vs 30.0 on s3) — so the primary stayed a gating test. It failed:

Per-lane under B: X 73/71/76/78/69/79 → claim-level 73/78/79/77/74/80 (of 120); the picker changed 85–97 of 120 picks per lane. H32a verdict: not supported. The claim-mean signal is not merely unhelpful, it is mildly harmful: switching the aggregation from sentence-max to claim-mean flipped most picks and the held-out flag rate rose on five of six lanes. Combined with the best-of-16 plateau (H26, +0.7), the single-family null (H21), and the third-family nulls (H28, H32c), every registered attempt to improve the selection signal has failed; sentence-level max over two disjoint NLI families remains the best picker found.

5.9H32b: verify-then-revise second pass — not supported (exactly nothing changed)

Design as pre-registered (24 October 2026). After the greedy (k0) pass, the standing D+R sentence scoring locates flagged sentences (union view; 439 of 720 items had at least one flagged sentence — set measured and frozen on CPU before any generation: per lane 70/68/76/75/77/73 of 120). For each such item, one extra pass with the lane's own DPO adapter receives the draft, the current text, and the flagged sentences with their indices, and must output the full text with only those sentences repaired and all others copied verbatim (greedy decode, frozen prompt, no sweeps). The revised text joins the candidate pool: E-pick over {k0…k7, revised}, revised scored by D and R exactly like the others; B never consulted at pick time. Items without flagged sentences keep the standing selection. Pre-registered eligibility gates per lane: median revised/k0 length ratio in [0.85, 1.1] and zero revised items below 0.5×; manipulation checks (revised differs on at least one flagged sentence; unflagged sentences present verbatim) reported but never gating. Same pools and statistics as H32a.

Result (written 25 October 2026, 00:08 AWST, by the pre-registered analyzer; reported as-is, no re-spin). All six lanes passed the eligibility gates (median revised/k0 ratio 1.0, zero items under 0.5×), so the primary ran as registered. The pooled paired delta under B is exactly zero:

H32b verdict: not supported. The revision candidate was picked on zero items in every lane (per-lane bo9′ B counts identical to the standing bo8: 73/71/76/78/69/79 of 120), and the manipulation checks show why: the revised text differed from k0 on at least one flagged sentence in only 0–4.3% of a lane's revised items (exactly 0.0 on lane s5), while 98.4–99.7% of unflagged sentences were copied verbatim as instructed. The repair pass did its copying job and, almost always, copied the flagged sentences too; the revise-only texts themselves score one item worse than k0 on the flagged subset (374 → 375 of 439 B-flagged).

The reading is the locate-vs-fix caveat inverted. Locating was solved — the checkers named the offending sentences and the prompt asked only for those repairs — and the 9B DPO editor still returned near-identical text in one greedy pass. A usable repair pass presumably needs a stronger repair model or training on repairs, not just a prompt; with this model, checker-in-the-loop revision buys nothing at 1.19 GPU-hours of revision generation and scoring. (H32a used 2.94 GPU-hours of claim scoring; both arms together 4.13 GPU-hours.)

6Negative results

Negative results are first-class content in this study. The table lists all twenty-three pre-registered hypotheses whose decision rules failed, with the key numbers as logged in the phase artifacts. CIs are item-cluster bootstrap 95%; p is the exact McNemar test; "/720" and "/240" are paired-item pools.

6.1The ledger

Three later nulls sit outside the ledger (registered after it closed): H32a (claim-decomposed picker, Section 5.8), H32b (verify-then-revise, Section 5.9), and H32c (AlignScore picker member, Section 5.6).

6.2Length-side interventions (phases 3–13)

The first block of failures attempted to fix the length collapse without changing the targets. Filtering targets by their own length ratio (S8: cuts at 0.60 and 0.70, retaining 1,767 and 1,203 of 2,020 pairs) barely moved the editors' median output ratio (0.638 / 0.681 vs 0.636): the target's own ratio is a poor proxy for what the editor learns to emit. Repairing targets by content-preserving deletion heuristics (S9, 647 pairs) was likewise inert (0.647) against its matched control. Forbidding the EOS token until 0.85× the draft length at inference degenerated into repetition loops in 48 of 51 outputs. Upweighting long drafts 3× (phase 5 rebal; long share 12.7% → 30.3%) slightly worsened the long-draft quartile (Δ −0.041). Fine-grained early stopping found no eligible checkpoint; the 4B dose–response failed on both seeds; learning-rate reductions (×0.1, ×0.3) produced no eligible arm. An explicit length line in the prompt (H9) gave ratios 0.592 / 0.652 with increased loops; paragraph-chunked decoding with stitching (H10) gave 0.725 / 0.708 with roughly 12–13% of contradictions arising at chunk boundaries. Filtering targets harder (H12 ratio ≥ 0.80; H13 ratio ≥ 0.75) never made either seed eligible. The consistent picture: nothing on the prompt, schedule, optimisation, or decoder side recovers length; only changing the target distribution (v4lp) did.

6.3Training-side interventions (phases 14, 18–20, 26)

The DPO story is the study's clearest replication failure. H18c passed (Δ6.2, CI [1.2, 11.2], p = 0.028; pooled 26.2% vs 32.5% baseline), and both pre-registered replications failed (H19 Δ1.2, CI [−3.8, 6.2]; H20 six-seed pool Δ2.8, CI [−0.1, 5.7] — the lower bound grazing zero). The initial success is attributed to noise in pair construction or seeds rather than a reliable effect, and DPO is treated as not established. Strictly re-filtering the targets a second time (v5fc, H18b) was directionally fine but decisively inconclusive (Δ2.9, CI [−1.2, 7.1]). Combining DPO with best-of-4 decoding (H18d) did not clear its gate over the decoding-only arm. The most direct attempt to bake the ensemble signal into training — T-Edpo, DPO on ensemble-mean best-vs-worst pairs over 532 train-prompt candidate pairs per seed (H27) — missed its CI (Δ5.0, CI [−0.8, 10.8], p = 0.1189) on the 240 pool, and its greedy arm (156/240) clearly loses to plain decode-time E-pick bo8 on the same items (144/240). The gain resists training-in so far; the diagnosis (consistent with fine-grained-feedback results in the literature) is that whole-response preferences are too blunt for sentence-level contradictions.

6.4Judges: the blind-panel null (phase 22)

The most consequential null concerns the headline claim itself. H22 presented the greedy (A) and E-pick bo8 (B) texts of each item to an external LLM panel blind to arm assignment, asking which text better preserves the draft's facts. The panel preferred A in 220 and B in 214 of 720 items (Δ0.8, CI [−2.6, 4.2]) where the held-out B scorer sees Δ7.1. Three readings are live: (i) the NLI-measured gain is partly or wholly NLI-family-specific (picker and scorer share the NLI paradigm even though no model is shared); (ii) the panel is insensitive to the contradiction patterns the checkers flag; (iii) the checkers flag stylistic or spurious patterns that are not human-meaningful contradictions. The pre-registered family-breaking study (blind judges over arms A/B/C, phase 25j) and the author's hand labels were still open when this was written. Until they land, the reranking result stands as "reduces NLI-measured contradiction under a held-out NLI family", not as an established improvement in human-judged faithfulness.

6.5Picker construction (phases 21, 28; H32c)

H21 tested whether the picker must be an ensemble at all: selecting best-of-4 by R alone and scoring under D produced 208/720 vs greedy 203/720 (Δ −0.7, CI [−3.6, 2.2]) — single-family picking does not transfer across checker families (contrast the ensemble's 473/720 under held-out B in H23). H28 and H32c (Section 5.6) tested third families and found nothing; H32a (Section 5.8) tested finer-grained claim scoring and found it slightly harmful. Two checkers from disjoint families, scoring whole sentences, appear both necessary and sufficient for the transfer measured.

6.6Decode-side interventions (phases 28–29)

A "preserve facts" instruction appended at decode time (H29) moved the B-contradicted share from 497 to 488/720 (Δ1.2, CI [−2.1, 4.6], p = 0.515), with the manipulation check passing (decoding genuinely changed on every lane). Lowering the sampling temperature (H30, primary arm 0.3) gave 488/720 (Δ1.2, CI [−1.9, 4.3], p = 0.488), with the descriptive sweep non-monotone and flat (0.7: 497; 0.5: 499; 0.3: 488; 0.1: 501). Sampling less does not buy faithfulness here; selection among samples does.

6.7Scale-up (H31)

Covered in Section 5.5: better fit on every recorded loss, no detectable contradiction reduction (Δ2.9, CI [−2.1, 7.9], p = 0.3368); the 27B composite does not beat the 9B composite (−1.7, CI [−6.2, 2.9]). Combined with the decode-side nulls, the ledger says: nothing that changes where the text comes from (bigger model, lower temperature, longer instruction) has passed a pre-registered test; the only lever that has is selection among samples.

7Threats to validity

7.1Checker-family circularity (the open one)

The E-pick picker (D, R) and the headline scorer (B) share the NLI paradigm though no model or training set. The held-out-family design reduces model-level contamination but cannot rule out family-level correlation: the same E-pick bo8 composites show Δ16.5 under D and Δ14.4 under R, versus Δ7.1 under B. The blind LLM panel (H22) saw Δ0.8. The pre-registered family break — blind judges over arms A/B/C — remains open: external-panel credits stopped on 7 October and Grok rows are missing. Every reranking number in this paper should be read with that qualification attached. The precursor chain of Section 5.4 is circular at the model level as well and is reported only for completeness.

A second-order form of the same threat applies to the training-side arms: the v5fc filter and the DPO rejected side were constructed to satisfy DeBERTa, and DeBERTa-family scoring defined the early metrics (phases 18b–20). Held-out-B scoring is the mitigation; it is why the H23/H24/H26 lines, not the phase-18 numbers, carry the paper.

7.2Judge self-preference and panel reliability

GLM wrote the training-target rewrites and is one of the judges; GLM also wrote the v4lp targets. Family exclusion (Grok never judges Grok text) handles the clearest case, but a GLM-lenient direction was measured directly: on the 50-pair v3 data spot check, three-judge Fleiss kappa was −0.223 (worse than chance), with per-judge ok counts Astra 6, GLM 42, Grok 1 — GLM, which wrote the rewrites, is the lenient outlier. Judge results in this paper are reported per judge, never pooled, and no headline claim rests on a GLM-only number. The Astra/Grok backends run through agentic CLIs without temperature control, so their verdicts are not guaranteed reproducible; all judges share one rubric, so rubric errors are correlated.

7.3Pre-registration slips and deviations

Two registrations missed strict pre-registration and are disclosed: the length-threshold ablation's rule was committed 1 minute 48 seconds after its training gate started (no adapter or output existed at commit time; every later phase committed before launch), and one mid-run script edit by a concurrent session was contained by sha reconstruction and output-side attestation with a pre-committed rollback. Protocol deviations (a mid-phase resume with --no-early-abort, three overnight OOM fix cycles in phase 18c, an invalid base-model generation_config that applies equally to every compared system, judge-credit outages) are logged in the project deviation record with commits; the phase summaries note every table they touch.

7.4Scope, power, and multiplicity

Evidence rests on one task, one corpus family, 120 locked evaluation items pooled to at most 720 paired items, 4B/9B/27B models of a single family, and LoRA fine-tuning only. Wide intervals (roughly ±0.09 on 120-item judge shares) are the norm. With thirty-plus registered hypotheses, one or two borderline supports would be expected by chance alone; this is why replication is required before a result is treated as established (fresh seeds for v4lp; six-lane pooling for E-pick), and why the DPO initial success was demoted after failing replication. Six trained length-selection arms plus untrained candidates constitute a forking-paths set for the length threshold; the both-seed requirement and fixed thresholds limit, but do not remove, that risk.

7.5Residual specifics

The reference rewrites were written from the brief rather than the draft, so they are not a like-for-like editor comparison. The human targets themselves invent relative to the draft (51.1% of items by the number regex), so the reference ceiling is not fact-faithful by construction. Long drafts are under-represented in training relative to the locked evaluation set, and every system's length shortfall concentrates in the long-draft quartile. Provider dependence shaped what could run (judge credits, API availability); no result depends on a provider's outputs at scoring time except the judge-side analyses flagged above. All human gates, approvals, and pending hand labels are by one author.

8What would make it usable

The editor in its current state is not usable unattended: after the strongest intervention, about 62% of outputs still carry a machine-flagged contradiction, and the one confirmed lever reduces an automatic measure whose human-meaningfulness has not been demonstrated. In priority order, the missing pieces:

1. The family break (registered, pending). A blinded judge family that shares no paradigm with the NLI checkers, scoring the greedy vs E-pick arms, plus the author's hand labels on contested units. This decides whether the 7.1-point gain exists for any instrument other than NLI. It is the highest-information, zero-GPU experiment in the queue, and no deployment claim should be made before it lands.
2. Checker precision against humans. The flagged-share scale (69% → 62%) is only meaningful if flagged items correspond to human-recognised contradictions at a known rate. The 20-label calibration sheets are the minimum; a larger stratified labelling round is the real fix.
3. A better picker signal, not more samples. Best-of-16 is exhausted (+0.7 over best-of-8); the verifier, not coverage, is binding — and every registered attempt to improve the verifier has now failed: a third family twice (H28, H32c), claim-level scoring (H32a, slightly worse), and checker-located repair (H32b, no change: the 9B editor returned near-copies when handed the offending sentences). What remains untried is a genuinely different checker paradigm (QA-overlap rather than another encoder NLI or aligned RoBERTa) and a repair model strong enough to actually rewrite the located sentences.
4. Absolute levels, then product gates. Usability means a flagged share low enough that a human spot-check pass is cheap, plus a calibrated estimate of what the flag means. Neither exists yet; both are measurable with the existing artifacts.
5. If the family break confirms the gain: bake selection into training (RAFT-style winner fine-tuning is the registered next form, after T-Edpo missed), and re-test scale only after the picker signal improves — H31 showed scale alone converts to fit, not consistency.

9Reproducibility

All work ran on one machine (2 × RTX 3090, Ryzen 9 5900X, 60 GB RAM) under per-GPU job isolation with UUID checks. The project repository holds, per phase: the pre-registration entry (git-committed before compute), the machine-written SUMMARY.md and stats.json, generation manifests (temperature, seeds, row counts, GPU UUIDs), sha256 manifests for data, outputs, and adapters, and the full environment snapshot (pip freeze, versions, hardware). The research log is append-only with a logged backup discipline; every deviation and bug fix is an entry with commit hashes. Seeds: generation seeds follow training seeds (candidates at train seed + 1000k); bootstrap seed 18 (10,000 resamples); early judge intervals seed 0 (2,000 resamples). Known non-reproducible: commercial API outputs (GLM rewrites and judging; CLI judges without temperature control), and phase-0 timings measured on a GPU shared with a game stream. The locked evaluation set, its drafts, and all dataset shas are in the artifact manifest. A NUMBERS.tsv ships alongside this paper mapping every number above to its source file.

References

Verified against the project literature review (soho-ev2-lit-h32/LITERATURE_MAP.md); arXiv identifiers link-checked, logs in the project repository.

1. Brown et al. (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.
2. Cao et al. (2020). Factual Error Correction for Abstractive Summarization Models. EMNLP.
3. Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems.
4. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS.
5. Dhuliawala et al. (2023). Chain-of-Verification Reduces Hallucination in Large Language Models.
6. Dubois et al. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators.
7. Durmus et al. (2020). FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization. ACL.
8. Gao et al. (2023). Scaling Laws for Reward Model Overoptimization.
9. Gekhman et al. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? EMNLP.
10. Gou et al. (2023). CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. ICLR.
11. Gupta et al. (2024). Judging the Judges.
12. Honovich et al. (2022). TRUE: Re-evaluating Factual Consistency Evaluation. Findings of EMNLP.
13. Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
14. Kim et al. (2023). Prometheus: Inducing Fine-grained Evaluation Capability in Language Models.
15. Kryściński et al. (2020). Evaluating the Factual Consistency of Abstractive Text Summarization (FactCC). EMNLP.
16. Laban et al. (2022). SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization. TACL.
17. Lee et al. (2022). Factuality Enhanced Language Models for Open-Ended Text Generation. NeurIPS.
18. Liu et al. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.
19. Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS.
20. Maynez et al. (2020). On Faithfulness and Factuality in Abstractive Summarization. ACL.
21. Min et al. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP.
22. Nakano et al. (2021). WebGPT: Browser-assisted question-answering with human feedback.
23. Ouyang et al. (2022). Training language models to follow instructions with human feedback.
24. Peng et al. (2023). Check Your Facts and Try Again. Findings of EMNLP.
25. Rafailov et al. (2023). Direct Preference Optimization. NeurIPS.
26. Raheja et al. (2023). CoEdIT: Text Editing by Task-Specific Instruction Tuning.
27. Roit et al. (2023). Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback. EMNLP.
28. Schick et al. (2022). PEER: A Collaborative Language Model.
29. Song et al. (2024). VERISCORE. Findings of ACL.
30. Stiennon et al. (2020). Learning to Summarize with Human Feedback. NeurIPS.
31. Tyen et al. (2024). LLMs cannot find reasoning errors, but can correct them given the error location.
32. Wang et al. (2020). Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (QAGS). ACL.
33. Wei et al. (2024). Long-form factuality in large language models.
34. Yu et al. (2023). RLHF-V.
35. Zha et al. (2023). AlignScore: Evaluating Factual Consistency with a Unified Alignment Function. EMNLP.
36. Zhang et al. (2019). Optimizing the Factual Correctness of a Summary. ACL.
37. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.

Corrections and errata will be posted alongside this document; the append-only project log is the authoritative record.

Cite this

To cite this paper, use the following entry.

@techreport{canarybuilds-2026-editor-v2,
  title       = {Fact-Faithful Editing with Ensemble Reranking: A Pre-Registered Study},
  year        = {2026},
  month       = {10},
  institution = {Canary Builds},
  url         = {https://canarybuilds.com/papers/fact-faithful-editing-ensemble-reranking}
}