Kuku Yalanji v21 Independent Model Comparison

Date: 2026-07-12 UTC
Comparison author: Codex, auditing both sealed lanes from their stored artifacts
Model A: v21.1-codex-synthetic-direct-gvn-3epoch-lr2e-5
Model B: v21.2-claude-balanced-replay-gvn-3epoch-lr2e-5
Status: research comparison; no elder certification or public-routing authorization

1. Decision in one paragraph

Claude's v21.2 model is the better research candidate of record. It is statistically indistinguishable from the Codex v21.1 model on the primary tagged synthetic test by paired mean sentence chrF++ (+0.082, 95% CI [-0.225, +0.396]), while retaining much more of the replayed domains: +12.74 corpus chrF on held-out dictionary usage and about +15.2 on each held-out Bible condition. This is a strong practical Pareto improvement. It is not evidence that v21.2 is a competent general Kuku Yalanji translator: both models score 0/43 exact on the elder-shared set, produce only about 68% of the reference token length there, and remain far below the natural-register target. The current public service must therefore remain research-labelled and retrieval-first; replacing one unrestricted generative route with the other would not solve the central linguistic failure.

2. What was compared

The comparison uses sealed greedy prediction archives. It does not trust either lane's prose report and does not rerun inference. Every shared evaluation was accepted only after the ordered (id, input, reference) triples and decoder settings matched exactly.

Shared held-outs:

The treatment-specific 1,024-row train samples and 128-row memorization gates were excluded. Their hashes and membership differ, so comparing their scores would be invalid.

3. Experimental controls and non-controls

The following controls match:

• frozen v12 merged base checkpoint, including identical model.safetensors, tokenizer, and config hashes;
• frozen v12 architecture derived from facebook/nllb-200-distilled-1.3B and the eng_Latn -> gvn_Latn direction;
• seed 42;
• LoRA rank 64, alpha 128, dropout 0.0, and the same six target-module families;
• batch 8, gradient accumulation 2, effective batch 16;
• 3 epochs, learning rate 2e-5, no warmup, and no weight decay;
• source/target caps 192/208;
• greedy decoding: one beam, no n-gram ban, repetition penalty 1.0, length penalty 1.0;
• PyTorch 2.4.1+cu124, Transformers 4.48.3, PEFT 0.19.1, SacreBLEU 2.6.0.

The training treatment differs:

Claude's builder found ten replay/evaluation collisions before training: one synoptic Bible surface and eight DB base rows, represented as ten quarantined task rows and 34 would-be expanded training rows. They were preserved in quarantine/replay_overlap.jsonl and excluded. The effective replay mixture therefore passes the source, target surface, pair, canonical-reference, and dictionary word-ID separation gates.

There are two important non-controls:

1. v21.2 has 4,155 optimizer steps versus 3,120 for v21.1, about 33% more. The comparison identifies the better training recipe, but cannot attribute every difference solely to replay composition rather than update budget.
2. v21.1 trained on an RTX 4090 and v21.2 on an A40. The software stack matches, but this is not a same-hardware, multi-seed determinism experiment.

Accordingly, "replay dominates" is justified as an operational/Pareto result, not as a fully isolated causal estimate of replay data.

4. Statistical method

• Corpus BLEU and corpus chrF++ are recomputed with SacreBLEU 2.6.0; chrF++ uses word_order=2.
• The primary paired estimand is model B minus model A in mean per-sentence chrF++.
• Confidence intervals use 50,000 paired nonparametric bootstrap resamples, with deterministic seeds derived from seed 42.
• Win/tie/loss counts compare sentence chrF++; exact two-sided sign tests remove ties.
• Exact-match discordance uses the exact McNemar/binomial test.
• Slice intervals use 20,000 paired resamples and are descriptive, unadjusted for multiple comparisons.
• Whitespace is collapsed before exact comparison; Unicode and punctuation are otherwise preserved.

BLEU/chrF are reference-overlap metrics, not grammaticality or community acceptability judgments. This report does not treat a high automatic score as elder certification.

5. Main results

All fourteen model/set combinations have zero empty outputs.

5.1 Synthetic performance is parity, not a clean win

On tagged test rows, the models emit identical text on 816/1,606 rows. v21.1 has the higher sentence-chrF score on 425 non-tied rows and v21.2 on 360, but v21.2 has more exact matches (426 versus 416) and larger positive outliers. That distribution explains why corpus chrF favors v21.1, BLEU/exact favor v21.2, and paired mean sentence chrF is essentially zero. The correct conclusion is synthetic parity with heterogeneous row-level trade-offs.

Untagged input slightly favors v21.2 in mean sentence chrF (+0.385), though corpus chrF still favors v21.1 by 0.58. This metric disagreement is another reason not to rank the models on one scalar.

5.2 Replay strongly retains adjacent domains

The dictionary-usage gain is broad: v21.2 wins 64/84 rows by sentence chrF, versus 20 for v21.1. Its mean output length ratio is 0.95 rather than v21.1's pathological 1.53. On Bible direct, v21.2 wins 303/325 rows; on Bible reference-conditioned, 301/325. Both paired confidence intervals exclude zero by wide margins.

These controls are surface-disjoint from replay after quarantine, so this is not direct held-out memorization. Nevertheless, they come from the same source domains and are best interpreted as domain retention, not general language transfer.

5.3 Elder/natural-register transfer remains unsolved

Both models score 0/43 exact and under-translate severely:

The paired chrF interval includes zero. v21.2's small point advantage is not evidence that replay repaired elder syntax, discourse structure, information density, or natural lexical selection. Both systems compress multi-clause English prompts into shorter, normalized clauses and omit relational or discourse material present in the elder references.

6. Linguistic and corpus-structure diagnostics

6.1 Annotated grammar slices

On the tagged synthetic test, the strongest descriptive v21.2 regressions are:

Ergative (+0.06), ablative (+0.01), and subordinate (+0.07) rows are effectively tied. Imperatives have a +1.35 point estimate for v21.2, but the interval [-0.88, +3.67] crosses zero. These slice intervals are exploratory and not multiplicity-adjusted; they identify review targets rather than confirmatory discoveries.

6.2 The synthetic test is internally stratified

Mean sentence chrF rises from about 44 in the earliest source-order quartile to about 89 in the latest. The latest rows are dominated by short, highly regular lexical grids, while early rows contain more varied constructions and the longest authored clauses. Tier B scores much higher than tier A because tier labels correlate with those authoring phases and templates. This is a property of the evaluation distribution, not evidence that B-quality language is intrinsically easier or better.

The comparison therefore reports both synthetic and elder controls. A 54 corpus chrF on this synthetic test cannot be read as 54 chrF on unconstrained natural Kuku Yalanji.

6.3 Lexical and morphological contrasts

Paired examples show genuine specialization trade-offs:

• For "Father is deaf", v21.1 exactly reproduces Nganjan milka-kari; v21.2 substitutes dukul-kari, replacing the reference's ear-based privative with a head-based form.
• For "The woman went to the scrub", v21.2 exactly reproduces Jalbu madja-nga dungan; v21.1 selects a different place lexeme (jikan-ka).
• For "Mother made the clothes clean", v21.2 recovers the causative reference dayirr-bungan; v21.1 emits a different verb (julurrin).
• For "The spear, the man got", v21.1 preserves the marked agent/topic form dingkar-angka; v21.2 reduces it to bare dingkar, losing reference morphology.

These examples are reference-agreement observations. A speaker must decide whether alternatives are grammatical, semantically equivalent, dialectal, or wrong.

7. Degeneration and output-shape safety

Average scores conceal repetition failures. A generic orthographic segment detector, splitting words and hyphen chains without a language-specific denylist, found:

Thus replay shifts rather than eliminates failure modes. It nearly removes v21.1's dictionary/Bible over-generation, but v21.2 is more prone to long synthetic morpheme loops. One v21.2 output repeats -mu dozens of times; another repeats yindu- until the generation limit. These are hard safety failures even when corpus averages are acceptable. A production candidate needs explicit degeneration gates and a re-evaluation after any decoding guard is added.

8. Compute and archive record

The GPU/runtime comparison is operational only: the hardware and end-to-end pod lifetimes differ. It must not be used as a model-quality claim. Both merged models and adapters have distinct hashes, and both full archive seals and prepared-dataset seals passed locally on 2026-07-12.

9. Limitations

1. One training seed and one run per treatment do not estimate run-to-run variance.
2. Update count and hardware are not matched.
3. Bible and dictionary held-outs measure adjacent-domain retention, not unseen-domain transfer.
4. The elder set has only 43 rows and no independent blind speaker ratings in this comparison.
5. Synthetic references inherit the corpus's own constructions, authoring phases, and possible errors.
6. chrF/BLEU reward surface overlap and can penalize valid variants or reward ungrammatical overlap.
7. Exploratory grammar-slice intervals are not corrected for multiple comparisons.
8. Neither model was evaluated here for reverse-direction translation, calibrated uncertainty, harmful output, or robustness to arbitrary public input.

10. Recommendation

1. Preserve v21.1 as the synthetic-only ablation and designate v21.2 as the next research candidate; do not delete or overwrite either archive.
2. Do not promote v21.2 to an unrestricted public translator. Its elder control and degeneration failures fail the linguistic and safety bars.
3. Freeze a new natural-register held-out before consuming any additional elder/photo-story material for training.
4. Run a step-matched ablation: replay at 3,120 updates and synthetic-only oversampling at 4,155 updates, preferably over multiple seeds. This separates replay composition from extra compute.
5. Add automatic generation gates for morpheme loops, severe under-translation, and length-limit termination; then rerun the exact frozen battery.
6. Have Kuku Yalanji speakers blind-review the same paired outputs for adequacy, grammaticality, morphology, dialect/register, and cultural acceptability before any promotion decision.

11. Reproducibility

Machine-readable result:

analysis/v21.1-codex-vs-v21.2-claude-2026-07-12/comparison.json
SHA-256 066ba9dc284fd568f8d45bd29f3495c89af02c9e6d9a0ca0cfa6bb522a033e24

Analyzer:

/mnt/donto-data/workspace/mobtranslate.com/training/translation/compare_v21_models.py
SHA-256 60ccbe55d945f1f71331578f1b1aca730a9eaaf3aba2da17226a423ab4ce7a2d

Invocation:

PROGRAM=/mnt/donto-data/donto-resources/research/translation-training/kuku-yalanji-runpod-2026-06-30
PY="$PROGRAM/venvs/v14-cpu-py312/bin/python"

"$PY" /mnt/donto-data/workspace/mobtranslate.com/training/translation/compare_v21_models.py \
  --model-a-dir "$PROGRAM/runpod/v21.1-codex-synthetic-direct-20260710T171317Z/models/v21.1-codex-synthetic-direct-gvn-3epoch-lr2e-5" \
  --model-b-dir "$PROGRAM/runpod/v21.2-claude-balanced-replay-20260711T050900Z/models/v21.2-claude-balanced-replay-gvn-3epoch-lr2e-5" \
  --output "$PROGRAM/analysis/v21.1-codex-vs-v21.2-claude-2026-07-12/comparison.json" \
  --bootstrap-replicates 50000 \
  --slice-bootstrap-replicates 20000

Source archives:

runpod/v21.1-codex-synthetic-direct-20260710T171317Z/
runpod/v21.2-claude-balanced-replay-20260711T050900Z/
prepared/v21.1-codex-synthetic-direct/
prepared/v21.2-claude-balanced-replay/
coordination/v21/claude-incident-20260711-replay-overlap.md

12. Final ruling

Claude's balanced-replay model wins the practical comparison, because it retains adjacent-domain knowledge at no measurable mean synthetic cost. Neither model wins the real deployment test. The decisive next target is natural-register Kuku Yalanji, speaker judgment, and degeneration control, under a cleaner step-matched experiment.