# Kuku Yalanji v22 Step-Matched Replay and v21.2 Decoder-Transfer Report

**Experiment date:** 2026-07-14 UTC  
**Primary treatment:** continue the sealed v21.2 step-2,770 state to exactly step 3,120  
**Primary result:** FAIL; do not promote the v22.0 checkpoint  
**Secondary treatment:** transfer the development-selected decoder to the exact published v21.2 weights  
**Secondary result:** PASS; retain v21.2 weights and use the guarded decoder  
**Language direction:** English (`eng_Latn`) -> Kuku Yalanji (`gvn_Latn`)  
**Validity:** noncommercial research only; not speaker-certified; not authoritative translation

## 1. Decision

This run did not produce a better trained model. The step-3,120 v22.0 checkpoint failed its preregistered promotion
gate because greedy decoding still produced 37 tagged-test rows in which an orthographic word or morpheme segment
occurred at least ten times; the maximum was 25. In paired comparison with v21.2 step 4,155, v22.0 was also worse
on both synthetic held-outs and the dictionary-usage control. The v22.0 checkpoint is retained as negative evidence
and is not a downloadable model release.

The run did identify a better inference policy for the existing v21.2 model. A decoder selected using v22.0
development output only was frozen before transfer:

```text
num_beams=1
no_repeat_ngram_size=4
repetition_penalty=1.10
length_penalty=1.0
```

Applied once to the exact published v21.2 model, this policy passed all 14 frozen transfer checks. On the tagged
synthetic test, corpus chrF++ rose from 53.8374 to 55.7097 and repeated-segment rows fell from 37 to 2. The
unchanged v21.2 weights remain the model of record. This is a decoding-policy improvement, not a new model.

## 2. Research question and preregistration

The primary question was whether v21.2 had been over-exposed to its balanced replay mixture. The falsifiable
hypothesis was that stopping at step 3,120 would reduce degeneration while retaining synthetic, dictionary-usage,
and Bible performance. Before any v22 output existed, the following were frozen:

- exact base, checkpoint, optimizer, scheduler, RNG, tokenizer, data, and code hashes;
- seed 42 and unchanged v21.2 hyperparameters;
- an exact stop at optimizer update 3,120, with best-checkpoint reload disabled;
- a four-member deterministic decoder grid selected only on the 1,609-row development set;
- test/control thresholds and an all-checks-must-pass promotion rule;
- retention and publication of negative results;
- explicit exclusion of the 43 elder-shared rows from training and model or decoder selection.

The preregistration was frozen at `2026-07-14T02:15:40Z` with SHA-256
`46c5581e80511679b715d504980297868805e6d35bf46c3cb2fbb4f26846dad4`.

The secondary decoder-transfer protocol was frozen at `2026-07-14T03:29:00Z`, after the primary checkpoint gate
had failed but before any transfer output existed. Its SHA-256 is
`23cc078b0d2a3a8cf681684d3af3086841feed0a819b033b46154f2dc8872989`.

## 3. Experimental controls

### 3.1 Invariants

| Variable | Frozen value |
|---|---|
| Upstream family | `facebook/nllb-200-distilled-1.3B` |
| Project base | exact frozen v12 merged checkpoint with `gvn_Latn` extension |
| Resume state | sealed v21.2 `checkpoint-2770` |
| Training rows per epoch | 22,164 balanced-replay rows |
| Development rows | 1,609 |
| Synthetic test rows | 1,606 tagged plus the same 1,606 without task tags |
| External controls | 84 dictionary usage; 325 Bible direct; 325 Bible reference-conditioned |
| Diagnostic only | 43 elder-shared sentence pairs |
| Effective batch | 16: batch 8, gradient accumulation 2 |
| LoRA | rank 64, alpha 128, dropout 0 |
| Target modules | `q_proj,k_proj,v_proj,out_proj,fc1,fc2` |
| Learning-rate schedule | v21.2 schedule for a 4,155-step horizon; no warmup or weight decay |
| Stop condition | exact global step 3,120 |
| Test visible to trainer | no; manifest records `test_file: null` |

Key input identities include:

| Artifact | SHA-256 |
|---|---|
| Frozen base weights | `26bb5fc5ca75eca215081699e5d8a1dc64f69153431a4e78d6e87ebaee131a0f` |
| Step-2,770 adapter | `47758f5438124144a4f429065f855f34f23db1894b401e9b20ca2deb77cc67fc` |
| Step-2,770 optimizer | `3ae1573412dd896b6d7f1b137473abd52497100155798cfe1c607bb4fec854a1` |
| Step-2,770 scheduler | `692d06781bf1f6b232206b83afc7b736f59200480612622fdf366e967a52435f` |
| Step-2,770 RNG state | `0528adf37a822d037170ddfa2b42ea652e9ada10a543dfc7dc70b4e3d9002486` |
| Train split | `7cb97c1c1d1afd05d0b7ddd04f64180182bf9de261bf5473d07b28bdb43967f0` |
| Development split | `b5f13e3a7c6348d5475a3d045a14d9cefa1b05b3c453cd087e27169098b71e35` |
| Tagged test split | `8ff7cc149dc77ab37c11cc9503ea9b581efd01b03b199b273961734307f9877c` |

The complete 48-file input manifest is archived as `inputs/SHA256SUMS.v22-inputs`.

### 3.2 Reconstruction limitation

The continuation restored optimizer, scheduler, RNG, adapter, and trainer state from step 2,770. It therefore
reconstructs the intended trajectory more closely than a fresh continuation from adapter weights alone. Bitwise
identity to the original, unobserved step-3,120 trajectory cannot be established because the original v21.2 run did
not save that checkpoint. This is one reconstructed seed, not an estimate of training variance.

## 4. RunPod execution

The run used one secure NVIDIA A40 48 GB pod at USD 0.44/hour with:

```text
PyTorch       2.4.1+cu124
Transformers  4.48.3
PEFT          0.19.1
Datasets      4.8.5
Accelerate    1.14.0
SacreBLEU     2.6.0
```

Training resumed at step 2,770 and stopped at exactly 3,120. The final manifest records `global_step: 3120`, epoch
2.2526, no test file, train runtime 219.16 seconds, and development loss 1.0443. End-to-end primary execution,
including merge and all evaluations, ran from `02:59:56Z` to `03:38:48Z`.

The primary resource trace contains 458 five-second samples:

| Measure | Mean | Maximum |
|---|---:|---:|
| GPU utilization | 32.45% | 94% |
| VRAM used | 6,640 MiB | 22,453 MiB |
| GPU power | 141.81 W | 186.07 W |
| Python RSS | 1,359 MiB | 6,420 MiB |

The transfer evaluation ran from `03:54:09Z` to `04:03:34Z`; maximum VRAM was 3,609 MiB. The pod existed from
`02:43:06Z` to `04:32:18Z`, 6,552 seconds. Wall-clock cost at the advertised rate is approximately USD 0.8008;
RunPod billing remains authoritative. The pod was deleted after pullback and checksum verification, and the saved
post-deletion pod list is empty.

## 5. Primary v22.0 result

### 5.1 Step curve under the same greedy decoder

All values below use beams 1, no n-gram blocking, repetition penalty 1.0, and length penalty 1.0. `Loops` counts
rows with any Unicode orthographic segment repeated at least ten times. The detector is a safety diagnostic; it does
not classify ordinary Kuku Yalanji reduplication as inherently erroneous.

| Set | Step 2,770 chrF | Step 3,120 chrF | Step 4,155 chrF | Exact 2,770 / 3,120 / 4,155 | Loops 2,770 / 3,120 / 4,155 |
|---|---:|---:|---:|---:|---:|
| Synthetic dev 1,609 | 51.8015 | 52.8583 | 52.8471 | 389 / 390 / 396 | 50 / 31 / 39 |
| Synthetic tagged 1,606 | 52.0265 | 53.1613 | **53.8374** | 415 / 417 / 426 | 46 / 37 / 37 |
| Synthetic untagged 1,606 | 52.2497 | 53.3007 | **53.8142** | 413 / 419 / 429 | 45 / 33 / 36 |
| Elder diagnostic 43 | 28.7866 | 28.9510 | **29.3558** | 0 / 0 / 0 | 0 / 0 / 0 |
| Dictionary usage 84 | **47.3542** | 46.1959 | 47.1391 | 4 / 3 / 3 | 0 / 0 / 0 |
| Bible direct 325 | **43.8913** | 43.8572 | 43.7751 | 0 / 0 / 0 | 0 / 1 / 2 |
| Bible reference 325 | **43.9381** | 43.7605 | 43.9093 | 0 / 0 / 0 | 0 / 1 / 1 |

The trajectory does not support the preregistered over-training account. Step 3,120 did not reduce tagged-test loops
below step 4,155, and the final step retained better synthetic-test agreement.

### 5.2 Paired v21.2 step 4,155 versus v22.0 step 3,120

The comparison revalidated model hashes, identical ordered IDs, inputs, references, and greedy decode controls.
It used 50,000 paired nonparametric row bootstraps. The interval estimand is mean per-sentence chrF++ difference,
`step 3,120 minus step 4,155`; corpus chrF++ differences are reported separately because corpus chrF++ is nonlinear.

| Set | Corpus chrF delta | Mean sentence delta | Paired 95% interval |
|---|---:|---:|---:|
| Synthetic dev | +0.011 | -0.317 | [-0.570, -0.069] |
| Synthetic tagged | **-0.676** | **-0.491** | **[-0.738, -0.245]** |
| Synthetic untagged | **-0.513** | **-0.577** | **[-0.838, -0.322]** |
| Elder diagnostic | -0.405 | -0.367 | [-1.097, +0.318] |
| Dictionary usage | **-0.943** | **-0.916** | **[-1.956, -0.055]** |
| Bible direct | +0.082 | +0.068 | [-0.258, +0.389] |
| Bible reference | -0.149 | -0.131 | [-0.487, +0.219] |

These intervals are descriptive across multiple endpoints and are not multiplicity-adjusted. They nevertheless
give no basis for promoting step 3,120 over step 4,155.

### 5.3 Development-only decoder selection

The decoder lock was written at `03:16:28Z`, before any v22 test/control inference.

| Decoder | Dev chrF | Loop rows | Severe shape failures | Maximum segment count | Eligible |
|---|---:|---:|---:|---:|---|
| Greedy | 52.8583 | 31 | 11 | 102 | no |
| No-repeat 4 | 54.1936 | 3 | 0 | 12 | yes |
| Repetition penalty 1.10 | **54.3771** | 1 | 0 | 24 | yes |
| No-repeat 4 + penalty 1.10 | 54.3553 | **0** | **0** | **7** | **selected** |

The selected decoder was within 0.0218 chrF of the best development score and won the preregistered lexicographic
safety ordering.

### 5.4 v22.0 selected-decoder diagnostics

| Set | chrF | BLEU | Exact | Empty | Loop rows |
|---|---:|---:|---:|---:|---:|
| Synthetic dev | 54.3553 | 25.4777 | 387 | 0 | 0 |
| Synthetic tagged | 54.8860 | 25.6674 | 415 | 0 | 2 |
| Synthetic untagged | 54.9264 | 25.6619 | 416 | 0 | 1 |
| Elder diagnostic | 29.2684 | 6.4477 | 0 | 0 | 0 |
| Dictionary usage | 46.9605 | 10.9084 | 1 | 0 | 0 |
| Bible direct | 43.9838 | 12.2656 | 0 | 0 | 0 |
| Bible reference | 43.9427 | 12.0993 | 0 | 0 | 0 |

The decoder controls degeneration, but it cannot rescue the failed training-exposure hypothesis. The primary
checkpoint gate therefore remains **FAIL**.

## 6. Secondary decoder-transfer result

### 6.1 Exact-model enforcement and one failed attempt

The first transfer attempt reconstructed a merged model from the frozen base and v21.2 adapter. The harness hashed
the result before evaluation and rejected it: the reconstructed float32 weights were 4,315,520,560 bytes with
SHA-256 `a38709e98e7103e2f8f7bf1ec8927569f7f3d7d0bec69d997b43d3a0607af440`, not the published v21.2 merged-model
identity `7f9d0fe325e9e4568e45f13179adb336b93bbd53e83ddab2826e999eba3c76f7`. No score from that reconstruction was
generated or used.

The exact published merged model was then uploaded, all 11 transfer inputs passed their remote checksums, and the
evaluation proceeded. This failure is retained because it demonstrates why adapter/base lineage is not a substitute
for exact serialized-model identity when claiming a same-model decoder intervention.

### 6.2 v21.2 greedy versus guarded decoding

| Set | Greedy chrF | Guarded chrF | Corpus delta | Exact greedy / guarded | Loop rows greedy / guarded |
|---|---:|---:|---:|---:|---:|
| Synthetic dev | 52.8471 | **55.1631** | +2.3160 | 396 / 394 | 39 / 1 |
| Synthetic tagged | 53.8374 | **55.7097** | +1.8723 | 426 / 426 | 37 / 2 |
| Synthetic untagged | 53.8142 | **55.7278** | +1.9137 | 429 / 422 | 36 / 2 |
| Elder diagnostic | 29.3558 | 29.3565 | +0.0007 | 0 / 0 | 0 / 0 |
| Dictionary usage | 47.1391 | **47.9648** | +0.8256 | 3 / 1 | 0 / 0 |
| Bible direct | 43.7751 | 43.7197 | -0.0554 | 0 / 0 | 2 / 0 |
| Bible reference | 43.9093 | 43.8353 | -0.0741 | 0 / 0 | 1 / 0 |

The exact-match decreases on development, untagged synthetic, and dictionary usage are reported rather than hidden.
The frozen decision rule did not optimize exact match; it required noninferior corpus chrF++, zero empties, and
strong loop suppression.

### 6.3 Paired decoder audit

The audit verified the exact model and protocol hashes, identical ordered evaluation rows, and the two frozen
decoding signatures. It used 50,000 paired row bootstraps. The estimand is `guarded minus greedy` on identical
weights.

| Set | Mean sentence chrF delta | Paired 95% interval |
|---|---:|---:|
| Synthetic dev | +0.290 | [+0.057, +0.524] |
| Synthetic tagged | +0.325 | [+0.097, +0.560] |
| Synthetic untagged | +0.106 | [-0.128, +0.345] |
| Elder diagnostic | +0.022 | [-0.761, +0.825] |
| Dictionary usage | +0.410 | [-1.129, +1.923] |
| Bible direct | +0.040 | [-0.416, +0.510] |
| Bible reference | -0.077 | [-0.440, +0.290] |

All seven sets had zero empty outputs. Tagged and untagged loop counts were each 2, below the frozen maximum of 5;
all corpus chrF++ floors passed. The transfer gate is therefore **PASS**.

## 7. Linguistic interpretation and limits

Automatic overlap metrics are necessary regression instruments, not linguistic validation. In particular:

- the synthetic sets measure agreement with the governed authored corpus, not independent natural Kuku Yalanji;
- corpus chrF++ rewards character and word n-gram overlap but cannot establish semantic equivalence, argument
  structure, TAM, case, discourse status, idiomaticity, dialect fit, register, or cultural acceptability;
- the 84 dictionary examples and Bible controls measure retention in known adjacent domains, not general transfer;
- the elder-shared set was already observed in earlier work, contains only 43 rows, remains 0/43 exact, and was not
  a blind endpoint;
- no-repeat decoding can suppress legitimate morphology. Kuku Yalanji reduplication is linguistically meaningful,
  so four tokenizer tokens is a conservative engineering guard, not a grammatical prohibition;
- one reconstructed seed cannot distinguish a stable training effect from seed or hardware variance;
- neither PASS result licenses authoritative, safety-critical, legal, medical, cultural, or educational translation.

Before any broader claim, fluent Kuku Yalanji speakers should conduct blind paired review of greedy and guarded
outputs, with explicit annotation of adequacy, grammaticality, morphology, dialect/register, unacceptable semantic
changes, and cases where the n-gram guard damages valid reduplication.

## 8. Post-hoc version probe: dictionary words and elder sentences

After the promotion decisions above were frozen, a separate preregistered diagnostic ran the reconstructed
step-2,770 model, the v22.0 step-3,120 model, and the current step-4,155 v21.2 weights under the same guarded decoder.
It evaluated 297 unique isolated English dictionary glosses and the same 43 elder sentence pairs. All 1,020
model-row outputs were nonempty.

| Model checkpoint | Dictionary accepted exact | Dictionary mean chrF++ | Elder corpus chrF++ | Elder exact |
|---|---:|---:|---:|---:|
| Step 2,770 | 42/297 (14.14%) | 28.06 | 29.18 | 0/43 |
| Step 3,120 | 43/297 (14.48%) | 28.83 | 29.26 | 0/43 |
| Step 4,155 guarded | 48/297 (16.16%) | 30.38 | 29.43 | 0/43 |

The step-4,155 versus step-3,120 paired dictionary difference was +1.68 exact-match percentage points with a
10,000-sample 95% interval of [+0.34, +3.37]. The corresponding mean sentence chrF++ difference was +1.55
[+0.37, +2.95]. Continued exposure therefore improved canonical headword retrieval on this nonblind coverage
probe. The result does not overturn the v22 promotion failure because it tests a different, narrower outcome.

The current model retrieved 46/192 (23.96%) accepted forms seen as full training tokens and only 2/105 (1.90%)
unseen forms. On elder sentences, step 4,155 minus step 3,120 was +0.23 mean sentence chrF++, with interval
[-0.42, +0.99]; all elder checkpoints remained 0/43 exact. Thus the probe supports lookup-first lexical routing
and supplies no evidence of improved natural elder register.

This dictionary was involved in earlier corpus development, and the elder set was observed in earlier evaluation,
so neither component is a blind generalization test. Exact dictionary match measures canonical-form retrieval, not
free-sentence adequacy. The complete public package exposes the
[protocol, combined inputs, row-level predictions, inference logs, paired analysis, and checksums](/model-artifacts/kuku-yalanji-nllb-lora/v21.2-claude-balanced-replay-guarded-20260714/lexicon-elder-benchmark-20260714/).

### 8.1 Independent A40 reproduction

The 340-row probe was subsequently packaged as a sealed one-command RunPod job and rerun twice on an NVIDIA A40
under exact runtime, deterministic float32, no-sampling, and no-TF32 controls. The final GPU run matched all three
frozen local CPU prediction files on 1,020/1,020 rows, and the two A40 executions matched each other on 1,020/1,020
rows. Remote and post-download output checksums both passed. The replicated run therefore confirms the reported
point estimates and intervals without changing the promotion decision.

The [RunPod evaluation and fast feedback loop manual](/docs/kuku-runpod-evaluation-loop.html) records the exact kit,
runtime, resource use, cost, transfer optimization, failed dependency assumptions, rejected first result seal,
corrected rerun, public artifacts, and future post-training workflow.

## 9. Reproducibility and archive

Canonical local archive:

```text
/mnt/donto-data/donto-resources/research/translation-training/
kuku-yalanji-runpod-2026-06-30/runpod/
v22.0-step-matched-replay-20260714T023516Z/
```

The archive contains preregistrations, exact input and code snapshots, manifests, full prediction rows, both model
checkpoints, adapter and merged weights, TensorBoard events, resource traces, stdout/stderr, gate reports, paired
audits, pod metadata, cost/deletion records, and the empty post-deletion pod list.

| Archive identity | SHA-256 |
|---|---|
| Remote 182-file output manifest | `4a561128774f5b554788febac8ce160aea2b2a43ed96b83f8ad8aa9e7a16a5ab` |
| Local 182-file verification log | `52ef91e6880726d12ba2533437054bc31d4b0190d1de925fa78e6470debb3533` |
| v22.0 adapter | `9371066a5e7eaa70f6b24e199dd76118ca871136ae5a6285bc590542c8287fc0` |
| v22.0 merged weights | `8ce31236b051195fbbd01df8d7149ce729249a30959592e9762b2dabf404c2aa` |
| Step-2,770 reconstructed merged weights | `1f2a31593a350f373f2c4345f1bd4f16a29ece18b53fa552ba1828acff7c84f7` |
| Primary promotion gate | `231a456871e7003207cb841867c93de8744bd909edaace11f032fcf32d1592a1` |
| Transfer gate | `66661a0e9bee71f81467ce41bf04c696fce82cf966acb6033b0e8bcf41f5bb23` |
| v21.2 vs v22.0 paired audit | `a110545c0749b87418a7c6a188cb115901bbe5c47f421918b05a80310d736da4` |
| v21.2 decoder-transfer paired audit | `54a302b884963b8b1f30515a8529bb5a9220cc7e06d8e5958cd4ac737623afa7` |

All 182 remote-manifest files verified locally; failures: 0. The comparison scripts do not rerun inference. They
score the sealed prediction archives, validate ordered row identity and decode controls, and preserve representative
row-level differences for audit.

The promoted serving policy has a separate immutable public release directory. Its Atlas archive contains the
exact frozen v12 base, exact v21.2 adapter, tokenizer, guarded decoder manifest, and smoke-test instructions. The
merged archive is a hard link to the historical v21.2 archive, proving that the serving-policy release did not
silently replace the model bytes.

| Public release artifact | Bytes | SHA-256 |
|---|---:|---|
| Guarded Atlas LoRA bundle | 2,530,623,924 | `012335cfb393ca386c7817bd3850dd52f1a751e0e1a58e80e9e5bc9bae9c89d5` |
| Guarded merged bundle | 2,200,740,249 | `920e179d8b4fe45ce75bdd1e9cc2784be51ff8d55d1a190cd45e19cf01a5f490` |
| Hosting manifest | 3,487 | `e40fd2586a3d4eabfdf6052bd65e4f3a92532b1141ea446bc81feb03a7a01c95` |
| Release checksum file | 1,302 | `81e992b0e07ca37d99422ddcf65449fca88bf831a18d4ab1cfec08b1d37c15a1` |
| Local release verification | 434 | `f629b12812405f8f7c04e9dfa978a6a745bba9afae86e779e3c4e38abdda8f6b` |

Full archive traversal found 21 Atlas entries. All 14 targets in the public release checksum file reverified with
zero failures. The public model API exposes the [guarded serving release](/api/v1/models/kuku-yalanji-nllb-lora/versions/v21.2-claude-balanced-replay-guarded-20260714),
the [rejected v22 experiment](/api/v1/models/kuku-yalanji-nllb-lora/versions/v22.0-step-matched-replay-3120-failed),
and the historical greedy weights release as separate records.

## 10. Operational consequence

The model and adapter hashes do not change. The original
`v21.2-claude-balanced-replay-v2-candidate` release remains byte-for-byte available as the historical greedy
baseline. The new serving release, `v21.2-claude-balanced-replay-guarded-20260714`, identifies the unchanged weights
plus the validated decoder policy. Hosts should pin that literal serving release rather than relying on `latest`.
The v22.0 model is registered as a failed-promotion research run with documentation but no public model download.

Exact dictionary entries, community-approved examples, and source recordings remain lookup-first. Generated output
must remain visibly labelled as a research draft. The next scientifically justified model run should add genuinely
independent natural-language supervision and replicated seeds rather than merely selecting another point on this
same single training trajectory.