Kuku Yalanji v22 Step-Matched Replay and v21.2 Decoder-Transfer Report

Experiment date: 2026-07-14 UTC
Primary treatment: continue the sealed v21.2 step-2,770 state to exactly step 3,120
Primary result: FAIL; do not promote the v22.0 checkpoint
Secondary treatment: transfer the development-selected decoder to the exact published v21.2 weights
Secondary result: PASS; retain v21.2 weights and use the guarded decoder
Language direction: English (eng_Latn) -> Kuku Yalanji (gvn_Latn)
Validity: noncommercial research only; not speaker-certified; not authoritative translation

1. Decision

This run did not produce a better trained model. The step-3,120 v22.0 checkpoint failed its preregistered promotion gate because greedy decoding still produced 37 tagged-test rows in which an orthographic word or morpheme segment occurred at least ten times; the maximum was 25. In paired comparison with v21.2 step 4,155, v22.0 was also worse on both synthetic held-outs and the dictionary-usage control. The v22.0 checkpoint is retained as negative evidence and is not a downloadable model release.

The run did identify a better inference policy for the existing v21.2 model. A decoder selected using v22.0 development output only was frozen before transfer:

num_beams=1
no_repeat_ngram_size=4
repetition_penalty=1.10
length_penalty=1.0

Applied once to the exact published v21.2 model, this policy passed all 14 frozen transfer checks. On the tagged synthetic test, corpus chrF++ rose from 53.8374 to 55.7097 and repeated-segment rows fell from 37 to 2. The unchanged v21.2 weights remain the model of record. This is a decoding-policy improvement, not a new model.

2. Research question and preregistration

The primary question was whether v21.2 had been over-exposed to its balanced replay mixture. The falsifiable hypothesis was that stopping at step 3,120 would reduce degeneration while retaining synthetic, dictionary-usage, and Bible performance. Before any v22 output existed, the following were frozen:

• exact base, checkpoint, optimizer, scheduler, RNG, tokenizer, data, and code hashes;
• seed 42 and unchanged v21.2 hyperparameters;
• an exact stop at optimizer update 3,120, with best-checkpoint reload disabled;
• a four-member deterministic decoder grid selected only on the 1,609-row development set;
• test/control thresholds and an all-checks-must-pass promotion rule;
• retention and publication of negative results;
• explicit exclusion of the 43 elder-shared rows from training and model or decoder selection.

The preregistration was frozen at 2026-07-14T02:15:40Z with SHA-256 46c5581e80511679b715d504980297868805e6d35bf46c3cb2fbb4f26846dad4.

The secondary decoder-transfer protocol was frozen at 2026-07-14T03:29:00Z, after the primary checkpoint gate had failed but before any transfer output existed. Its SHA-256 is 23cc078b0d2a3a8cf681684d3af3086841feed0a819b033b46154f2dc8872989.

3. Experimental controls

3.1 Invariants

Key input identities include:

The complete 48-file input manifest is archived as inputs/SHA256SUMS.v22-inputs.

3.2 Reconstruction limitation

The continuation restored optimizer, scheduler, RNG, adapter, and trainer state from step 2,770. It therefore reconstructs the intended trajectory more closely than a fresh continuation from adapter weights alone. Bitwise identity to the original, unobserved step-3,120 trajectory cannot be established because the original v21.2 run did not save that checkpoint. This is one reconstructed seed, not an estimate of training variance.

4. RunPod execution

The run used one secure NVIDIA A40 48 GB pod at USD 0.44/hour with:

PyTorch       2.4.1+cu124
Transformers  4.48.3
PEFT          0.19.1
Datasets      4.8.5
Accelerate    1.14.0
SacreBLEU     2.6.0

Training resumed at step 2,770 and stopped at exactly 3,120. The final manifest records global_step: 3120, epoch 2.2526, no test file, train runtime 219.16 seconds, and development loss 1.0443. End-to-end primary execution, including merge and all evaluations, ran from 02:59:56Z to 03:38:48Z.

The primary resource trace contains 458 five-second samples:

The transfer evaluation ran from 03:54:09Z to 04:03:34Z; maximum VRAM was 3,609 MiB. The pod existed from 02:43:06Z to 04:32:18Z, 6,552 seconds. Wall-clock cost at the advertised rate is approximately USD 0.8008; RunPod billing remains authoritative. The pod was deleted after pullback and checksum verification, and the saved post-deletion pod list is empty.

5. Primary v22.0 result

5.1 Step curve under the same greedy decoder

All values below use beams 1, no n-gram blocking, repetition penalty 1.0, and length penalty 1.0. Loops counts rows with any Unicode orthographic segment repeated at least ten times. The detector is a safety diagnostic; it does not classify ordinary Kuku Yalanji reduplication as inherently erroneous.

The trajectory does not support the preregistered over-training account. Step 3,120 did not reduce tagged-test loops below step 4,155, and the final step retained better synthetic-test agreement.

5.2 Paired v21.2 step 4,155 versus v22.0 step 3,120

The comparison revalidated model hashes, identical ordered IDs, inputs, references, and greedy decode controls. It used 50,000 paired nonparametric row bootstraps. The interval estimand is mean per-sentence chrF++ difference, step 3,120 minus step 4,155; corpus chrF++ differences are reported separately because corpus chrF++ is nonlinear.

These intervals are descriptive across multiple endpoints and are not multiplicity-adjusted. They nevertheless give no basis for promoting step 3,120 over step 4,155.

5.3 Development-only decoder selection

The decoder lock was written at 03:16:28Z, before any v22 test/control inference.

The selected decoder was within 0.0218 chrF of the best development score and won the preregistered lexicographic safety ordering.

5.4 v22.0 selected-decoder diagnostics

The decoder controls degeneration, but it cannot rescue the failed training-exposure hypothesis. The primary checkpoint gate therefore remains FAIL.

6. Secondary decoder-transfer result

6.1 Exact-model enforcement and one failed attempt

The first transfer attempt reconstructed a merged model from the frozen base and v21.2 adapter. The harness hashed the result before evaluation and rejected it: the reconstructed float32 weights were 4,315,520,560 bytes with SHA-256 a38709e98e7103e2f8f7bf1ec8927569f7f3d7d0bec69d997b43d3a0607af440, not the published v21.2 merged-model identity 7f9d0fe325e9e4568e45f13179adb336b93bbd53e83ddab2826e999eba3c76f7. No score from that reconstruction was generated or used.

The exact published merged model was then uploaded, all 11 transfer inputs passed their remote checksums, and the evaluation proceeded. This failure is retained because it demonstrates why adapter/base lineage is not a substitute for exact serialized-model identity when claiming a same-model decoder intervention.

6.2 v21.2 greedy versus guarded decoding

The exact-match decreases on development, untagged synthetic, and dictionary usage are reported rather than hidden. The frozen decision rule did not optimize exact match; it required noninferior corpus chrF++, zero empties, and strong loop suppression.

6.3 Paired decoder audit

The audit verified the exact model and protocol hashes, identical ordered evaluation rows, and the two frozen decoding signatures. It used 50,000 paired row bootstraps. The estimand is guarded minus greedy on identical weights.

All seven sets had zero empty outputs. Tagged and untagged loop counts were each 2, below the frozen maximum of 5; all corpus chrF++ floors passed. The transfer gate is therefore PASS.

7. Linguistic interpretation and limits

Automatic overlap metrics are necessary regression instruments, not linguistic validation. In particular:

• the synthetic sets measure agreement with the governed authored corpus, not independent natural Kuku Yalanji;
• corpus chrF++ rewards character and word n-gram overlap but cannot establish semantic equivalence, argument structure, TAM, case, discourse status, idiomaticity, dialect fit, register, or cultural acceptability;
• the 84 dictionary examples and Bible controls measure retention in known adjacent domains, not general transfer;
• the elder-shared set was already observed in earlier work, contains only 43 rows, remains 0/43 exact, and was not a blind endpoint;
• no-repeat decoding can suppress legitimate morphology. Kuku Yalanji reduplication is linguistically meaningful, so four tokenizer tokens is a conservative engineering guard, not a grammatical prohibition;
• one reconstructed seed cannot distinguish a stable training effect from seed or hardware variance;
• neither PASS result licenses authoritative, safety-critical, legal, medical, cultural, or educational translation.

Before any broader claim, fluent Kuku Yalanji speakers should conduct blind paired review of greedy and guarded outputs, with explicit annotation of adequacy, grammaticality, morphology, dialect/register, unacceptable semantic changes, and cases where the n-gram guard damages valid reduplication.

8. Post-hoc version probe: dictionary words and elder sentences

After the promotion decisions above were frozen, a separate preregistered diagnostic ran the reconstructed step-2,770 model, the v22.0 step-3,120 model, and the current step-4,155 v21.2 weights under the same guarded decoder. It evaluated 297 unique isolated English dictionary glosses and the same 43 elder sentence pairs. All 1,020 model-row outputs were nonempty.

The step-4,155 versus step-3,120 paired dictionary difference was +1.68 exact-match percentage points with a 10,000-sample 95% interval of [+0.34, +3.37]. The corresponding mean sentence chrF++ difference was +1.55 [+0.37, +2.95]. Continued exposure therefore improved canonical headword retrieval on this nonblind coverage probe. The result does not overturn the v22 promotion failure because it tests a different, narrower outcome.

The current model retrieved 46/192 (23.96%) accepted forms seen as full training tokens and only 2/105 (1.90%) unseen forms. On elder sentences, step 4,155 minus step 3,120 was +0.23 mean sentence chrF++, with interval [-0.42, +0.99]; all elder checkpoints remained 0/43 exact. Thus the probe supports lookup-first lexical routing and supplies no evidence of improved natural elder register.

This dictionary was involved in earlier corpus development, and the elder set was observed in earlier evaluation, so neither component is a blind generalization test. Exact dictionary match measures canonical-form retrieval, not free-sentence adequacy. The complete public package exposes the protocol, combined inputs, row-level predictions, inference logs, paired analysis, and checksums.

8.1 Independent A40 reproduction

The 340-row probe was subsequently packaged as a sealed one-command RunPod job and rerun twice on an NVIDIA A40 under exact runtime, deterministic float32, no-sampling, and no-TF32 controls. The final GPU run matched all three frozen local CPU prediction files on 1,020/1,020 rows, and the two A40 executions matched each other on 1,020/1,020 rows. Remote and post-download output checksums both passed. The replicated run therefore confirms the reported point estimates and intervals without changing the promotion decision.

The RunPod evaluation and fast feedback loop manual records the exact kit, runtime, resource use, cost, transfer optimization, failed dependency assumptions, rejected first result seal, corrected rerun, public artifacts, and future post-training workflow.

9. Reproducibility and archive

Canonical local archive:

/mnt/donto-data/donto-resources/research/translation-training/
kuku-yalanji-runpod-2026-06-30/runpod/
v22.0-step-matched-replay-20260714T023516Z/

The archive contains preregistrations, exact input and code snapshots, manifests, full prediction rows, both model checkpoints, adapter and merged weights, TensorBoard events, resource traces, stdout/stderr, gate reports, paired audits, pod metadata, cost/deletion records, and the empty post-deletion pod list.

All 182 remote-manifest files verified locally; failures: 0. The comparison scripts do not rerun inference. They score the sealed prediction archives, validate ordered row identity and decode controls, and preserve representative row-level differences for audit.

The promoted serving policy has a separate immutable public release directory. Its Atlas archive contains the exact frozen v12 base, exact v21.2 adapter, tokenizer, guarded decoder manifest, and smoke-test instructions. The merged archive is a hard link to the historical v21.2 archive, proving that the serving-policy release did not silently replace the model bytes.

Full archive traversal found 21 Atlas entries. All 14 targets in the public release checksum file reverified with zero failures. The public model API exposes the guarded serving release, the rejected v22 experiment, and the historical greedy weights release as separate records.

10. Operational consequence

The model and adapter hashes do not change. The original v21.2-claude-balanced-replay-v2-candidate release remains byte-for-byte available as the historical greedy baseline. The new serving release, v21.2-claude-balanced-replay-guarded-20260714, identifies the unchanged weights plus the validated decoder policy. Hosts should pin that literal serving release rather than relying on latest. The v22.0 model is registered as a failed-promotion research run with documentation but no public model download.

Exact dictionary entries, community-approved examples, and source recordings remain lookup-first. Generated output must remain visibly labelled as a research draft. The next scientifically justified model run should add genuinely independent natural-language supervision and replicated seeds rather than merely selecting another point on this same single training trajectory.