Kuku Yalanji v23 Attested-Narrative Adaptation Report

Experiment date: 2026-07-14 UTC
Experiment ID: v23.0-attested-narrative-adaptation
Language direction: English (eng_Latn) -> Kuku Yalanji (gvn_Latn)
Research promotion: FAIL
CPU deployment: PROHIBITED
Model of record: v21.2 balanced replay with the guarded decoder, downloadable but not CPU-mounted
Validity: noncommercial research only; not speaker-certified, community-approved, or authoritative translation

1. Decision

v23 did not become a model edition. The selected seed improved the speaker-disjoint Text 3 point estimate by 0.6414 corpus chrF++, but the preregistered minimum was +1.0 and the paired mean-sentence interval crossed zero. It also increased tenfold repeated-segment rows on the untagged synthetic control from two to five. The frozen rule required every check to pass, so the research promotion result is FAIL.

The candidate also failed the separate CPU rule by a wide margin. It returned a canonical accepted dictionary headword for only 46 of 297 unique English glosses: 15.49%, with a two-sided 95% Wilson interval of 11.82%-20.04%. CPU mounting requires at least 252/297 exact and a Wilson lower endpoint of at least 80%. The exact candidate weights were therefore never downloaded, registered, published, or loaded by a MobTranslate CPU service. All v23 adapters, merged weights, checkpoints, optimizer states, and trainer caches were deleted after compact evidence was verified.

Positive secondary observations are retained rather than hidden. All three seeds improved the development point score, Text 3 had 14 candidate wins versus seven baseline wins, and all evaluated sets had zero blank outputs. Those facts do not override the failed primary and degeneration gates.

2. Research question and evidential hierarchy

The preregistered question was whether a low-learning-rate second-stage LoRA over v21.2 could add manually reconciled native-speaker narrative evidence while preserving the existing dictionary, everyday-usage, elder, and synthetic behavior.

The hierarchy was fixed before training:

1. Primary held-out natural test: Patz Text 3, Ivy Walker, Yalanji, 56 clauses.
2. Development and replication: Patz Text 36, Bobby Roberts, Nyungkul, 53 clauses.
3. Nonblind cultural diagnostic: 43 previously inspected elder-shared pairs.
4. Behavioral controls: 84 held-out database-usage rows, 297 multi-reference dictionary glosses, and tagged and untagged synthetic tests of 1,606 rows each.
5. Bible catastrophic guard: 325 rows with zero training, seed-selection, or positive objective weight.

Text 36 alone selected checkpoints and one of three seeds. The Text 3 files were not passed to the trainer or seed selector. The immutable selection record says test_data_opened_for_selection: false; a preselection filesystem audit found no Text 3 predictions. The source test file nevertheless existed on the pod, so this is a documented procedural seal, not a claim of cryptographic inaccessibility.

3. Frozen data

3.1 Training mixture

No Bible row and no raw XIGT row entered training. The 795-row XIGT export remained audit-only because its mixture of paradigms, words, phrases, unresolved gloss interleaving, and systematic l/J/1 OCR substitutions had not received clause-level normalization. A mechanical screen is not linguistic attestation.

Nine narrative clauses carrying explicit source/transcription warnings were excluded. Text 51.87 was quarantined because its normalized target duplicated held-out Text 3.52. Nine database rows and six synthetic rows were removed by the frozen exact/token-overlap rules. Database word IDs were disjoint from the 84-row control.

3.2 Speaker and variety split

The split is speaker-disjoint and deliberately tests a Nyungkul-to-Yalanji transfer step. It is not corpus-source independent: all three texts come through the same published grammatical source and editorial/transcription tradition. Three Text 3 clauses carry transcription-uncertainty flags; the other 53 form the unflagged slice.

3.3 Leakage checks

Candidate training rows were compared against development, test, and inherited controls using normalized English exact equality, letters-only Kuku surface equality, English token Jaccard >=0.85 for rows of at least four tokens, Kuku token Jaccard >=0.80 for rows of at least three tokens, and database word-ID disjointness. Every exclusion is recorded additively in the quarantine ledger.

Key frozen identities:

Source materials retain their own rights. The public complete 20,047-pair synthetic process corpus is available as ZIP, tar.gz, JSONL, and SQLite. The rejected v23 narrative treatment is preserved in the internal checksum archive but is not represented as a new public model or as a relicensing of source excerpts.

4. Model and RunPod treatment

The exact v21.2 merged model was adapted independently under seeds 17, 42, and 73:

No v23 decoder search occurred. The fixed decoder was inherited from the separately frozen v22 transfer result.

The pod used one NVIDIA A40 48 GB at USD 0.44/hour with Python 3.11.10, PyTorch 2.4.1+cu124, Transformers 4.48.3, PEFT 0.19.1, Datasets 4.8.5, Accelerate 1.14.0, SacreBLEU 2.6.0, and SentencePiece 0.2.2.

Each seed completed 882 optimizer updates. Training runtimes were 1,635.8, 1,608.3, and 1,605.0 seconds. Maximum observed VRAM was 21,477 MiB; maximum Python RSS was 9,189.8 MiB; maximum GPU utilization was 100%. The resource trace has 1,286 five-second samples covering the computational workload.

The pod existed for at most 7,931 seconds (2.2031 hours), from 14:37:33 to deletion verification at 16:49:44 UTC. Advertised A40 wall-time cost was approximately USD 0.9693. RunPod billing is authoritative. The deletion response was successful and the post-deletion pod list was empty.

5. Training and seed selection

5.1 Internal checkpoint curves

Internal values are trainer-time validation chrF++ under the configured generation path. Final seed selection used a separate fixed guarded evaluation over the same 53 ordered development clauses.

All three trainers restored step 444 before merge. The final update was step 882; because the fixed 148-step cadence ends at step 740, there was no distinct scheduled checkpoint evaluation at 882.

5.2 Fixed development selection

Seed 73 was selected at 2026-07-14T16:14:16.464087+00:00. Its corpus advantage over seed 17 was only 0.0123, so the lock identifies the preregistered candidate; it does not demonstrate a meaningful seed-level difference. Selection-lock SHA-256: 97ab204f53524477d1d1d8b74278f1037d8f7accfe7c92130774c622969656ec.

6. Held-out and control results

The paired interval is a 50,000-replicate percentile interval for the mean per-clause sentence chrF++ difference, candidate minus v21.2. Corpus chrF++ is reported separately because it is a nonlinear corpus-level statistic.

All candidate sets had zero empty outputs. The Text 3 sign test was not significant (p=0.1892), 34 of 56 predictions were byte-identical across models, and exact match remained 1/56 for both. The interval and sign test are descriptive diagnostics, not competing promotion rules.

6.1 Linguistic reading

The treatment changed only 22 Text 3 predictions. Several changes moved output closer to the reference in visible morphological material, including adding final -da in the translation of "We got some food and then we went" and using dunga-nya in clauses involving continued motion. Other changes moved away from attested forms: for "My children were in the back of the Toyota," v21.2 produced kangkal-kangkal, while v23 produced kangkal-karra against reference Kangkal-kangkal. These are useful error-localization observations, not grammatical verdicts.

Character overlap cannot establish correct participant reference, case assignment, tense/aspect/mood, derivational choice, information structure, discourse continuity, idiomaticity, dialect fit, or cultural acceptability. A model can improve chrF++ by copying a suffix while mistranslating the proposition; it can also express a plausible variant that automatic reference overlap penalizes. No speaker reviewed these outputs.

The tagged synthetic mean-sentence interval is entirely below zero even though its corpus decline remains inside the preregistered noninferiority margin. This suggests a small, distributed retention cost. The untagged loop rise is operationally important because it violates the frozen requirement that adaptation must not increase severe repetition under the already guarded decoder.

7. Dictionary and CPU deployment gate

The isolated-word probe treats one unique English gloss as one row and accepts any canonical Kuku Yalanji headword registered for that gloss. Normalization is Unicode-aware and case/spacing/punctuation tolerant; approximate spelling, token containment, semantic plausibility, and chrF++ do not count as exact retrieval.

The Wilson requirement makes "reliable 80%" stricter than a point estimate of 80%: 238/297 is not enough. The v23 exact count is two lower than v21.2 and about 69 percentage points below the first passing observed rate.

The generated CPU gate binds the result to model ID v23.0-seed73 and exact merged-weight SHA-256 8ad3d986d8a3d869d4ab1013a7828e1086db1123dc8d80593cc1fd4ec30c7254. Gate SHA-256: fca0cc8e0b82a8e098e7759e68c0bdc8e24069a3a6b1bb1a87a63d90ac03a8e5.

Both local Kuku inference units have root-owned ExecStartPre interlocks. Startup requires a passing gate, every required check true, an exact model-ID match, and a byte-for-byte weight-hash match. The services remain stopped and disabled, with no listener on ports 7955 or 8765. A future Kuku-to-English service requires a separately frozen reverse benchmark; this English-to-Kuku result cannot be reused in reverse.

8. Preregistered gate outcome

Three checks failed:

1. Text 3 corpus chrF++ delta was +0.6414, below +1.0.
2. Text 3 paired interval lower endpoint was -0.2448, not greater than zero.
3. Untagged synthetic tenfold-loop rows rose from two to five.

All other research checks passed, including three eligible seeds, replication on development, positive unflagged Text 3 point delta, more Text 3 wins than losses, all corpus noninferiority floors, dictionary loss no greater than three rows, zero blanks, tagged loop nonincrease, elder retention, and the deliberately broad Bible catastrophic floor. The Bible score could not compensate for any natural-text failure and was not used that way.

Promotion gate SHA-256: 75cce5bd9067c6c405a1859fbf590fe08dc42764757897ba8dc9943faba3b05e. Paired audit SHA-256: fc98057c5bc7e7008e50482afdcc640f5afca28fcd0414c22993c4e06dbe65f6. Lexicon/elder audit SHA-256: 38deae0e9b5600035919052de8bbf313a94cdb0f2303a473e35cb6343d474604.

9. Retention, integrity, and deletion

The user-directed policy was to retain process knowledge but not weak model iterations. After the compact result was pulled and verified, the retention utility deleted every seed's two checkpoints, trainer cache, adapter, and merged model: 15 directories, 135 files, and 17,360,903,656 bytes. No v23 weight file was downloaded or retained. The retained record contains predictions, metrics, analyses, manifests, reports, logs, checksums, the seed lock, both gates, and the deletion ledger.

Retention ledger SHA-256: cfd8a774198ca67ccfd637d70790e57d5ddd5dce34ef4c1c711e5d1abf48a952.

The original runner's 230-entry output ledger had a sealing-order defect: it hashed launcher.log and v23-driver.log, then appended the final completion line to those files. Every other artifact matched. The flawed ledger is preserved rather than rewritten. After CPU gating and retention, a correctly ordered 102-entry terminal ledger verified on the pod and again after pullback:

POSTRETENTION_SHA256SUMS
b8c3d906bdab8d7d709de4e4993ff16a7203df191cfcd41b80c95c59a1f55d07

The reusable runner was corrected to write its final driver line before hashing and exclude the still-open outer launcher log and PID. Shell syntax validation and all 12 CPU-gate, interlock, and retention tests pass.

The full local run record is on the mounted research drive at:

/mnt/donto-data/donto-resources/research/translation-training/
kuku-yalanji-runpod-2026-06-30/runpod/
v23.0-attested-narrative-20260714T143510Z/

10. Limits of inference

• One held-out text is one cluster. Fifty-six clauses do not represent 56 independent speakers or discourse situations. Row bootstrap resamples clauses and cannot estimate across-text, across-speaker, or community-level uncertainty.
• One speaker per natural split. The design prevents direct speaker overlap but confounds speaker, text, topic, variety label, and split. A speaker effect cannot be separated from a Yalanji/Nyungkul or narrative-content effect.
• Shared source tradition. Train, development, and test derive from one published grammatical source. Editorial conventions and elicitation/transcription practices may transfer even when exact clauses do not.
• Observed dictionary benchmark. The dictionary informed prior corpus and model work. Exact retrieval is a useful operational deployment requirement, not a blind estimate of general lexical competence.
• Automatic references are incomplete. A single natural reference cannot enumerate legitimate word order, morphology, lexical choice, discourse packaging, or dialect variants.
• No community validation. Neither chrF++, exact match, a Wilson interval, nor a model gate establishes semantic safety or cultural authority.
• Model-family scope. This is one low-rate LoRA recipe over one 1.3B NLLB lineage. Failure does not show that attested narrative adaptation is ineffective in general.
• Direction scope. Every reported gate is English-to-Kuku Yalanji. Nothing here measures Kuku Yalanji-to-English.

11. Next experiment

The next run should not repeat this treatment unchanged. Its design should be frozen only after adding more manually reconciled, clause-level attested material from multiple texts and speakers, with independent orthographic review and explicit source/variety metadata. Development and final test should each contain multiple text/speaker clusters so uncertainty can be estimated at the cluster level rather than by pretending clauses are independent.

Lexical deployment needs its own objective and evaluation lane. A candidate CPU service must be trained or composed to retrieve canonical headwords robustly, then audited end to end on the same 297-row gate. A dictionary-backed or hybrid served system may be evaluated, but the whole served path must pass; an external lookup cannot be used to claim that weak model weights themselves achieved 80%.

For sentence translation, preserve the Text 3-style natural endpoint, expand it to multiple clusters, keep synthetic sets only as retention controls, and continue giving Bible data no positive objective weight. Manual linguistic review should score semantic adequacy, participant tracking, case, TAM, derivation, negation, discourse continuity, and variety/register fit independently of character overlap.

Until such a system passes both research and CPU gates, MobTranslate should continue using the dictionary-guided prompt route for live drafts, clearly labelled for community verification, with all custom Kuku model CPU services unloaded.

12. Related public material

• Frozen v23 preregistration
• RunPod evaluation and fast-feedback handbook
• v22 experiment and guarded-decoder transfer
• v21.2 model, data, and hosting guide
• Versioned model API
• Complete synthetic process corpus
• Raw Markdown for this report