# MobTranslate Kuku Yalanji Model Program: Technical Audit, Attempt History, and v24 Training Plan **Prepared for:** independent technical, linguistic, statistical, and operational review **Prepared:** 2026-07-15 UTC **Revision:** 2, after independent-review disposition; revision 1 SHA-256 was `565cb8d70aecec2095ba4da9b9324595f61c0b447b759b7a84dee43a8ebb18b1` **Public canonical URL:** **Rendered edition:** **Review disposition:** **Language direction covered:** English (`eng_Latn`) to Kuku Yalanji (`gvn_Latn`) **Current model of record:** `v21.2-claude-balanced-replay-guarded-20260714` **Current deployment decision:** downloadable research artifact; custom Kuku Yalanji CPU inference is stopped and disabled **Next proposed experiment:** v24 lexical reconstruction, lexicon uptake, and sentence-retention study with fixed-token treatment comparisons **Certification:** none. No model described here is speaker-certified, community-approved, or suitable for authoritative translation. > Revision note: the independent review was treated as advice, not a specification. The project adopted corrections > that improve scientific validity or remove unsupported public claims, modified recommendations that were too broad, > and deferred new architectures and review infrastructure that are not needed for the narrow screen. Automatic > scores remain engineering measurements, not evidence of community authority. ## 1. Executive assessment MobTranslate has built a substantial and unusually well-documented low-resource translation research program, but it does not yet have a reliable free-form Kuku Yalanji translator. The strongest retained custom model is an NLLB-derived 1.3B sequence-to-sequence model, not a chat LLM. Its v21.2 weights were trained on a leakage-audited 20,047-pair synthetic corpus treatment plus controlled replay. Under the current guarded decoder it performs reasonably on held-out material produced by the same synthetic authoring process, and replay preserved dictionary-usage and Bible domains. It still fails the decisive lexical and natural language tests: | Current evidence | Result | Consequence | |---|---:|---| | Isolated dictionary glosses | 48/297 accepted exact, 16.16% | Far below the model lexical-reconstruction gate | | Wilson 95% interval for that exact rate | 12.41%-20.78% | Fails the conservative frozen-benchmark rule | | Frozen elder-shared sentences | 0/43 exact; corpus chrF++ about 29.43 | Natural-register transfer is weak | | Synthetic tagged test | chrF++ 55.71 under guarded decoding | Useful same-process regression signal only | | Empty outputs | 0 on the frozen probes | Necessary, but not sufficient | | Repetition failures | Reduced from 37 to 2 tagged rows by guarded decoding | Decoder repair helped; model competence did not become general | At 297 benchmark rows, the deployment policy requires at least 252 accepted exact outputs. That is an observed rate of 84.85%, because the two-sided Wilson 95% lower endpoint must itself be at least 80%. A bare point estimate of 80% would not pass. The current result is 204 exact answers short of the first passing count. The central training-design error is now clear. The project has 2,688 dictionary entries and 1,870 working lexemes, but v21.2 did not train every eligible dictionary mapping as an isolated English-to-Kuku Yalanji sequence pair. It replayed only 357 dedicated dictionary-usage examples four times. Most dictionary vocabulary appeared only incidentally inside sentences. The frozen benchmark then showed exactly the expected exposure dependence: ```text Target headword appeared as a complete training token: 46/192 exact (23.96%) Target headword not observed as a complete training token: 2/105 exact (1.90%) ``` The next model should therefore include explicit lexical pairs under a task shape distinct from sentence translation. In a training JSONL record, the source and target are separate fields: ```json { "id": "lexeme:dictionary-123:canonical", "direction": "eng-gvn", "input_text": " woman", "unconditioned_input_text": "woman", "output_text": "jalbu", "pair_kind": "dictionary_lexeme", "source_lang": "eng_Latn", "target_lang": "gvn_Latn" } ``` The conceptual pair is `woman -> jalbu`. The literal text `woman : jalbu` should not be put into the source field. Training and serving shapes must match, and `` remains reserved for sentence translation. This lexical objective must not be confused with sentence translation. Memorizing dictionary entries can improve closed-set reconstruction and may help sentence vocabulary, but deterministic lookup remains safer for known words. Lexical reconstruction does not by itself teach case marking, agreement, derivation, tense/aspect/mood, argument structure, discourse continuity, word order, dialect choice, or idiomatic usage. v24 therefore reports separate tasks: 1. **direct lexical reconstruction** after an explicit `` pair; 2. **contextual lexical acquisition** where a form occurs in sentences but not as an isolated pair; 3. **lexicon uptake** where a held-out mapping is supplied at inference under ``; 4. **morphological generalization** only where a governed lemma/form analysis exists; and 5. **sentence retention** under the unchanged decoder. A fully hidden arbitrary mapping is not a productive-generalization test: `jalbu` cannot be derived from the English word `woman`. Exact success there mostly reflects prior exposure, leakage, cognacy, or chance. The all-lexicon fit is therefore scored as closed-set reconstruction and never as unseen-language generalization. For known dictionary queries, deterministic database lookup remains the technically correct first route and can be near-exact without asking a neural model to regenerate stored facts. A model may be evaluated on sentence drafts, but it cannot enter that route unless a separate natural-text, degeneration, and speaker-review gate passes. ## 2. Review questions and disposition The original dossier asked an external reviewer the following questions. The resulting adopt/modify/defer/reject decisions are preserved in the [independent-review disposition](/docs/kuku-v24-review-disposition.html): 1. Is the evidential separation among synthetic, dictionary, Bible, usage-example, and native-speaker narrative material linguistically defensible? 2. Are the normalization, grouping, leakage, and multi-reference rules for isolated lexical evaluation valid? 3. Can the v24 tasks distinguish closed-set reconstruction, contextual acquisition, glossary uptake, morphology, and sentence retention without claiming an arbitrary hidden mapping is inferable? 4. Is NLLB-200 distilled 1.3B plus LoRA still a defensible architecture, or should a controlled alternative be added without discarding the established baseline? 5. Are the proposed task prefixes, replay ratios, fixed-step controls, and three-seed design sufficient to estimate the effect of lexical supervision? 6. Are chrF++, BLEU, exact match, CER, length, and repetition diagnostics being interpreted within their limits? 7. What speaker-review instrument should be used for semantic adequacy, grammaticality, morphology, variety, register, discourse, and cultural acceptability? 8. Are data rights and community governance strong enough for the proposed research use and public artifacts? 9. Which separate gates should govern deterministic lookup, model lexical reconstruction, and model sentence generation? 10. Which assumptions below are weak enough that v24 should not start until they are resolved? ## 3. Claims, non-claims, and terminology ### 3.1 Claims supported by the record - The governed synthetic corpus contains exactly 20,047 English-Kuku Yalanji pairs, not 210,000 pairs. - The corpus closure audit passed 19 checks with no failed checks. - v21.2 is the best retained custom research recipe under the existing comparison battery. - v21.2 retained adjacent replay domains much better than the synthetic-only v21.1 treatment. - v21.2 does not meet the project's dictionary reliability requirement. - v21.2 and v23 do not establish natural elder-register translation competence. - Guarded decoding materially reduced severe repetition on unchanged v21.2 weights. - The v23 attested-narrative adaptation failed its preregistered research gate and lexical-model gate. - v23 candidate weights and checkpoints were deleted after compact evidence was verified. - Current live homepage translation uses a dictionary-guided hosted LLM route, while custom Kuku CPU services are stopped, disabled, and blocked by a model-bound gate. ### 3.2 Claims not supported - The 20,047 synthetic pairs are not 20,047 independently speaker-attested utterances. - Process review is not elder review or community certification. - A high synthetic chrF++ score is not a population estimate for natural Kuku Yalanji. - Exact dictionary recall is not sentence-level grammatical competence. - `Claude` in the v21.2 name does not mean that Claude is the deployed translation network. It identifies the operator/preparation lane. The network is NLLB-derived. - The custom `gvn_Latn` token is a project tokenizer extension, not evidence that upstream NLLB natively supports a complete Kuku Yalanji model. - No reported result applies automatically to Kuku Yalanji-to-English translation. - Public downloadability is not a commercial license or speaker endorsement. ### 3.3 Language and variety scope The project uses ISO 639-3 `gvn` and the custom NLLB token `gvn_Latn` as engineering identifiers. Those identifiers are coarser than the linguistic variation represented in the sources. Patz narrative material includes Yalanji and Nyungkul labels, with different speakers and observable form differences. The 43 elder-shared sentences, dictionary, Bible, synthetic corpus, and Patz clauses should not be pooled under an assumption of homogeneous speaker, community, topic, register, orthography, or variety. Every future row should carry, where known: - source work and exact location; - speaker and recorder/editor; - variety or dialect label as given by the source; - orthographic system and normalization history; - genre, register, and discourse context; - rights and approval state; - whether the text is attested, reconstructed, synthetic, translated, or mechanically extracted; - review and uncertainty flags; - supersession history rather than destructive replacement. ### 3.4 Governance and release boundary The dossier does not establish a formal Kuku Yalanji authorizing body, negotiated governance agreement, or scope of community approval. That absence is not proof that no relationships exist; it means the public evidence is insufficient for platform-wide claims of community ownership, certification, or authority. The project will not invent an approval record. Unresolved governance blocks public model release and transfer of speaker, restricted, or uncertain material to third-party providers. It does not block private builder, evaluator, and method work over material whose research use is already documented. Uncertain records are excluded or quarantined. Before public release, permissions for training, redistribution, derived weights, hosted-provider transfer, publication, withdrawal, and culturally sensitive material must be recorded at the applicable source and scope level. ## 4. Current operational state, 2026-07-15 UTC | Component | Current state | |---|---| | Public site | live behind Caddy and Next.js | | Homepage translator | Complete selected dictionary injected into a `gpt-5.4-mini` prompt; required Zod-validated tool call | | Kuku custom CPU service | `kuku-v2-infer.service`: inactive and disabled | | Previous Kuku CPU service | `mobtranslate-translate-v2.service`: inactive and disabled | | Local model ports | 7955 and 8765 closed | | Web service port | 3300 listening behind Caddy | | Current model lexical gate | FAIL, bound to v21.2 model ID and exact merged-weight hash | | Current model sentence gate | FAIL, independently model-bound; lexical PASS cannot satisfy it | | Current RunPod workload | `runpodctl pod list` returns no active pods | | Model distribution | Public, immutable version API and downloadable research bundles | | Root disk | 193 GB total, 11 GB available, 95% used at report time | | Mounted research disk | 984 GB total, about 130 GB available, 87% used at report time | The current root-disk margin is too small for casual model duplication. New research artifacts must remain on `/mnt/donto-data`; large candidate weights should remain on RunPod until their compact evaluation result passes. The deprecated RunPod list command still exposes an old `EXITED` v20 record, but the current pod-list endpoint has no active pod. The live model API is: ```text https://mobtranslate.com/api/v1/models https://mobtranslate.com/api/v1/models/kuku-yalanji-nllb-lora https://mobtranslate.com/api/v1/models/kuku-yalanji-nllb-lora/versions/latest https://mobtranslate.com/api/v1/models/openapi.json ``` The latest alias currently resolves to: ```text v21.2-claude-balanced-replay-guarded-20260714 ``` Hosts should pin the literal version and SHA-256, not the mutable `latest` alias. ## 5. Data inventory ### 5.1 Governed synthetic corpus The completed process corpus contains: | Record family | Count | |---|---:| | English-Kuku Yalanji sentence pairs | 20,047 | | Kuku Yalanji word tokens | 135,547 | | Accepted sentence rows | 19,136 | | Revised sentence rows | 911 | | Dictionary entries | 2,688 | | Working lexemes | 1,870 | | Reviews | 101,235 | | Revisions | 1,555 | | Lessons | 8,035 | | Process events | 4,745 | The canonical split is: ```text train 16,820 development 1,609 test 1,618 total 20,047 ``` The v21 leakage treatment preserved rather than deleted 190 collisions: ```text effective synthetic train 16,642 validation 1,609 effective synthetic test 1,606 quarantine 190 ``` After quarantine, train, validation, and test had zero intersections under the recorded normalized source, target-surface, and exact-pair keys. The corpus content identity recorded by the closure audit is: ```text cb32b539e0bf9763a707afe22d702c9520f5b41ec244ea8e63013fee2798763a ``` The synthetic corpus was authored against the source-preserving dictionary, Elisabeth Patz's grammar, explicit morphological and frame records, batch review, lint, phantom-lexeme controls, revision ledgers, lessons, and closure audits. That is a strong process record. It does not remove model-assisted authoring bias, shared-template effects, possible analysis errors, or dependence between training and synthetic evaluation. ### 5.2 Dictionary The working dictionary has 2,688 entries and 1,870 working lexemes. It includes canonical headwords, parts of speech, phonemic forms where available, English glosses and definitions, semantic domains, commentary, loanword notes, and source information. Not every dictionary entry is a valid one-to-one training pair: - some English glosses map to multiple canonical Kuku Yalanji headwords; - one headword may have multiple senses; - names, abbreviations, particles, bound forms, and multiword explanations need separate treatment; - variant, dialect, register, and uncertainty information must not be collapsed; - an English definition can be descriptive rather than a substitutable translation; - isolated lemma identity does not encode inflected or derived forms. The existing frozen lexical probe deliberately used a narrow, reproducible subset. It began with 1,640 JSON-LD graph entries, retained 395 entries where both headword and English definition were a single lowercase Unicode word optionally containing an internal hyphen/apostrophe, and grouped them into 297 unique English prompts. Every canonical headword sharing a prompt was accepted. This is the origin of the current 297-row model lexical-reconstruction gate. The v24 builder should begin from all 2,688 source entries, emit an eligibility and exclusion ledger, and construct more than one lexical task class. It must not silently claim that all entries are simple bilingual equivalents. ### 5.3 Database usage examples The original database export contains 449 English-Kuku Yalanji usage examples. The v21.2 effective training mixture used 357 non-colliding training rows, replayed four times for 1,428 row presentations per epoch. An 84-row held-out set was grouped by dictionary word ID. These examples are high-value because they are short, relatively literal, and closer to everyday dictionary use than Bible verses, but they are not a substitute for broad connected speech. ### 5.4 Elder-shared material There are 43 rights-cleared connected-narrative sentence pairs used as a frozen diagnostic. They have been observed through many iterations, so they are no longer a blind final test. They remain valuable as a stable regression set. v18 was trained directly on them and reached 43/43 exact, proving memorization; later models that excluded them returned 0/43 exact. Neither outcome estimates performance on unseen elder speech. ### 5.5 Patz narrative clauses Clause-level extraction from Patz supplied the v23 natural-text experiment: | Role | Speaker | Source variety label | Clauses | |---|---|---|---:| | Train | Charlie Tayley | Nyungkul | 156 unique | | Development | Bobby Roberts | Nyungkul | 53 | | Test | Ivy Walker | Yalanji | 56, of which 53 were unflagged | The split was speaker-disjoint, but speaker, story, topic, variety, and split were confounded. All texts share one published grammar's transcription/editorial tradition. One text is one cluster; 56 clauses are not 56 independent speakers. The v23 row bootstrap is therefore descriptive over clauses, not a community-level uncertainty estimate. ### 5.6 Bible corpus The eBible asset contains 8,297 unique Kuku Yalanji verses aligned to two English editions, producing 16,597 English-to-Kuku rows. It was critical for early pipeline development and target-language form learning, but it is a free, interpretive, domain-specific translation. It can restructure clauses, expand content, and introduce biblical register. Early mixtures were approximately 94% Bible by row count, and small literal/narrative sets were washed out. The v24 positive objective should give Bible rows zero weight. A frozen Bible held-out may remain only as a broad catastrophic-forgetting diagnostic. It must not select the model, compensate for failed natural-text endpoints, or be described as everyday-language competence. ### 5.7 XIGT, historical material, and audio - About 795 automatically extracted interlinear examples exist in XIGT form. They mix paradigms, words, phrases, clauses, interleaved glosses, OCR substitutions, and uncertain segmentation. They are audit material until clause-level linguistic reconciliation is complete. - Historical mission papers and dictionaries are valuable diachronic sources but need orthographic, temporal, genre, rights, and provenance labels. They must not be normalized into contemporary training without review. - Synthetic TTS and human recordings are separate speech assets. They do not become text-translation supervision merely because audio exists. ### 5.8 Public data artifacts | Artifact | SHA-256 | URL | |---|---|---| | Complete process corpus ZIP | `5db11f1f10f60b1c6df0ea00f0bd4b2f8a313dfc487ae11145c495c96d880e73` | [download](/datasets/kuku-yalanji-synthetic-research-corpus-v2.0.0-20260711.zip) | | Complete process corpus tar.gz | `f3d671921944a065e83c11d028471fd7bf0127e6665cae4ded699cd1b23e56ba` | [download](/datasets/kuku-yalanji-synthetic-research-corpus-v2.0.0-20260711.tar.gz) | | All sentence rows JSONL | `b06ec5f5bbc1f4e39cf20a30fefe04d496c42f1408c73e75616ebbd531a714f0` | [download](/datasets/kuku-yalanji-synthetic-research-corpus-v2.0.0-20260711/data/sentences.jsonl) | | Dictionary JSONL | `d1929dab4cb5cd9b8399f2a63e2567a836a6cb40463ad70afefa4cc6f698e074` | [download](/datasets/kuku-yalanji-synthetic-research-corpus-v2.0.0-20260711/data/dictionary.jsonl) | | Process SQLite database | `b9db281cccfa50ced087747d812b5712ceac22dc7b0cd1f3227d7c2e36988703` | [download](/datasets/kuku-yalanji-synthetic-research-corpus-v2.0.0-20260711/data/synthetic.db) | | Exact v21.2 treatment | `91fc18f539b863f9a0e87ed3c372b5f040f359e9bcd427dc506875e1b4f6a467` | [download](/datasets/kuku-yalanji-v21.2-training-20260711.tar.gz) | ## 6. Training architecture and harness ### 6.1 Current model lineage ```text Upstream family facebook/nllb-200-distilled-1.3B Transformers class M2M100ForConditionalGeneration Project base frozen v12 Bible continuation Source token eng_Latn Target token custom gvn_Latn, ID 256204 Task conditional sequence-to-sequence translation Adapter PEFT LoRA, SEQ_2_SEQ_LM Serving input prefix ``` The v21.2 adapter cannot safely be attached to untouched stock NLLB. The frozen project base includes the added target token and project-specific weights. Dynamic hosts must load the distributed exact base and adapter together, or use the merged model. ### 6.2 v21.2 LoRA and optimization ```text LoRA rank / alpha / dropout 64 / 128 / 0.0 Target modules q_proj,k_proj,v_proj,out_proj,fc1,fc2 Epochs 3 Learning rate 2e-5 Batch / accumulation 8 / 2 Effective batch 16 Warmup / weight decay 0 / 0 Source / target cap 192 / 208 Seed 42 Optimizer steps 4,155 Checkpoint selection validation chrF++ ``` The exact merged weight hash is: ```text 7f9d0fe325e9e4568e45f13179adb336b93bbd53e83ddab2826e999eba3c76f7 ``` ### 6.3 v21.2 effective mixture | Component per epoch | Rows | |---|---:| | Leakage-audited synthetic | 16,642 | | Bible direct replay | 2,047 | | Bible reference-conditioned replay | 2,047 | | Dictionary-usage replay | 1,428, from 357 rows x4 | | Elder rows | 0 | | **Total per epoch** | **22,164** | Across three epochs this is 66,492 row presentations. It is not 66,492 unique sentence pairs. The intended canonical mix was 22,198 rows per epoch; one synoptic Bible collision and eight database base rows were quarantined, excluding 34 expanded rows while preserving their records. ### 6.4 Executed trainer behavior The trainer reads structured JSONL, filters the requested direction, tokenizes `input_text` and `output_text` separately, applies LoRA, evaluates with SacreBLEU, saves adapter and merged forms, and writes a model manifest with data hashes, environment, training arguments, metrics, and artifact paths. It also contains an NLLB merge canonicalization fix so serialized shared/encoder/decoder embeddings cannot silently diverge after PEFT merge. Core executed preprocessing is equivalent to: ```python def preprocess(batch): tokenizer.src_lang = "eng_Latn" tokenizer.tgt_lang = "gvn_Latn" model_inputs = tokenizer( [" ".join(text.split()) for text in batch["input_text"]], max_length=192, truncation=True, ) labels = tokenizer( text_target=[" ".join(text.split()) for text in batch["output_text"]], max_length=208, truncation=True, ) model_inputs["labels"] = labels["input_ids"] return model_inputs ``` This is why the colon notation matters. `input_text` and `output_text` are distinct model sequences; a colon is not an abstract separator unless it is literally present in `input_text`. ### 6.5 Current guarded decoder ```python output = model.generate( **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("gvn_Latn"), num_beams=1, max_new_tokens=208, no_repeat_ngram_size=4, repetition_penalty=1.10, length_penalty=1.0, ) ``` The no-repeat guard is an engineering safety intervention. Kuku Yalanji has meaningful reduplication, so fluent speaker review must check whether the tokenizer-level four-gram ban suppresses valid morphology. ## 7. Complete material experiment history Scores across different rows, preprocessing, decoders, and objectives are not directly rankable. The table records what each run established within its own contract. ### 7.1 Initial engineering line | Version | Treatment | Material result | Decision | |---|---|---|---| | `0.1.0-smoke` | 256-row/one-epoch pipeline smoke | chrF about 5.45 | Pipeline proof only | | `0.1.0-mini-pilot` | 2,048 rows, one epoch | English copying and loops | Plumbing artifact | | `0.2.0` gvn/tpi bakeoff | Capped shuffled Bible candidates | Anti-repeat decode helped, still unfaithful | No promotion claim | | `0.3.0-full-gvn-token` | 600M full corpus, four epochs | Test chrF++ 29.95, semantic drift | Engineering baseline | | `0.4.0-full-gvn-token` | 600M, eight epochs | Standalone chrF++ 31.07 | Former internal proof | | `0.5.0-full-tpi-proxy` | 600M, eight epochs | chrF++ 31.20, effectively tied | Negative/control | | `0.6.0-full-gvn-token-1.3b` | 1.3B custom token | Step-800 chrF++ 10.54 | Stopped early | | `0.7.0-full-tpi-proxy-1.3b` | 1.3B tpi proxy, eight epochs | Standalone chrF++ 34.49 | Former internal proof, still unfaithful | The early line proved the RunPod, LoRA, merge, evaluation, download, and CPU-serving pipeline. It also showed that larger base size and higher overlap scores did not cure clause compression or semantic drift. ### 7.2 Diagnostic and curriculum line, v8-v16 | Version | Purpose | Key result | Interpretation | |---|---|---|---| | `v8` | 1/8/32/256-row overfit gates | Could memorize | Tokenization and optimization path worked | | `v9.0` | 3,094 Bible rows, 3 epochs | Held-out chrF 25.48 | Underfit | | `v9.1` | Longer schedule | Held-out chrF 37.10 | Improved, not faithful | | `v9.2` | 512-row short curriculum | Train chrF 96.15, held-out 34.28 | Reproduction gate passed | | `v9.3` | 1,024-row curriculum | Held-out chrF 39.75 | Scale helped modestly | | `v9.4` | 2,048-row curriculum | Held-out chrF 42.84 | Better overlap, not exact | | `v9.5` | 2,571-row full short curriculum | Held-out chrF 42.64 | No gain over v9.4 | | `v9.6` | Reference-conditioned Bible | Held-out chrF 43.33 | Conditioning helped slightly | | `v9.7` | Tagged direct plus reference Bible | Direct/ref 44.13/44.34 | Strong Bible branch, exact 0 | | `v9.8` | Bible plus raw DB usage | Usage 39.47 | Usage task needed better conditioning | | `v9.9` | Exact resource retrieval diagnostic | 734/734 lookup exact | Known resources should be lookup-first | | `v9.9A` | Usage-only adapter | 2/84 exact | Narrow memorization did not generalize | | `v9.9B` | Glossary usage gates | Up to 60/84 headword hits, 1/84 exact | Headword presence is not sentence fidelity | | `v9.9C` | Full glossary-conditioned usage | 5/84 exact, chrF 43.58 | Better but insufficient | | `v10` | Tagged Bible plus glossary usage | Usage chrF 56.18, exact 3/84 | Best early usage branch | | `v11` | ByT5-small control | Held-out chrF about 5.9 | Negative control, not a sufficient architecture test | | `v12` | Custom `gvn_Latn` target token | Bible direct/ref 44.37/44.36 | Frozen base for later work | | `v13` | Retrieval prefix inside NLLB input | Held-out Bible about 29.7 | This prefix design hurt | | `v14` | Hard lexical constraints at decode | Oracle terms raised overlap but over-generated | `force_words_ids` too brittle | | `v15` | Soft lexical hints | Direct/ref 44.15/43.72 | Did not beat v12 | | `v16/16.1` | Lookup router and candidate ceiling | Resource lookup 734/734 exact | Product routing beats regeneration for known data | | `v16.2` | Candidate selector | Insufficient reranking gain | Do not add complexity without evidence | ### 7.3 Narrative and full-corpus line, v17-v20 | Version | Treatment | Result | Decision | |---|---|---|---| | `v17` | Planned domain-separated usage continuation | Not executed under this ID | Superseded by v18 | | `v18` | v10 plus DB usage and 43 elder rows | Elder 43/43 exact; usage chrF 54.52; Bible regressed | Demonstrated direct memorization and forgetting | | `v19` | Balanced replay from v12 | Elder 29/43 exact; usage 53.03; Bible retained | Better broad compromise, not universal | | `v20` | 35,394-row heterogeneous candidate corpus | Test chrF 39.39; elder 0/43 | Full-scale mixture failed | v20 falsified the hypothesis that simply adding all available material would produce a faithful translator. The small high-value sets were overwhelmed by heterogeneous, Bible-heavy supervision. Row count was not the right proxy for linguistic value. ### 7.4 v21.1 versus v21.2 v21.1 trained only the 16,642 leakage-audited synthetic rows. v21.2 used the same synthetic treatment plus Bible and dictionary-usage replay. Both started from the same v12 base and used the same seed and LoRA configuration, but v21.2 had 4,155 optimizer updates versus 3,120 for v21.1 and ran on different hardware. The result identifies the better practical recipe, not a perfectly isolated causal replay effect. | Shared evaluation | v21.1 chrF / exact | v21.2 greedy chrF / exact | Corpus delta | |---|---:|---:|---:| | Synthetic development, 1,609 | 53.86 / 391 | 52.85 / 396 | -1.01 | | Synthetic tagged, 1,606 | 54.70 / 416 | 53.84 / 426 | -0.87 | | Synthetic untagged, 1,606 | 54.40 / 417 | 53.81 / 429 | -0.58 | | Elder, 43 | 28.80 / 0 | 29.36 / 0 | +0.56 | | Dictionary usage, 84 | 34.40 / 0 | 47.14 / 3 | +12.74 | | Bible direct, 325 | 28.55 / 0 | 43.78 / 0 | +15.23 | | Bible reference, 325 | 28.76 / 0 | 43.91 / 0 | +15.15 | On tagged synthetic rows, the paired mean sentence chrF++ difference for v21.2 minus v21.1 was +0.082 with 95% interval [-0.225, +0.396]. The correct conclusion is synthetic parity with large replay-domain retention gains. Neither model solved elder/natural-register transfer. ### 7.5 v22 step-matched replay and decoder transfer v22 reconstructed the v21.2 trajectory at step 2,770 and continued to exactly step 3,120 to test whether v21.2 had been over-trained. The checkpoint failed. Under greedy decoding it retained 37 tagged repeated-segment rows and was worse than step 4,155 on synthetic tests and dictionary usage. The development-only decoder search did produce a useful secondary result. The chosen guarded policy was frozen before transfer to exact v21.2 weights. On unchanged v21.2 weights: | Set | Greedy chrF | Guarded chrF | Exact greedy/guarded | Loop rows greedy/guarded | |---|---:|---:|---:|---:| | Synthetic dev | 52.8471 | 55.1631 | 396/394 | 39/1 | | Synthetic tagged | 53.8374 | 55.7097 | 426/426 | 37/2 | | Synthetic untagged | 53.8142 | 55.7278 | 429/422 | 36/2 | | Elder | 29.3558 | 29.3565 | 0/0 | 0/0 | | Dictionary usage | 47.1391 | 47.9648 | 3/1 | 0/0 | | Bible direct | 43.7751 | 43.7197 | 0/0 | 2/0 | | Bible reference | 43.9093 | 43.8353 | 0/0 | 1/0 | The current edition is therefore a serving-policy release over unchanged v21.2 weights, not a v22 model. ### 7.6 v23 attested-narrative adaptation v23 was a preregistered low-rate second-stage LoRA over exact v21.2 weights. It used three seeds and no Bible rows: ```text 156 reconciled Text 51/12 clauses x4 624 DB usage direct 356 DB usage glossary 356 Synthetic retention 1,024 Total per epoch 2,360 Epochs 6 Seeds 17,42,73 ``` Seed 73 was selected on Text 36 development only. The held-out Text 3 result was: ```text v21.2 corpus chrF++ 31.6512 v23 seed 73 corpus chrF++ 32.2926 corpus delta +0.6414 mean sentence delta +0.4494 paired 95% interval [-0.2448,+1.1397] ``` The preregistered natural-test minimum was +1.0 corpus chrF++, the interval lower endpoint had to be positive, and untagged severe loops could not increase. All three failed: the point delta was +0.6414, the interval crossed zero, and untagged loop rows rose from 2 to 5. The separate model lexical-reconstruction result was 46/297 exact, 15.49%, Wilson 95% 11.82%-20.04%. The research and lexical gates both failed. Candidate merged-weight identity before deletion was: ```text 8ad3d986d8a3d869d4ab1013a7828e1086db1123dc8d80593cc1fd4ec30c7254 ``` All three seeds' adapters, merged weights, checkpoints, optimizer states, and trainer caches were deleted after the compact evidence was verified: 17,360,903,656 bytes. The retained 181-file archive manifest hashes to: ```text 63261bf0139a075ed5292525bda6f704b35680b902526895bbd8b2da9112912d ``` The observed A40 pod cost was approximately USD 0.9693 at the then-advertised USD 0.44/hour rate. The pod was deleted and the post-deletion active list was empty. ## 8. Evaluation framework ### 8.1 Frozen automatic endpoints The current battery separates: - synthetic development and tagged/untagged synthetic test; - isolated dictionary glosses with multiple accepted references; - dictionary usage examples; - elder-shared connected sentences; - speaker-disjoint Patz narrative development/test where applicable; - Bible catastrophic-retention controls; - empty, source-copying, length, truncation, and repeated-segment failures. ### 8.2 Exact lexical scoring One row is one normalized English gloss. Every canonical Kuku Yalanji headword grouped under that gloss is an accepted reference. The primary score uses NFC/casefold/spacing normalization and exact full-output equality. Approximate spelling, token containment, semantic plausibility, and chrF++ do not count as exact. They remain secondary diagnostics. Executed scoring logic is conceptually: ```python accepted = {normalize(form) for form in row["accepted_references"]} is_exact = normalize(prediction) in accepted ``` This prevents an arbitrary synonym from being treated as the only reference while maintaining a strict deployment criterion. ### 8.3 Confidence-adjusted lexical-reconstruction rule The gate computes a two-sided Wilson interval: ```python def wilson_interval(successes, rows, z=1.959963984540054): rate = successes / rows denominator = 1 + z * z / rows center = rate + z * z / (2 * rows) margin = z * ((rate * (1 - rate) / rows + z * z / (4 * rows * rows)) ** 0.5) return ((center - margin) / denominator, (center + margin) / denominator) ``` The model lexical-reconstruction gate passes only when all are true: ```text rows >= 297 normalized accepted-reference exact rate >= 0.80 Wilson 95% lower endpoint >= 0.80 empty outputs == 0 ``` For 297 rows the first passing count is 252, not 238. This is a deliberately conservative operational rule on a mechanically constructed frozen benchmark. The prompts are not a random sample of future requests, so the interval must not be presented as a population estimate or as proof of general translation reliability. ### 8.4 Statistical limits - Paired row bootstrap intervals describe the frozen rows, not new speakers or communities. - Sentence rows inside one narrative are dependent through discourse, speaker, topic, and transcription. - Multiple exploratory slices increase false-positive risk; they are diagnostic unless multiplicity is controlled. - Corpus chrF++ is nonlinear. Corpus deltas and mean sentence-score deltas answer different questions. - BLEU and chrF++ reward reference overlap and can both reward wrong propositions or penalize valid variants. - Exact match is appropriate for a stored canonical headword objective but too narrow for open sentence adequacy. - Three training seeds estimate some optimization variability, not speaker, dialect, or corpus variability. ### 8.5 Portable RunPod benchmark The sealed kit contains code, protocols, the 340 frozen rows, local prediction baselines, runtime locks, and hashes. Model weights are separate pinned payloads. On an NVIDIA A40, two deterministic executions matched each other and local CPU output on all 1,020 predictions across three checkpoints. ```bash cd /workspace/kuku-eval-kit ./bootstrap_and_run.sh test -f RUN_COMPLETE sha256sum -c OUTPUT_SHA256SUMS ``` All three checkpoint passes took about 3.5 minutes together. A candidate already present on a training pod should return a compact evaluation in roughly one to two minutes. ## 9. Forensic diagnosis of the current model ### 9.1 Isolated lexical supervision was underweighted v21.2 saw 16,642 synthetic sentences and 4,094 Bible replay rows per epoch, but only 357 unique dedicated dictionary-usage examples. It did not receive an explicit isolated pair for every eligible canonical headword. A word occurring inside a sentence is not the same objective as mapping an isolated English gloss to an exact lemma. ### 9.2 The existing dictionary probe is exposure-sensitive The 46/192 versus 2/105 seen/unseen split indicates that training exposure is strongly associated with exact retrieval. It does not prove causality by itself, because frequency and lexical difficulty are confounded, but it is strong enough to justify a controlled lexical-replay experiment. ### 9.3 Synthetic evaluation shares the authoring process The synthetic test has strong source-order and template difficulty structure. Later rows are shorter and more regular; early rows include longer and more varied clauses. A corpus chrF++ around 55 on this distribution is useful for regression control, but it cannot be read as 55 chrF++ on spontaneous or connected natural language. ### 9.4 Natural material is too small and too clustered v23 added only 156 unique native-speaker narrative clauses from one training speaker, selected on one development speaker, and tested on one speaker/text. The design was cleaner than prior work but underpowered at the cluster level. More epochs or another seed cannot manufacture speaker diversity. ### 9.5 Historical Bible dominance distorted objectives Bible data taught useful target-language surface patterns but dominated early mixtures and rewarded a free religious translation style. v20 showed that even oversampling small sets did not guarantee preservation in a large heterogeneous mixture. v24 should not use Bible rows as a positive training objective. ### 9.6 Repetition is partly a decoding problem and partly a model problem The guarded decoder greatly reduced loops on exact v21.2 weights. v23 still increased untagged loop rows under that guard, showing that training can alter degeneration risk. Decoder guards must be frozen on development data and cannot replace model-level retention checks. ### 9.7 Exact lookup and generation were previously conflated v9.9 and v16 achieved exact answers on known stored resources through lookup. Asking a model to regenerate the same fact adds error. The product should route known words and known sentence resources deterministically, then use a model only for novel composition. Component scores and whole-system scores must both be reported so lookup success is never attributed to model weights. ## 10. Current served translation architecture The homepage currently loads the complete selected dictionary, constructs a glossary, and calls `gpt-5.4-mini`. Translation mode requires a structured tool call validated by Zod. The system prompt treats user text as untrusted data rather than instructions. The response includes both a Kuku Yalanji draft and a literal English gloss. This is the temporary live path while custom models remain unmounted. The intended long-term router is: ```text request -> exact rights-cleared sentence lookup -> exact/unique dictionary lookup for lexical queries -> custom model only if that exact version passed its model gate -> optional dictionary-grounded hosted LLM fallback -> generation guards and research-draft label -> bilingual event log with model/version/provenance ``` The whole route should expose which branch answered. An exact dictionary lookup may be called `dictionary_exact`; it must not be logged as `model_exact`. ## 11. Why dictionary pairs belong in training The user-proposed intuition is correct: if a production requirement asks the model to translate isolated known words, the training objective should include isolated words. A sentence-only model has no reason to learn that its entire output for `woman` must be exactly `jalbu`. Explicit pairs provide: - direct loss on canonical headword retrieval; - repeated exposure to low-frequency lexical items; - a production-shaped one-word objective; - a measurable lexical learning curve; - a controlled way to test catastrophic forgetting against sentence tasks. They do not automatically provide: - sense disambiguation in context; - inflected forms such as case-marked nominals; - transitivity and argument structure; - verb conjugation or tense/aspect/mood; - derivational morphology; - discourse anaphora or participant tracking; - idiomatic collocations; - dialect and register selection; - semantic composition for unseen sentences. The correct conclusion is therefore **include the dictionary, but give it its own task design and its own claim**. ## 12. Proposed v24 hypotheses ### 12.1 Primary lexical hypothesis Adding source-audited isolated dictionary mappings under a visible `` task will materially increase closed-set accepted-reference exact reconstruction over a token- and step-accounted continuation control without increasing blank, degeneration, or sentence-retention failures. ### 12.2 Exposure-response hypothesis At a fixed token and optimizer-update budget, lexical exact reconstruction will show a positive but saturating response across low, medium, and high lexical exposure. The best recipe will be selected across seeds on lexical development plus sentence-retention gates, not on the final lexical test. ### 12.3 Acquisition and uptake hypotheses A model may reconstruct explicitly trained mappings while remaining weak at learning isolated lemmas from sentence context or applying a newly supplied glossary mapping in a novel sentence. v24 measures those three behaviors separately. Morphological generalization is measured only where the lemma is observed and a source-governed inflected or derived form can be held out. ### 12.4 Natural-language noninferiority hypothesis Lexical adaptation must not materially degrade held-out natural narrative, elder regression, dictionary usage, or synthetic retention. No claim of improved natural translation will be made unless a multi-text, multi-speaker natural endpoint improves under cluster-aware analysis and speaker review. ## 13. Proposed v24 data construction ### 13.1 Freeze source identities first Before generating a row, freeze and hash: - dictionary YAML, JSON-LD, and public JSONL; - source schema and renderer/export code; - v21.2 model and tokenizer files; - synthetic retention split; - database usage split; - narrative training, development, and test sources; - current 297-row probe; - builder, evaluator, trainer, environment lock, and RunPod image digest. The builder must refuse a nonempty output directory and any input hash drift. ### 13.2 Emit an eligibility ledger for all 2,688 entries Every source entry receives one decision record, even when no training pair is emitted: ```json { "entry_id": "dictionary-123", "headword": "jalbu", "decision": "include_canonical_lexeme", "reasons": [], "source_sha256": "...", "review_state": "project_reviewed_pending_speaker_verification" } ``` Exclusions and special handling must be additive and inspectable, for example: ```json { "entry_id": "dictionary-456", "decision": "defer_descriptive_definition", "reasons": ["english_definition_not_substitutable_translation"] } ``` This is a classification output, not a hand-maintained list of particular words. Mechanical schema rules should handle obvious structure; ambiguous semantic cases require recorded linguistic adjudication. ### 13.3 Task-visible row classes 1. **Direct lexical-reconstruction rows** ```text woman -> jalbu ``` These train the closed-set reconstruction objective. They do not share the sentence prefix. 2. **Sense-qualified rows** ```text bank river edge -> [canonical headword] ``` Use only where the source supplies a defensible sense distinction. Do not invent English sense labels from a model without review. 3. **Part-of-speech qualified rows** ```text light noun -> ... light adjective -> ... ``` POS qualification can reduce conflicting targets, but the unqualified production row still needs a documented preferred answer or deterministic lookup policy. 4. **Ordinary sentence rows** ```text The woman returned. -> [Kuku Yalanji sentence] ``` These remain the sentence task and provide the retention endpoint. 5. **Glossary-conditioned sentence rows** ```text The woman returned. woman = jalbu -> [Kuku Yalanji sentence] ``` These test whether supplied lexical evidence is used in context. A final uptake mapping is absent from all training and supplied only at evaluation. 6. **Attested usage rows** Preserve existing sentence pairs with their word ID and source. They teach context, not just lemma identity. 7. **Attested inflection/derivation rows** Extract only from reconciled examples and narratives. Record lemma, surface form, morphological analysis, and evidence span. Do not generate a paradigm mechanically and call it attested. 8. **Governed synthetic morphology rows** These may be generated from verified grammar rules when clearly labelled synthetic, checked against lexical and morphological constraints, reviewed, and held separate from attested material. ### 13.4 Multi-reference policy For evaluation, every canonical headword attached to one normalized English gloss may be accepted. For training, blindly duplicating one unqualified source with many unrelated target senses creates a conflicting conditional distribution. Use the following order: 1. retain a documented preferred canonical form for the unqualified production query; 2. emit sense/POS-qualified rows for distinguishable alternatives; 3. retain true synonyms as multi-reference metadata; 4. if synonymous targets are deliberately sampled, keep equal, recorded weights and evaluate all as accepted; 5. never discard alternatives from the source dictionary merely to make exact scoring easier. ### 13.5 Task-specific split and exposure design One seen/unseen flag is inadequate. Every lexical endpoint records four exposure states: ```text direct isolated-pair exposure in v24 sentence-context exposure in v24 documented exposure in earlier MobTranslate model lineage upstream NLLB exposure unknown ``` **Stage A: research fit** - For direct reconstruction, evaluate governed closed-set development mappings that received explicit lexical supervision; this is a retention/memorization endpoint and is labelled as such. - For contextual acquisition, withhold the isolated pair while allowing governed sentence-context exposure. - For lexicon uptake, withhold the mapping from all documented project-lineage training and supply it only in the inference glossary. Upstream NLLB exposure remains unknown. - For morphology, expose the lemma but hold out the governed surface form and context family. - Cluster by normalized Kuku headword family, English gloss/sense, source entry, and variant relationship so related rows cannot straddle development and final test accidentally. - Use development to select the exposure recipe and open each final task only after the recipe is locked. **Stage B: all-lexicon product fit** - Refit the selected recipe with every eligible dictionary cluster. - Evaluate the 297 production probe as closed-set model reconstruction and enforce only the lexical-model gate. - Do not call this an unseen lexical test. - Keep deterministic database lookup ahead of the model even if this gate passes. ### 13.6 Natural-language splits The next natural claim requires more reconciled source material before training. Development and test should each contain multiple speaker/text clusters. If that data is not yet available, v24 may proceed as a lexical experiment, but natural sentence performance must be a noninferiority claim only. At minimum: - no speaker in more than one split; - no story/text in more than one split; - variety labels balanced where possible rather than perfectly confounded with split; - exact and high-similarity source/target leakage checks; - discourse segments kept intact rather than splitting neighboring clauses across sets; - cluster-level intervals or randomization analyses in addition to row diagnostics; - blind fluent-speaker review before any natural-translation promotion. ## 14. Proposed v24 experiment design ### 14.1 Base model Primary controlled experiment: continue from the exact v21.2 merged weights. This isolates the effect of lexical supervision and reuses a known stable baseline. A stock-base or alternate architecture may be added as a separate arm, but it must not replace the controlled continuation in the same comparison. ### 14.2 Arms The primary causal question is substitution at fixed compute: does replacing scheduled retention tokens with lexical-task tokens improve the intended endpoint without unacceptable retention loss? Use fixed tokens and fixed optimizer updates, not merely fixed epochs: | Arm | Treatment | Purpose | |---|---|---| | B0 | none | Untouched v21.2 baseline; no continuation | | C0 | retention only | Token- and step-accounted continuation control | | L1 | low direct lexical exposure | Minimum explicit reconstruction treatment | | L2 | medium direct lexical exposure | Moderate reconstruction treatment | | L4 | high direct lexical exposure | Strong reconstruction treatment | | G | glossary-conditioned sentences | Lexicon-uptake treatment | C0 and each treatment arm must use the same exact optimizer horizon and planned non-padding token budget. The manifest reports actual examples and source/target tokens consumed by task. One additive-compute treatment may be run later, but it answers a different question and is not part of the primary screen. ### 14.3 Efficient two-stage search To control RunPod cost without weakening the confirmatory result: 1. Run a preregistered one-seed screening study for C0/L1/L2/L4/G using development endpoints only; B0 is evaluated without training. 2. Eliminate arms only by frozen rules, not informal sample reading. 3. Carry C0 and the top one or two treatment recipes into paired seeds 17, 42, and 73. 4. Select the recipe from aggregate development behavior across seeds; do not tune the seed. 5. Open final acquisition, uptake, morphology, and natural-retention tests for every confirmatory seed once. 6. If Stage A passes, perform one all-lexicon product refit with the locked recipe. Screening results are exploratory; the three-seed confirmation supplies the reported effect. Five seeds are added only if the screen shows material variance that three seeds cannot characterize. ### 14.4 Initial retention mixture Exact counts depend on the eligibility audit and must be frozen before the run. The recommended categories are: ```text lexical rows none / low / medium / high by C0/L1/L2/L4 glossary-conditioned rows G arm only database usage rows retained, source-disjoint from tests synthetic retention fixed stratified sample, not all 16,642 by default reconciled narrative train only attested, rights-cleared clauses Bible rows 0 in training elder diagnostic rows 0 in training contextual-acquisition test no isolated-pair exposure; governed sentence context permitted lexicon-uptake test mapping absent from documented project lineage; supplied only at inference morphology test lemma observed; governed surface form/context family held out held-out speakers/texts 0 in training ``` The synthetic retention sample should be stratified by grammar, frame, domain, source-order quartile, sentence length, and quality tier. Do not use an ad hoc handful of easy templates. ### 14.5 Hyperparameter starting point Use the conservative v23 continuation settings as a starting proposal, not as a frozen decision: ```text LoRA rank / alpha / dropout 32 / 64 / 0.05 Target modules q_proj,k_proj,v_proj,out_proj,fc1,fc2 Learning rate 1e-5 Batch / accumulation 8 / 2 Effective batch 16 Warmup / weight decay 0.05 / 0.01 Decoder frozen v21.2 guarded policy Seeds 17, 42, 73 ``` The final preregistration must set an exact positive `max_steps`; epoch count is descriptive only. Checkpoint cadence should align to complete passes through the smallest high-value component, and early stopping must be based on the frozen development composite, not training loss. v23 showed validation loss continuing to fall after natural chrF++ peaked. ### 14.6 Development selection function Do not collapse every objective into a weighted average that permits one domain to buy off a failure in another. Use lexicographic/all-checks logic: 1. zero blanks and hard degeneration ceiling; 2. natural and synthetic noninferiority floors; 3. contextual-acquisition or glossary-uptake development improvement, according to the arm; 4. closed-set lexical-reconstruction development improvement; 5. usage-example retention; 6. among eligible checkpoints, choose the lowest update count or highest primary lexical result as preregistered. Bible is not a positive selector. It is only a catastrophic floor. ## 15. Proposed v24 evaluation and promotion gates ### 15.1 Research-task gates - **Direct reconstruction:** ambiguity-aware normalized accepted-reference exact after explicit pair exposure. - **Contextual acquisition:** isolated reconstruction after sentence-context exposure but no direct pair. - **Lexicon uptake:** supplied held-out mapping used correctly in a novel sentence, including governed morphology where the source supports that analysis. - **Morphology:** required feature and surface-form accuracy over lemma-observed, form-held-out records. - Secondary diagnostics: grapheme-cluster and code-point CER, multi-reference chrF++, copying, tokenization class, orthographic validity, and error class. - Report all documented-lineage exposure strata and mark upstream exposure unknown. - Compare systems with paired, cluster-aware analysis at the lexeme/sense/source-entry level. No numeric minimum should be invented after looking at results. The preregistration should set a meaningful effect floor after a baseline power/sensitivity analysis on the frozen row count. ### 15.2 Model lexical-reconstruction gate The existing rule remains: ```text at least 297 rows at least 252 accepted exact at 297 rows point exact >= 80% Wilson 95% lower bound >= 80% zero blank outputs exact model ID and exact merged-weight hash bound into the gate ``` This gate is a conservative rule on the frozen benchmark, not an estimate of future-query reliability. It authorizes only a model lexical-reconstruction route. Add these operational requirements: - no source copying; - no output outside accepted orthographic shape for isolated words; - no generation-limit termination; - service smoke and concurrency test; - model-only score and whole-router score reported separately. ### 15.3 Model sentence-generation gate - Synthetic tagged and untagged retention within a frozen noninferiority margin. - Dictionary-usage retention within a frozen margin. - Elder regression reported with no positive promotion weight unless speaker review supports it. - Multi-speaker natural endpoint required for any improvement claim. - Zero blanks. - Repeated-segment rows no worse than v21.2 guarded. - Severe under- and over-translation no worse than baseline. A lexical PASS cannot satisfy this gate. Public sentence generation additionally requires multiple independent speaker/text clusters, fluent-speaker semantic and grammatical review, no critical participant/negation/safety errors, and documented release authority for that scope. Every current Kuku model is `FAIL` on this gate. ### 15.4 Dictionary-router gate The deterministic route is tested separately for exact retrieval, sense/POS handling, ambiguity display, source and variety labels, route metadata, latency, and absence of accidental model attribution. It is not authorized by either model gate and does not require a neural model to be mounted. ### 15.5 Human linguistic review Blind reviewers should not see model names. At minimum, annotate: | Dimension | Question | |---|---| | Semantic adequacy | Are participants, events, properties, negation, quantity, and relations preserved? | | Grammaticality | Is the output well-formed independent of the reference wording? | | Case and argument structure | Are agent, patient, dative, locative, ablative, and other relations correct? | | TAM and clause type | Are tense, aspect, mood, imperative, question, and subordination represented correctly? | | Morphology | Are inflection, derivation, reduplication, and clitic/particle behavior valid? | | Lexical choice | Is the sense appropriate, not merely a dictionary-related word? | | Discourse | Are reference tracking, topic continuity, information structure, and clause linkage coherent? | | Variety/register | Is the output compatible with the intended variety, speaker context, and genre? | | Cultural acceptability | Is the expression appropriate to publish or teach? | Corrections should become versioned evidence with reviewer role, confidence, reason, and rights, not silent overwrites. ## 16. RunPod execution plan ### 16.1 Preflight on the local box ```bash PROGRAM=/mnt/donto-data/donto-resources/research/translation-training/kuku-yalanji-runpod-2026-06-30 REPO=/mnt/donto-data/workspace/mobtranslate.com # Build into a new empty directory; never overwrite a prior edition. python3 "$REPO/training/translation/build_v24_lexicon_grounded.py" \ --output-dir "$PROGRAM/prepared/v24.0-lexicon-grounded" cd "$PROGRAM/prepared/v24.0-lexicon-grounded" sha256sum -c SHA256SUMS.v24-inputs ``` `build_v24_lexicon_grounded.py` is proposed work and does not exist at the publication time of this report. The command defines the intended contract, not a claim that v24 data has already been built. Preflight must also run unit tests, validate row and sense counts, verify disjoint groups, scan tokenizer fertility and lengths without truncation, audit exposure against the full documented model lineage, verify the exact v21.2 base and custom `gvn_Latn` embedding/output path, and produce the preregistration before a pod starts. Data whose research use or third-party transfer is unresolved is excluded or quarantined; no approval state is invented. ### 16.2 Proposed exact trainer invocation The executed trainer already supports the necessary controls. A candidate invocation will look like: ```bash python training/translation/train_nllb_lora.py \ --base-model /workspace/v24/base/merged \ --train-file /workspace/v24/data/train.eng-gvn.jsonl \ --validation-file /workspace/v24/data/validation.eng-gvn.jsonl \ --output-dir /workspace/v24/models/seed-42 \ --model-id mobtranslate/kuku-yalanji-nllb-lora \ --model-version v24.0-lexicon-grounded-seed42 \ --direction eng-gvn \ --source-lang eng_Latn \ --target-lang gvn_Latn \ --max-source-length 192 \ --max-target-length 208 \ --learning-rate 1e-5 \ --max-steps "$FROZEN_MAX_STEPS" \ --batch-size 8 \ --gradient-accumulation-steps 2 \ --warmup-ratio 0.05 \ --weight-decay 0.01 \ --lora-r 32 \ --lora-alpha 64 \ --lora-dropout 0.05 \ --lora-target-modules q_proj,k_proj,v_proj,out_proj,fc1,fc2 \ --generation-num-beams 1 \ --generation-no-repeat-ngram-size 4 \ --generation-repetition-penalty 1.10 \ --generation-length-penalty 1.0 \ --seed 42 \ --full-determinism ``` The exact update horizon, token budget, task schedule, and save cadence must come from the frozen v24 preregistration. The trainer fails if its observed optimizer step differs from `--max-steps` and records actual examples and post-truncation source/target tokens consumed by task. This snippet is a reviewable starting contract, not permission to launch with unfrozen counts. ### 16.3 Live verification while running A started process is not a working process. For every seed and arm, monitor: ```bash nvidia-smi ps -eo pid,etime,rss,pcpu,args | rg 'train_nllb_lora.py' find /workspace/v24/models -name trainer_state.json -print tail -n 50 /workspace/v24/logs/*.log df -h /workspace ``` Verify optimizer steps advance over wall-clock time, GPU activity is real, output files grow, RSS remains bounded, and no empty provider/service dependency is silently treated as success. ### 16.4 Evaluate before downloading weights ```bash cd /workspace/kuku-eval-kit ./bootstrap_and_run.sh \ /workspace/v24/models/selected/merged test -f RUN_COMPLETE sha256sum -c OUTPUT_SHA256SUMS ``` The current kit accepts manifest-registered checkpoints; a v24 wrapper must add the candidate and its hashes without mutating v1 evidence. Download compact predictions, metrics, manifests, logs, and gates first. Pull multi-gigabyte weights only after promotion remains possible. ### 16.5 Cost containment and cleanup - Register automatic stop and termination timers before training. - Keep a local operations ledger with UTC timestamps. - Download compact evidence incrementally after each seed. - On failed screening, delete all adapters, merged weights, optimizer checkpoints, and caches after compact evidence is verified. - For confirmatory runs, retain compact adapters when source rights permit independent reproduction; delete merged models, optimizer states, redundant checkpoints, and caches after the decision. - On pass, retain only the selected final adapter/merged model plus compact evidence; delete unselected bulky weights. - Verify deletion and the active pod list. - Record actual billed cost rather than extrapolating from a stale price. ## 17. Proposed v24 lexical builder excerpt This is proposed code for review. It illustrates structured handling, not a finished implementation: ```python from collections import defaultdict from dataclasses import dataclass import hashlib import json import unicodedata @dataclass(frozen=True) class LexicalPair: entry_id: str english: str kuku: str part_of_speech: str | None sense: str | None source: str def norm(text: str) -> str: return " ".join(unicodedata.normalize("NFC", text).casefold().split()) def stable_id(pair: LexicalPair, task: str) -> str: payload = "\0".join((pair.entry_id, pair.english, pair.kuku, task)) return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:24] def lexical_reconstruction_row(pair: LexicalPair) -> dict: english = " ".join(pair.english.split()) kuku = " ".join(pair.kuku.split()) return { "id": f"lexeme:{stable_id(pair, 'reconstruction')}", "direction": "eng-gvn", "input_text": f" {english}", "unconditioned_input_text": english, "output_text": kuku, "pair_kind": "dictionary_lexeme", "source_lang": "eng_Latn", "target_lang": "gvn_Latn", "lexicon": { "entry_id": pair.entry_id, "part_of_speech": pair.part_of_speech, "sense": pair.sense, "source": pair.source, }, } def group_references(pairs: list[LexicalPair]) -> dict[str, list[str]]: grouped: dict[str, list[str]] = defaultdict(list) for pair in pairs: key = norm(pair.english) if pair.kuku not in grouped[key]: grouped[key].append(pair.kuku) return dict(grouped) def write_jsonl(path, rows): with path.open("w", encoding="utf-8") as handle: for row in rows: handle.write(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n") ``` The finished builder must add source hash validation, eligibility/exclusion ledgers, sense qualification, separate contextual-acquisition and glossary-uptake records, group splitting, full-lineage exposure audits, deterministic sorting and schedules, quarantine, rights checks, token budgets, tests, manifests, and fail-closed assertions. ## 18. Split deployment interlocks The lexical gate generator is invoked after a candidate's closed-set reconstruction audit: ```bash python training/translation/verify_dictionary_cpu_deployment.py \ --lexicon-audit analysis/version_probe_analysis.json \ --model-label candidate \ --model-id v24.0-lexicon-grounded-product \ --model-sha256 "$MERGED_MODEL_SHA256" \ --minimum-rows 297 \ --minimum-exact-rate 0.80 \ --minimum-wilson-lower 0.80 \ --output analysis/cpu-deployment-gate.json ``` Systemd checks the gate before loading any model: ```bash python3 training/translation/check_dictionary_cpu_deployment_gate.py \ --gate /etc/mobtranslate/kuku-cpu-deployment-gate.json \ --model-id v24.0-lexicon-grounded-product \ --model-file /path/to/merged/model.safetensors ``` The lexical checker refuses startup unless the gate type is `model_lexical_reconstruction`, its decision is the matching allow value, every required check is true, the model ID matches, the model file exists, and its full SHA-256 matches. A passing report for one model cannot authorize another model. Sentence services have a second independent preflight: ```bash python3 training/translation/check_model_bound_release_gate.py \ --gate /etc/mobtranslate/kuku-v24-sentence-generation-gate.json \ --gate-type model_sentence_generation \ --allowed-decision SENTENCE_GENERATION_ALLOWED \ --model-id v24.0-selected \ --model-file /path/to/merged/model.safetensors ``` Both existing Kuku sentence-service units carry this second interlock with a model-bound `FAIL` gate. Therefore a future 297-word lexical PASS cannot mount a free-form sentence service. The deterministic dictionary router has its own route tests and does not depend on either model service. ## 19. Model hosting after a pass Only a candidate whose applicable governance, rights, and release scope are documented can become a public model release. That release should include: - exact frozen base plus adapter bundle for dynamic LoRA hosts such as Atlas; - standalone merged Transformers model; - tokenizer and custom `gvn_Latn` metadata; - guarded decoding policy; - immutable version and all SHA-256 values; - model card, training report, data manifest, rights, and limitations; - model-only and whole-router benchmark results; - smoke inputs and expected outputs; - OpenAPI links and a minimal HTTP request/response contract. Dynamic LoRA loading remains: ```python from peft import PeftModel from transformers import AutoModelForSeq2SeqLM, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained( "./adapter", src_lang="eng_Latn", tgt_lang="gvn_Latn" ) base = AutoModelForSeq2SeqLM.from_pretrained("./base") model = PeftModel.from_pretrained(base, "./adapter") ``` The host must use the distributed exact base, not stock NLLB. Every response must identify immutable model version, decode policy, and route. Public output remains a research draft unless the applicable rights and release authority are documented for that scope. ## 20. Artifact and retention policy The program keeps scientific evidence while avoiding a warehouse of weak model weights: | Artifact | Passing run | Failed run | |---|---|---| | Preregistration and input hashes | retain | retain | | Builder/trainer/evaluator source snapshot | retain | retain | | Environment and image lock | retain | retain | | Row-level predictions and metrics | retain | retain | | Resource traces, logs, cost, pod ledger | retain | retain | | Research, task, and release gates | retain | retain | | Selected adapter and merged model | retain | compact adapter only for confirmatory run when rights permit | | Unselected seed weights | delete bulky weights | delete screening weights; confirmatory compact adapter may remain | | Optimizer checkpoints/caches | delete after decision | delete after decision | No archive is overwritten. A new model receives a new immutable edition. Negative results remain registered as reports without hostable model downloads. Deletion is recorded by file count, byte count, reason, and post-delete verification. ## 21. Risks and mitigations | Risk | Why it matters | Mitigation | |---|---|---| | Closed-set score presented as generalization | Training all 297 prompts makes the gate a reconstruction/memorization test | Publish reconstruction, contextual acquisition, glossary uptake, and morphology as separate tasks | | Arbitrary hidden mapping called productive transfer | A target string is not derivable from its English gloss | Test supplied-glossary uptake or lemma-observed morphology instead; audit upstream exposure as unknown | | Polysemy collapsed | Same English gloss can have different senses/forms | Preferred production form plus sense/POS-qualified rows and multi-reference metadata | | Dictionary definition treated as translation | Descriptive prose may not substitute in a sentence | Eligibility ledger and linguistic adjudication | | Lemma memorization without morphology | Sentence output still fails case/TAM/derivation | Attested and governed contextual morphology tasks | | Synthetic authoring bias | Train and test share templates/analysis conventions | Independent natural clusters and speaker review | | Variety confounding | Nyungkul/Yalanji, speaker, text, and split can align | Multi-cluster balanced design and explicit labels | | Bible register contamination | Large free translations dominate gradients | Zero Bible training weight; catastrophic test only | | More updates or shorter rows confound replay effect | Equal rows or steps do not equal token exposure | Fixed optimizer and token budget across primary arms; report actual consumed tokens by task | | Seed instability | One run can be accidental | Three-seed confirmation after frozen screening | | Test leakage through iteration | Repeatedly viewed controls cease to be blind | New sealed tests, one-time opening, immutable selection lock | | Valid reduplication blocked | No-repeat decoder can damage morphology | Speaker error analysis and separate guard-damage annotations | | Metric gaming | Exact or chrF can improve without semantic adequacy | Multi-endpoint hard gates plus blinded human review | | Weak weights consume local disk | Root and mounted disks are constrained | Evaluate on pod, result-gated pullback, delete failed weights | | Wrong base attached to adapter | Custom token/base lineage is project-specific | Bundle exact base; hash every behavior-affecting file | | Lexical score mounts sentence service | Word memorization does not establish sentence competence | Independent model-bound lexical and sentence gates in systemd | | Service mounted despite failed score | Operational drift can bypass prose policy | Root-owned gate-type, model-ID, and weight-hash systemd interlocks | | Rights or authority overclaim | Public availability and copyright terms do not establish every release scope | Source-level rights, unresolved governance state, no fabricated approval, fail-closed public release | ## 22. Decision tree for the next run ```text Is the 2,688-entry sense/eligibility ledger complete, with rights and uncertain records quarantined? no -> do not train; finish data decisions yes -> freeze task-specific clusters, full-lineage exposure, token schedules, source hashes, and preregistration Do all split/leakage/token-length/unit tests pass? no -> quarantine/fix additively; do not launch yes -> run fixed-token and fixed-step screening arms on RunPod Do any lexical arms pass development safety/noninferiority? no -> retain compact negative result; delete weights yes -> run control + finalists over seeds 17/42/73 Does a treatment improve its declared reconstruction/acquisition/uptake endpoint without sentence regression? no -> no model edition; preserve evidence yes -> lock recipe and run all-lexicon product fit Does product fit achieve >=252/297 exact, Wilson lower >=80%, zero blanks, and lexical safety gates? no -> do not mount lexical model; dictionary lookup remains first route yes -> generate model-bound lexical gate; this still does not authorize sentence generation Do multi-cluster natural evaluation, fluent-speaker review, critical-error rules, rights, and release authority pass? no -> keep sentence-generation gate FAIL and service unloaded yes -> issue a separate model-bound sentence gate and consider a research-draft sentence route ``` ## 23. Immediate implementation order 1. Preserve the independent-review disposition and revise this dossier. **Done.** 2. Correct public ownership/licensing claims, registry status semantics, and add the v23 negative result. **Done in source; static docs and the dynamic registry API are live, while the compiled Next.js copy awaits I/O quiescence.** 3. Add exact `max_steps`, actual task/token exposure accounting, and separate lexical/sentence interlocks. **Done and locally tested; a real trainer smoke test in the RunPod image remains.** 4. Build `build_v24_lexicon_grounded.py` with tests and a 2,688-entry sense/decision ledger. 5. Produce counts for canonical, ambiguous, sense-qualified, variant, deferred, restricted, and excluded records. 6. Freeze reconstruction, contextual-acquisition, glossary-uptake, morphology, and retention groups; audit exposure through every documented model ancestor. 7. Audit tokenizer fertility, the custom language-token embedding/output rows, and base+adapter versus merged parity. 8. Write the v24 preregistration with exact row/token counts, `max_steps`, arms, seeds, gates, and retention rules. 9. Extend the sealed RunPod kit to accept v24 manifest entries without changing v1 evidence. 10. Run one-seed fixed-token screening, then paired three-seed confirmation only for selected recipes. 11. Pull compact evidence first, apply frozen task gates, and refit all eligible lexical rows only after recipe lock. 12. Package weights only for a passed, rights-permitted scope; rerender living books and update public checksums. ## 24. Reviewer reproducibility map Canonical mounted-drive root: ```text /mnt/donto-data/donto-resources/research/translation-training/ kuku-yalanji-runpod-2026-06-30/ ``` Important internal records: ```text docs/KUKU-YALANJI-V21-2-MODEL-AND-HOSTING-GUIDE.md docs/KUKU-YALANJI-V22-STEP-MATCHED-REPLAY-AND-DECODER-TRANSFER-2026-07-14.md docs/KUKU-YALANJI-V23-ATTESTED-NARRATIVE-PREREGISTRATION-2026-07-14.md docs/KUKU-YALANJI-V23-ATTESTED-NARRATIVE-ADAPTATION-2026-07-14.md docs/KUKU-YALANJI-RUNPOD-EVALUATION-LOOP-2026-07-14.md reports/KUKU-YALANJI-V21-INDEPENDENT-MODEL-COMPARISON-2026-07-12.md reports/kuku-yalanji-v20-third-opinion-technical-report-2026-07-02.md reports/kuku-yalanji-training-resource-catalogue-2026-07-02.md prepared/v21.2-claude-balanced-replay/ runpod/v21.2-claude-balanced-replay-20260711T050900Z/ runpod/v22.0-step-matched-replay-20260714T023516Z/ runpod/v23.0-attested-narrative-20260714T143510Z/ ``` Reusable code: ```text /mnt/donto-data/workspace/mobtranslate.com/training/translation/ train_nllb_lora.py evaluate_nllb_lora.py build_v21_2_claude_balanced_replay.py build_v23_attested_adaptation.py build_kuku_lexicon_probe.py analyze_kuku_version_probe.py compare_v21_models.py verify_dictionary_cpu_deployment.py check_dictionary_cpu_deployment_gate.py apply_v23_retention.py ``` Public reports: - [v21.2 model, training, data, and hosting guide](/docs/kuku-v21-2-model-guide.html) - [v21 independent comparison](/docs/model-comparison.html) - [v22 step-matched replay and decoder transfer](/docs/kuku-v22-experiment.html) - [v23 preregistration](/docs/kuku-v23-preregistration.html) - [v23 negative result](/docs/kuku-v23-attested-adaptation.html) - [RunPod evaluation loop](/docs/kuku-runpod-evaluation-loop.html) - [Operator's guide](/docs/operator-guide.html) - [Grammar cheatsheet](/docs/grammar-cheatsheet.html) - [Dictionary errata and phantom watchlist](/docs/dictionary-errata.html) - [Model API](/api/v1/models/kuku-yalanji-nllb-lora) - [Human model browser](/models) ## 25. Final recommendation Do not run another broad, heterogeneous continuation and hope that aggregate scale solves the problem. The next experiment should be narrow enough to answer separable questions cleanly: **does explicit, source-audited lexical supervision improve closed-set reconstruction, and can the model acquire a lemma from context or use newly supplied glossary evidence without damaging sentence behavior?** Build visible task objectives, match update and token exposure, use paired seeds, audit the full model lineage, then refit all eligible entries for the closed-set model gate. Keep Bible out of the positive objective. Treat natural sentence improvement as a separate claim that requires more multi-speaker, multi-text attested data and blinded fluent-speaker review. Until the independent gates pass, keep exact dictionary and sentence resources lookup-first, keep custom Kuku model CPU services unloaded, and label hosted-LLM output as an unverified draft. A lexical PASS may justify a research reconstruction route but never a sentence route. Public release additionally remains bounded by source rights and documented authority rather than a governance state invented by the technical team. --- **Document status:** public post-review revision with an explicit disposition record. Corrections are additive and versioned. The raw Markdown at is the requested shareable edition; the spelling `trainignplan` is preserved intentionally to match the requested URL.