Kuku Yalanji v24.3 Model, Training, Evaluation, and Hosting Guide

Published: 2026-07-15 UTC
Model ID: kuku-yalanji-nllb-lora
Release: v24.3-joint-lexeme-dose29-s3598-20260715
Run: v24.3-joint-lexeme-dose29-rerun1
Selected checkpoint: J69_s3598
Direction: English (eng_Latn) to Kuku Yalanji (gvn_Latn)
Status: runtime-verified research candidate; closed-set lexical gate passed; sentence translation not validated
Rights: noncommercial research; upstream NLLB CC BY-NC 4.0 and source-specific data terms apply

1. Read this first

This is the complete handoff for the first MobTranslate Kuku Yalanji model that successfully reconstructs almost all entries in the governed 2,724-item dictionary census under its trained <lexeme> task. It also preserves and improves the measured development sentence endpoints relative to the v21.2 starting model. The adapter has been loaded and served on both an NVIDIA RTX A6000 and MobTranslate's CPU host.

The model is working in the engineering sense: its artifact is intact, its task token is restored correctly, both trained prefixes generate output, its CPU service fails closed during warm-up, and a CPU lexical smoke test matches the archived A6000 prediction. The scientific claim is narrower:

The run's machine decision was ADVANCE, meaning advance to further research. Its own manifest explicitly says promotion_or_deployment_authorized: false. Publishing a reproducible model and allowing an operator-requested preview does not convert that research decision into a linguistic certification.

For known dictionary facts, deterministic lookup remains the safer product route. Neural lexical reconstruction is useful as a capacity and regression audit, not as a replacement for the source dictionary.

2. Public release and API

2.1 Versioned API

The model registry is read-only, cross-origin accessible, and requires no API key.

Always pin the literal version and SHA-256. latest is intentionally mutable.

curl -fsS \
  https://mobtranslate.com/api/v1/models/kuku-yalanji-nllb-lora/versions/v24.3-joint-lexeme-dose29-s3598-20260715 \
  | jq '.release | {version, status, metrics, artifacts}'

2.2 Model downloads

Use the complete bundle for the shortest path to a compatible host. Use the adapter-only bundle only when the host already has the exact v21.2 merged project base identified in section 4. Stock NLLB is not a substitute.

2.3 Exact training and evaluation data

The compressed package is small because repeated presentations compress well. The raw JSONL preserves the exact materialized schedule used by the trainer.

3. What v24.3 changed

v21.2 was good at its synthetic sentence domain but reconstructed only 48/297 historical dictionary prompts. The diagnosis was lexical underexposure. Early v24 dose screens showed that a lexical-only objective could improve word recall while compressing sentence outputs. v24.3 therefore used joint replay:

<lexeme> woman
    -> jalbu

<translate> The woman saw the water.
    -> Jalbu-ngku bana nyajin.

The causal question was:

Can increasing every governed lexeme from 13 to 29 explicit presentations, while retaining the exact non-Bible sentence set at two presentations, pass the frozen closed-set lexical gate without the sentence compression seen in lexical-heavy earlier experiments?

The changed factors were lexical repetitions, total optimizer updates, and learning-rate trajectory length. The base weights, tokenizer lineage, sentence identities, sentence repetitions, seed, learning rate, LoRA topology, batch size, length caps, evaluation files, and primary decoder were held fixed.

4. Architecture and immutable identity

4.1 Lineage

facebook/nllb-200-distilled-1.3B
  -> MobTranslate project continuation with gvn_Latn
  -> v21.2 balanced-replay merged model
  -> v24.3 J69 LoRA + trainable <lexeme> token row
  -> selected checkpoint step 3,598

gvn_Latn is a project-added tokenizer language code. Its presence does not mean the upstream NLLB release listed Kuku Yalanji as a supported language.

4.2 Required base

The base already contains gvn_Latn at ID 256204 and has vocabulary size 256205. v24.3 adds <lexeme> at ID 256205, making the adapter tokenizer length 256206. The host must resize the base embeddings before attaching the adapter.

4.3 Adapter

The adapter is much larger than the 47.2 million LoRA parameters alone would imply. PEFT persisted full selective token embedding/output matrices along with the trainable <lexeme> row. That is correctness-preserving but adds roughly a gigabyte and raises dynamic CPU memory. Do not strip those tensors without an adapter-versus-repacked parity test over the full benchmark.

5. Loading the model

5.1 Proven package versions

The successful RunPod run used:

Python        3.11.10
PyTorch       2.4.1+cu124
Transformers  4.48.3
PEFT          0.19.1
Accelerate    1.14.0
Datasets      4.8.5
SacreBLEU     2.6.0
SentencePiece 0.2.2
GPU           NVIDIA RTX A6000 48GB

5.2 Dynamic PEFT load

import torch
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(
    "./adapter",
    src_lang="eng_Latn",
    tgt_lang="gvn_Latn",
)
base = AutoModelForSeq2SeqLM.from_pretrained(
    "./base",
    torch_dtype=dtype,
)

if base.get_input_embeddings().num_embeddings != len(tokenizer):
    base.resize_token_embeddings(len(tokenizer))

model = PeftModel.from_pretrained(
    base,
    "./adapter",
    is_trainable=False,
    low_cpu_mem_usage=True,
)
model.to(device=device, dtype=dtype)
model.eval()

The final dtype conversion matters. In a real CPU preflight, the base loaded as float32 while restored selective token/output tensors remained bfloat16, causing float != c10::BFloat16 in the output projection. Normalizing the complete PEFT model fixed the mismatch. The reference service refuses readiness if its warm-up generation fails.

5.3 Generate by task

def generate(text: str, task: str) -> str:
    if task not in {"lexeme", "translate"}:
        raise ValueError("task must be lexeme or translate")

    prefix = "<lexeme> " if task == "lexeme" else "<translate> "
    inputs = tokenizer(
        prefix + text,
        return_tensors="pt",
        truncation=True,
        max_length=128,
    ).to(device)

    if task == "lexeme":
        decoder = dict(
            max_new_tokens=32,
            num_beams=1,
            no_repeat_ngram_size=0,
            repetition_penalty=1.0,
            length_penalty=1.0,
        )
    else:
        decoder = dict(
            max_new_tokens=208,
            num_beams=1,
            no_repeat_ngram_size=4,
            repetition_penalty=1.10,
            length_penalty=1.0,
        )

    with torch.inference_mode():
        output = model.generate(
            **inputs,
            forced_bos_token_id=tokenizer.convert_tokens_to_ids("gvn_Latn"),
            **decoder,
        )
    return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

The lexical decoder above exactly matches the selected lexical evaluation. The guarded sentence policy is an operational preview setting inherited from the separately frozen v21.2 decoder-transfer study. The controlled v24.3 sentence results in section 9 used the neutral evaluation decoder: beam 1, 32 new tokens, no repeat guard, repetition penalty 1.0, and length penalty 1.0.

6. Reference HTTP service

The complete bundle includes serve_v2_infer.py.

python serve_v2_infer.py \
  --model-dir ./base \
  --adapter-dir ./adapter \
  --adapter-sha256 dd61583a60df2d538989e963e104cb626d78965d300e4e473e9a82ef59c04502 \
  --model-id v24.3-joint-lexeme-dose29-s3598-20260715 \
  --host 127.0.0.1 \
  --port 7955 \
  --dtype float32 \
  --threads 4 \
  --max-chars 400 \
  --max-waiting 2 \
  --max-time 45

Health:

curl -fsS http://127.0.0.1:7955/health | jq

Lexical reconstruction:

curl -fsS -H 'content-type: application/json' \
  --data '{"text":"woman","task":"lexeme"}' \
  http://127.0.0.1:7955/translate | jq

{
  "translation": "jalbu",
  "kuku": "jalbu",
  "model": "v24.3-joint-lexeme-dose29-s3598-20260715",
  "task": "lexeme",
  "sourceLang": "eng_Latn",
  "targetLang": "gvn_Latn"
}

Sentence research preview:

curl -fsS -H 'content-type: application/json' \
  --data '{"text":"The woman saw the water.","task":"translate"}' \
  http://127.0.0.1:7955/translate | jq

Measured smoke output:

Jalbu-ngku bana nyajin.

The server binds loopback, permits one active generation, bounds its wait queue, rejects oversized input, and returns 429 when saturated. Put authentication, TLS, rate limits, and public labels in the reverse proxy or caller.

7. Measured resources

7.1 Training

The 36.6 GiB peak includes training-state and checkpoint behavior. Inference does not require a 48GB GPU, but the minimum should be measured on the intended stack instead of inferred from parameter size.

7.2 Dynamic CPU serving

The CPU test used an eight-vCPU AVX2-only Haswell virtual host, four Torch threads, float32, no model-process swap, and one active generation.

The broad latency range was caused by variable page-cache and mounted-disk pressure on a shared VPS. It is evidence that the CPU runtime works, and also evidence that this dynamic form is not a good high-throughput service on that box. A GPU host is preferred. A correctly merged model should remove duplicated adapter embedding storage, but no merged v24.3 bundle should be released until its predictions match the dynamic adapter on the frozen suites.

8. Exact training treatment

8.1 Materialized mixture

Total non-padding tokens were 2,391,195. Every materialized row was seen once by the trainer. There were no Bible rows in the v24.3 positive objective.

The 115,136 count is not a claim of 115,136 independent speaker-attested examples. It is a deterministic presentation schedule built from 20,794 unique lexical or sentence rows. Most sentence material is synthetic and pending fluent speaker review.

8.2 Data contracts

• The 18,070 unique sentence rows were retained from the v21.2 treatment after excluding 4,094 verse rows.
• Sentence train and sentence validation had zero normalized source overlap.
• Sentence train and sentence validation had zero normalized target-surface overlap.
• The 128-row optimization monitor was balanced between 64 lexical and 64 sentence records.
• Its lexical half overlapped training by design and was labelled an optimization diagnostic.
• The 2,724-row census was evaluated at every saved checkpoint.
• The 2,086-row retention suite was evaluated at every saved checkpoint.
• Final tests for unseen lexical uptake, morphology, and speaker-diverse natural translation were not opened because those governed test resources do not yet exist at sufficient scale.

8.3 Training command

The archived driver enforced hashes, row counts, token identities, and fresh output roots before running this recipe:

python scripts/train_nllb_lora.py \
  --base-model ./base_model/merged \
  --train-file data/joint/train.eng-gvn.jsonl \
  --validation-file data/joint/validation.eng-gvn.jsonl \
  --output-dir models/J69 \
  --model-version v24.3-J69-rerun1 \
  --run-id v24.3-J69-seed17-rerun1 \
  --dataset-id v24.3-joint-lexeme-dose29 \
  --direction eng-gvn \
  --source-lang eng_Latn \
  --target-lang gvn_Latn \
  --additional-special-token '<lexeme>' \
  --trainable-token '<lexeme>' \
  --max-steps 3598 \
  --epochs 999 \
  --batch-size 32 \
  --gradient-accumulation-steps 1 \
  --max-source-length 128 \
  --max-target-length 128 \
  --learning-rate 2e-4 \
  --warmup-ratio 0.05 \
  --weight-decay 0.0 \
  --lora-r 32 \
  --lora-alpha 64 \
  --lora-dropout 0.0 \
  --lora-target-modules q_proj,k_proj,v_proj,out_proj,fc1,fc2 \
  --eval-steps 900 \
  --save-steps 900 \
  --save-total-limit 4 \
  --generation-num-beams 1 \
  --generation-no-repeat-ngram-size 0 \
  --generation-repetition-penalty 1.0 \
  --generation-length-penalty 1.0 \
  --seed 17 \
  --full-determinism \
  --no-shuffle-before-cap \
  --no-load-best-model-at-end \
  --no-merge-full-model

The effective endpoint saves were 900, 1,800, 2,700, and 3,598. max_steps, not epoch count, was the operative training contract.

9. Evaluation results

9.1 Closed-set lexical reconstruction

Accepted-reference scoring permits only forms explicitly attached to the governed prompt. It does not grant credit for an arbitrary dictionary word.

The final 28 failures were:

The machine taxonomy classified 25 failures as a known target for another prompt, two as near-orthographic, and one as other. That concentration suggests residual sense competition and lexical interference, not simple failure to produce Kuku Yalanji-shaped strings. The full row records include part of speech, semantic domain, token counts, closest reference, grapheme error, and checkpoint trajectory.

9.2 Why the older 297 benchmark reports 91/297

The same final checkpoint scored only 91/297 exact on the older historical benchmark. This is not a contradiction:

• only 208 of the 297 historical prompts overlap the governed 2,724-prompt census;
• accepted-reference surfaces differ across the two resources;
• 206/208 overlapping predictions were supported by the governed census;
• 89 prompts exist only in the historical resource;
• sense, source, spelling, variety, and curation differences remain unresolved.

Therefore the project reports each benchmark against its own frozen accepted-reference set. It does not merge them through automated string similarity and does not call either result population-level user-query reliability.

9.3 Sentence retention

All figures below are development diagnostics under one fixed neutral decoder. B0 is untouched v21.2.

Every selected-checkpoint hard retention check passed, including zero blank outputs and no increase in the tagged loop detector. This is encouraging retention evidence, not proof of natural competence:

• the experiment used one seed;
• model selection used development data;
• the 43 elder-shared rows have been repeatedly inspected;
• elder exact remains 0/43;
• elder mean token-length ratio is 0.5925 and 12/43 outputs are below half the reference token length;
• natural rows are not sufficiently speaker-, text-, genre-, or variety-diverse;
• automatic overlap cannot establish argument structure, case, TAM, polarity, discourse, or cultural suitability.

9.4 What the result does establish

It establishes that under this exact base, tokenizer, task prefix, data schedule, seed, optimizer path, and benchmark:

1. explicit repeated lexical supervision can drive near-complete closed-set reconstruction;
2. joint sentence replay prevented the broad sentence collapse seen in earlier lexical-heavy treatments;
3. the final checkpoint passes both the frozen lexical and development-retention gates;
4. residual lexical failures are concentrated in competing known targets and close forms;
5. further free-form claims require different evidence, not a larger number attached to the same closed set.

10. Product routing

Recommended current routing is:

known dictionary query
  -> deterministic dictionary lookup with source, sense, and variety metadata

research lexical reconstruction audit
  -> v24.3 task=lexeme

English sentence research preview
  -> v24.3 task=translate
  -> visible unverified-machine-draft label
  -> no high-stakes use

new or rare word in sentence
  -> retrieve a small relevant glossary
  -> future glossary-conditioned model, after separate evaluation

Every response or event log should preserve:

• source text;
• generated or retrieved target text;
• route (dictionary_exact, v24.3_lexeme, v24.3_translate, hosted fallback);
• immutable model version;
• adapter SHA-256;
• task prefix;
• decoder policy;
• latency and error state.

Raw user text may be sensitive. Set a short retention period, restrict access, provide deletion, and do not reuse logs for training without separate consent and governance.

11. Hosting multiple Aboriginal language adapters

LoRA makes a shared-base service feasible, but only under a strict compatibility contract.

11.1 What can be shared

One GPU process can keep a single encoder-decoder base resident and load, switch, or hotswap named adapters. This can reduce duplicate base memory and startup cost. PEFT supports multiple adapters and hotswapping for compatible LoRA configurations.

11.2 What must match

Adapters intended for one shared worker must agree on:

• exact base model weights and revision;
• model architecture and PEFT task type;
• target module set;
• compatible rank policy or a preallocated maximum rank;
• tokenizer files and token IDs;
• source and target language-code IDs;
• task-prefix token IDs;
• generation contract expected by the router.

Current MobTranslate language models do not automatically satisfy this. v24.3 is based on the exact v21.2 Kuku Yalanji continuation and adds <lexeme> at ID 256205. A Mi'gmaq adapter trained against another base or tokenizer cannot simply be attached to this process.

11.3 Recommended next-generation base contract

For future languages:

1. choose and checksum one shared NLLB base;
2. reserve all approved language codes and task tokens before any adapter training;
3. freeze and publish that tokenizer as a platform base version;
4. train one adapter per language first, using the same target modules and rank;
5. keep dictionary lookup outside the model;
6. evaluate every adapter independently on lexical, morphology, sentence, degeneration, and human-review gates;
7. load adapters on demand with a bounded LRU cache;
8. route by immutable (base_version, adapter_version, tokenizer_hash, decoder_policy);
9. do not let one language's benchmark authorize another language;
10. explore a language-family adapter only as a controlled transfer experiment, because shared training can help or cause cross-language interference.

The limiting work is not attaching another LoRA. It is governed data, language-specific morphology, independent natural evaluation, fluent-speaker review, rights, and authority over release.

11.4 Adapter-only storage is not automatically small

v24.3 demonstrates an operational issue: adding a trainable tokenizer row caused PEFT to save full embeddings. For a future shared platform, test PEFT trainable-token serialization and save_embedding_layers=False in an isolated parity study. Never apply that optimization retroactively without verifying every frozen prediction, because a small bundle that drops the learned task row is not the same model.

12. Reproduction and verification

12.1 Download and verify

curl --fail --location --continue-at - \
  --output kuku-yalanji-v24.3-complete-dynamic-bundle.tar.gz \
  https://mobtranslate.com/model-artifacts/kuku-yalanji-nllb-lora/v24.3-joint-lexeme-dose29-s3598-20260715/kuku-yalanji-v24.3-complete-dynamic-bundle.tar.gz

echo '59180518ba29865a7be54968f9710b723c726237ba441036c82cd9c6a3ffe920  kuku-yalanji-v24.3-complete-dynamic-bundle.tar.gz' \
  | sha256sum -c -

tar -xzf kuku-yalanji-v24.3-complete-dynamic-bundle.tar.gz
cd kuku-yalanji-v24.3
sha256sum base/model.safetensors adapter/adapter_model.safetensors

Expected internal identities:

7f9d0fe325e9e4568e45f13179adb336b93bbd53e83ddab2826e999eba3c76f7  base/model.safetensors
dd61583a60df2d538989e963e104cb626d78965d300e4e473e9a82ef59c04502  adapter/adapter_model.safetensors

12.2 Minimum acceptance checks for another host

1. Bundle hash matches.
2. Internal base, adapter, config, and tokenizer hashes match.
3. gvn_Latn resolves to ID 256204.
4. <lexeme> resolves to one special token at ID 256205.
5. Input and output embedding dimensions equal tokenizer length 256206 after resize.
6. Model and output projection dtypes agree.
7. Health remains unready until a real warm-up generation passes.
8. task=lexeme, text=woman returns exactly jalbu.
9. Both sentence smoke rows return the manifest outputs under the documented preview decoder.
10. Invalid tasks return 400, oversized input returns 413, and a saturated queue returns 429.
11. A representative frozen batch matches the archived dynamic-adapter predictions.
12. Host publishes the immutable version, route, research label, rights, and decoder policy.

12.3 Merging

Merging can reduce dynamic overhead, but it is a new serialized artifact. A merged release is acceptable only if:

• it starts from the exact base hash;
• the base is resized before PEFT load;
• tied encoder, decoder, shared, and output embeddings are canonicalized;
• model and tokenizer are saved together;
• adapter and merged outputs match on all 2,724 lexical rows and all 2,086 retention rows under both relevant decoder contracts;
• the merged model receives a new immutable SHA-256 and manifest entry.

Smoke-row agreement alone is insufficient.

13. Rights, governance, and public claims

The upstream NLLB checkpoint is released under CC BY-NC 4.0 and its model card describes it as a research model, not a production deployment. It is designed principally for single-sentence translation and carries domain and language-variety limitations.

This project also distinguishes:

• code licence;
• upstream model licence;
• dictionary source rights;
• synthetic-corpus release terms;
• speaker or narrative permissions;
• training permission;
• data redistribution permission;
• derived-weight redistribution permission;
• third-party host transfer permission;
• Indigenous Cultural and Intellectual Property and collective authority.

Public availability answers only some of those questions. It does not establish community authority. Current model outputs must be labelled as unverified drafts and must not be represented as speaker-certified, community-approved, or authoritative. Do not use them for health, legal, emergency, ceremonial, or culturally sensitive communication.

14. Known limitations and next research

The next useful work is not more epochs over the same closed-set dictionary. That objective is already near saturation. Priority evidence is:

1. speaker-reviewed sense decisions for the 28 residual lexical failures;
2. inference-time lexicon uptake, where a held-out mapping is supplied in a glossary and must be used in a novel sentence;
3. morphology tests that expose a lemma and hold out attested inflected or derived forms;
4. grammatical challenge sets covering participant roles, case, polarity, TAM, transitivity, clitics, subordination, reduplication, and discourse reference;
5. multiple independent speaker, text, genre, occasion, and variety clusters;
6. blinded fluent-speaker adequacy, grammaticality, naturalness, and critical-error review;
7. multi-seed confirmation of the selected recipe;
8. adapter repacking and merged-model parity work for lower hosting cost;
9. retrieval-conditioned translation that sends only relevant glossary entries rather than an entire dictionary.

15. Primary technical references

• NLLB-200 distilled 1.3B model card
• Fine-tuning NLLB-200 for a new language
• PEFT hotswap API
• Transformers PEFT integration
• PEFT trainable-token documentation
• Language-family adapters for low-resource NMT

16. Citation

MobTranslate. 2026. Kuku Yalanji v24.3 Model, Training, Evaluation, and Hosting Guide.
Release v24.3-joint-lexeme-dose29-s3598-20260715.
https://mobtranslate.com/docs/kuku-v24-3-model-guide.html

When reporting the lexical result, use this wording:

The v24.3 model reconstructed 2,696 of 2,724 accepted dictionary records exactly on a governed, training-overlapping closed-set census under the <lexeme> task. This is a reconstruction result, not unseen-word generalisation or sentence-translation accuracy.