Translate v2

internal proof

Model evaluation bench for the Kuku Yalanji training run. This page tracks the full v8-v20 diagnostic ladder, saved outputs, downloads, and resource profiles from each RunPod run.

Version lab Model registry

Where the model is after 48 hours

The training pipeline works. v20 trained the full candidate corpus and fixed the length-compression failure, but it also proved that scale alone is not enough. The hard problem is now product routing, mixture control, and faithful generalization, not GPU plumbing.

Saved evals only

Exact known text

lookup-first

Bible references, DB usage examples, and elder-shared sentence pairs should come from the approved source table, not generation.

Bible draft

v12.0

Best current Bible-draft fallback by heldout Bible chrF; still not faithful canonical reproduction.

Usage draft

v10.0

Best current heldout DB/general usage signal. Newer balanced runs have not beaten it.

Latest research

v20.0

Full-candidate-corpus diagnostic; length compression is fixed, but faithful generalization is still unresolved.

Latest run

Run: v20 full candidate corpus
Rows: 35,394 train / 4,433 validation / 4,397 test
Corpus shape: 20,911 source pairs expanded into tagged tasks
Training: 1h52m48s trainer time, 0.98 steps/s
RunPod cost: about $3.51

Model releaseDirection

Evaluation release

Live inference is intentionally off. Each release is reviewed through saved evaluation artifacts, exact-match counts, chrF/BLEU scores, resource usage, and downloadable model files.

Selected release: v19.0-balanced-replay-from-v12-gvn
Direction: eng-gvn
Saved eval rows: 452
Release role: Balanced replay artifact

Current eval summary

Kuku Yalanji · v19.0-balanced-replay-from-v12-gvn

BLEU: 13.1
chrF: 44.3
Rows: 452.0

This page shows reproducible eval artifacts only. Approved Bible text, database examples, and elder-shared sentence pairs should remain lookup-first in the product.

Dataset

v19.0 balanced Bible replay + DB usage + elder-shared sentence pairs

Training rows

5,900

GPU

NVIDIA RTX PRO 4500 Blackwell

Artifacts

model public · adapter public

Release verdict

Restores near-v12 Bible heldout while adding elder sentence-pair/usage signal, but does not replace v10 usage or v12 Bible routing.

Next move after v18: balanced replay continuation from v12, not another single-domain continuation.
Best broad heldout score after elder sentence-pair ingestion, but not a universal replacement.
Use for current research state; keep exact Bible/DB/elder sentence-pair resources lookup-first.

Downloads and artifacts

Public links resolve through the model-artifact file server. Merged models are large safetensor directories.

Merged modelmodel · directory LoRA adapteradapter · directory Release manifestmetadata · json Training manifestmetadata · json Post-eval analysismetadata · json Heldout Bible reference predictionsevaluation · json Heldout usage predictionsevaluation · json Elder-shared sentence-pair predictionsevaluation · json

Saved test outputs

Held-out examples from the saved evaluation artifacts for this model release.

Sources: ebible, Heldout usage predictions, Elder-shared sentence-pair predictions

BLEU: 13.1
chrF: 44.3
Rows: 452.0

SourceModel outputReference

ebible

MAT.4.18

<bible_ref> MAT.4.18 <eng> Walking by the sea of Galilee, he saw two brothers: Simon, who is called Peter, and Andrew, his brother, casting a net into the sea; for they were fishermen.

Jesus yikindarrda jalundurr Galileeburr dungan-dungan, nyulu bulanin jambul nyajin, bulal yaba-yabaju, Simon Peter, Andrew. Bula makarr yilbanya, minya kuyu maninka, sellim-bunganka.

Nyulu Jesus jalundurr Galileemundurr dungan-dungan, nyulu bulangan nyajin, yaba-yabaju. Nyubun burri Simon, nyungu burri yindu Peter. Yabaju burri Andrew. Bula makarrda kuyu manil-manin, yinya bulanga work.

ebible

REV.4.4

<bible_ref> REV.4.4 <eng> Around the throne were twenty-four thrones. On the thrones were twenty-four elders sitting, dressed in white garments, with crowns of gold on their heads.

Chair 24-bala yinyangkarr chairmundurr chairnga janan. Yala-yalabi chairnga bundandan elder-elder 24-bala, jananga kambi bingaji. Jananga dukulbu gold crownji.

Chair ngulkurr yindu-yindu 24-bala yinyangkarr chairmundurr. Elder-elder 24-bala bundandan yinyaymba chairnga 24-balanga. Jananga kambi bingaji, wandu crown gold-one dukulbu.

Heldout usage predictions

<glossary> junumundajin ; type=intransitive-verb ; domain=manipulation-impact <usage_example> The bananas are joined together

Banana junumundajin

Mayi banana junumundajin

Heldout usage predictions

<glossary> nyinki ; type=noun ; domain=body-part <usage_example> I saw him out of the corner of my eye

Ngayu nyungun nyinki nyajin nguwul-nguwulmun

Ngayu nyungun nyinkibu nyajin

Elder-shared sentence-pair predictions

<translate> When Bama first came to the rainforest country of Manjal Dimbi, it was new and strange. They did not know the landscape or what was good to eat or what was poisonous.

Yalayalaku Bama jakalba kadarin madjanga Manjal Dimbinga jana ngangkan kari bubuku, binal kari bubuku, binal kari wanyu mayi ngulkurr nukanka, wanyu mayi ngulkurr-kari nukanka.

Yalayalaku Bama jakalba kadarin madjanga Manjal Dimbinga jana ngangkan ngangkarin bubuku, binal kari bubuku, binal kari wanyu mayi ngulkurr nukanka, wanyu mayi ngulkurr-kari nukanka.

Elder-shared sentence-pair predictions

<translate> An old man holding his spear and woomera came to Bama. He had long grey hair and long fingernails. His body shone like lightning.

Nyubun mukul dingkar, kalka, milbayarr maranga kujil kujil, jananda kadan. Nyungu mungka kalbay, bingaji, nyungu biri maranga kalbay, nyungu bangkarr balbajin yala balbay balbajinya.

Metrics

train loss: 0.2129
validation bleu: 95.8
validation chrf: 97.2
heldout usage chrf: 53.0
heldout usage exact: 6.000
heldout bible direct chrf: 44.2
heldout bible ref chrf: 44.3
heldout all chrf: 44.5
heldout all plus elder sentence pair chrf: 47.5
elder sentence pair chrf: 93.9
elder sentence pair exact: 29.0
length ratio mean heldout all: 0.9760

Resource profile

Average GPU: 41.6%
Max GPU: 86%
Mean VRAM: 13,508 MiB
Max VRAM: 19,714 MiB
Max power: 184.89 W
Cost class: $0.74/hr
Estimated run cost: $0.68

Version ladder

Every material training branch from the last two days is listed here so the model line is auditable. chrF is useful for comparison, but exact approved known resources stay lookup-first.

23 releases

Version	Role	Bible ref	Usage	Elder rows	Decision	Links
v24.3-joint-lexeme-dose29-s3598-20260715 v24.3-joint-lexeme-dose29-s3598-20260715	Current runtime-verified Kuku Yalanji research candidate for closed-set lexical reconstruction and separately labelled sentence drafts runtime verified research candidate	n/a	53.17	0/43	The governed training-overlapping lexical gate and every frozen development retention check passed. ADVANCE authorizes further research, not unrestricted deployment: the model remains 0/43 exact on the elder diagnostic and lacks speaker-diverse human evaluation.	eval
v21.2-claude-balanced-replay-guarded-20260714 v21.2-claude-balanced-replay-guarded-20260714	Current Kuku Yalanji research model of record: frozen v21.2 weights with validated guarded decoding; CPU inference is intentionally unloaded research only	43.84	47.96	0/43	Retain the exact published v21.2 weights and guarded decoder as reproducible research artifacts, but do not load them for public CPU inference while natural elder-register transfer remains unsolved. The homepage uses separately labelled dictionary-context prompting until a stronger RunPod candidate clears the frozen linguistic and safety gates and receives appropriate speaker review.	model eval
v23.0-attested-narrative-adaptation-failed v23.0-attested-narrative-adaptation-failed	Rejected three-seed attested-narrative adaptation retained as compact negative evidence negative result	n/a	n/a	n/a	Do not promote. Seed 73 improved the held-out Text 3 corpus chrF++ point estimate by 0.6414, below the preregistered +1.0 floor; its paired interval crossed zero, untagged synthetic repetition worsened, and isolated lexical reconstruction fell to 46/297.	eval
v22.0-step-matched-replay-3120-failed v22.0-step-matched-replay-3120-failed	Rejected one-variable replay-exposure experiment retained as negative evidence negative result	n/a	n/a	n/a	Do not promote. Step 3,120 failed the frozen greedy checkpoint gate because tagged synthetic repeated-segment rows were 37 against a maximum of 25, and paired comparison showed lower tagged synthetic, untagged synthetic, and dictionary-usage agreement than the final v21.2 step 4,155 checkpoint.	eval
v21.2-claude-balanced-replay-v2-candidate v21.2-claude-balanced-replay-v2-candidate	Historical v21.2 weights release; superseded for serving by the guarded-policy release without changing weights research only	43.91	47.14	0/43	Balanced replay preserves synthetic performance while materially recovering dictionary-usage and Bible retention relative to v21.1. Natural elder-register transfer remains unsolved, and synthetic morpheme loops remain a hard safety failure; serve only as a clearly labelled research draft pending speaker review.	model eval
v21.1-codex-synthetic-direct-v2-candidate v21.1-codex-synthetic-direct-v2-candidate	Kuku Yalanji translation model v2 candidate research only	28.76	34.40	0/43	The model learned the synthetic treatment and remained stable without the task tag, but elder-shared, dictionary-usage, and Bible controls regressed severely. Preserve as a research artifact; keep retrieval-first routing and require balanced replay plus speaker review before promotion.
v20.0-full-candidate-corpus-gvn v20.0-full-candidate-corpus-gvn	Full-candidate-corpus diagnostic artifact internal proof	39.39	43.26	0/43	Full-corpus v20 fixed the length-compression failure and completed the 35k-row diagnostic, but it is not a faithful general model: Bible heldout stays near 39 chrF, heldout usage is 43.26 chrF, and elder-shared sentence pairs regressed to 0/43 exact.	model eval
v19.0-balanced-replay-from-v12-gvn v19.0-balanced-replay-from-v12-gvn	Balanced replay artifact internal proof	44.29	53.03	29/43	Restores near-v12 Bible heldout while adding elder sentence-pair/usage signal, but does not replace v10 usage or v12 Bible routing.	model eval
v18.0-usage-elder-sentence-continuation-from-v10 v18.0-usage-elder-sentence-continuation-from-v10	Elder sentence-pair memorization proof artifact internal proof	41.25	54.52	43/43	Exactly memorized all elder-shared sentence pairs, but regressed Bible and did not beat v10 on heldout usage.	model eval
v15.0-soft-lexical-hint-bible-gvn-token v15.0-soft-lexical-hint-bible-gvn-token	Lexical-hint diagnostic negative result	43.72	n/a	n/a	Reproduced train rows strongly, but did not beat v12 on heldout Bible.	model eval
v13.0-retrieval-context-bible-gvn-token v13.0-retrieval-context-bible-gvn-token	Retrieval-prefix diagnostic negative result	29.71	n/a	n/a	Retrieval context as an NLLB source prefix was harmful; heldout Bible dropped sharply.	model eval
v12.0-tagged-direct-plus-reference-bible-gvn-token v12.0-tagged-direct-plus-reference-bible-gvn-token	Current Bible draft fallback route candidate	44.36	n/a	n/a	Best Bible draft heldout score before v19, but still zero exact heldout reproduction.	model eval
v11.0-byt5-bible-control-32row v11.0-byt5-bible-control-32row	Byte-model control negative result	5.71	n/a	n/a	Failed the tiny overfit/control path; NLLB/LoRA remains the proven memorizing path for now.
v10.0-tagged-bible-plus-glossary-usage-tpi v10.0-tagged-bible-plus-glossary-usage-tpi	Current usage/general draft fallback route candidate	43.29	56.18	n/a	Best heldout usage signal so far; weaker Bible than v12/v19.	model eval
v9.8-tagged-bible-plus-db-usage-tpi v9.8-tagged-bible-plus-db-usage-tpi	First DB-usage multitask diagnostic internal proof	44.04	39.47	n/a	Memorized DB train examples but did not generalize well to word-id heldout usage.	model eval
v9.7-tagged-direct-plus-reference-bible-tpi v9.7-tagged-direct-plus-reference-bible-tpi	Reference-conditioning baseline internal proof	44.34	n/a	n/a	Established tagged Bible direct/reference training as viable, but still zero exact heldout reproduction.	model eval
v8.0-diagnostic-gates-summary v8.0-diagnostic-gates-summary	Pipeline proof gate internal proof	n/a	n/a	n/a	Proved tokenizer/LoRA/training/merge path could memorize; shifted problem from plumbing to data/task/generalization.
0.7.0-full-tpi-proxy-1.3b 0.7.0-full-tpi-proxy-1.3b	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.4.0-full-gvn-token 0.4.0-full-gvn-token	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.1.0-mini-pilot 0.1.0-mini-pilot	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-smoke 0.1.0-smoke	kuku_yalanji_ebible_parallel_smoke_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-baseline 0.1.0-baseline	kuku_yalanji_ebible_parallel_v0.1.0 training ready	n/a	n/a	n/a	No verdict recorded.
1.0.0-rc1 1.0.0-rc1	First leakage-controlled Mi'gmaq dictionary-example research candidate research only	n/a	n/a	n/a	Machine-integrity gates pass, but linguistic-quality promotion does not: frozen-test chrF++ is 21.43 with 0/742 exact matches and material lexical/content failures. Permit only a visibly warned research preview pending qualified Mi'gmaq review.	model eval