Translate v2

negative result

Model evaluation bench for the Kuku Yalanji training run. This page tracks the full v8-v20 diagnostic ladder, saved outputs, downloads, and resource profiles from each RunPod run.

Version lab Model registry

Where the model is after 48 hours

The training pipeline works. v20 trained the full candidate corpus and fixed the length-compression failure, but it also proved that scale alone is not enough. The hard problem is now product routing, mixture control, and faithful generalization, not GPU plumbing.

Saved evals only

Exact known text

lookup-first

Bible references, DB usage examples, and elder-shared sentence pairs should come from the approved source table, not generation.

Bible draft

v12.0

Best current Bible-draft fallback by heldout Bible chrF; still not faithful canonical reproduction.

Usage draft

v10.0

Best current heldout DB/general usage signal. Newer balanced runs have not beaten it.

Latest research

v20.0

Full-candidate-corpus diagnostic; length compression is fixed, but faithful generalization is still unresolved.

Latest run

Run: v20 full candidate corpus
Rows: 35,394 train / 4,433 validation / 4,397 test
Corpus shape: 20,911 source pairs expanded into tagged tasks
Training: 1h52m48s trainer time, 0.98 steps/s
RunPod cost: about $3.51

Model releaseDirection

Evaluation release

Live inference is intentionally off. Each release is reviewed through saved evaluation artifacts, exact-match counts, chrF/BLEU scores, resource usage, and downloadable model files.

Selected release: v13.0-retrieval-context-bible-gvn-token
Direction: eng-gvn
Saved eval rows: 325
Release role: Retrieval-prefix diagnostic

Current eval summary

Kuku Yalanji · v13.0-retrieval-context-bible-gvn-token

BLEU: 3.978
chrF: 29.7
Rows: 325.0

This page shows reproducible eval artifacts only. Approved Bible text, database examples, and elder-shared sentence pairs should remain lookup-first in the product.

Dataset

v13.0 retrieved Bible context prefix diagnostic

Training rows

512

GPU

NVIDIA RTX PRO 4500 Blackwell

Artifacts

model public · adapter public

Release verdict

Retrieval context as an NLLB source prefix was harmful; heldout Bible dropped sharply.

Retrieval remains useful for lookup/evidence display, not as this NLLB prefix format.

Downloads and artifacts

Public links resolve through the model-artifact file server. Merged models are large safetensor directories.

Merged modelmodel · directory LoRA adapteradapter · directory Training manifestmetadata · json Post-eval analysismetadata · json Heldout Bible reference predictionsevaluation · json

Saved test outputs

Held-out examples from the saved evaluation artifacts for this model release.

Sources: ebible

BLEU: 3.978
chrF: 29.7
Rows: 325.0

SourceModel outputReference

ebible

MAT.4.18

<retrieved_example> <eng> Now as he was walking by the sea of Galilee, he saw Simon, and Andrew his brother, casting a net into the sea: for they were fishers. <gvn> Jesus yikindarrda jalundurr Galileeburr dungan-dungan, nyulu bulanin yaba-yabaju Simon, Andrew nyajin, makarr yilbanya, minya kuyu maninka, sellim-bunganka. </retrieved_example> <bible_ref> MAT.4.18 <eng> Walking by the sea of Galilee, he saw two brothers: Simon, who is called Peter, and Andrew, his brother, casting a net into the sea; for they were fishermen.

Jesus warngku yirrkan jarramalingka jarramalinga. Nyulu bulanin yaba-yabaju Simon, Andrew, makarr yilbanya, minya kuyu maninka, sellim-bunganka.

Nyulu Jesus jalundurr Galileemundurr dungan-dungan, nyulu bulangan nyajin, yaba-yabaju. Nyubun burri Simon, nyungu burri yindu Peter. Yabaju burri Andrew. Bula makarrda kuyu manil-manin, yinya bulanga work.

ebible

REV.4.4

<retrieved_example> <eng> Therefore they are before the throne of God, and they serve him day and night in his temple. He who sits on the throne will spread his tabernacle over them. <gvn> Yinyaynka jana jakalba Godundumunbu chairnga janay, nyungunku wuljaljiku workmaniji nyunguwunbu bayanba, wujurrdurr, wungaraburr. God jananjikuda, jananin ngulkurrduku kujilkuda. </retrieved_example> <bible_ref> REV.4.4 <eng> Around the throne were twenty-four thrones. On the thrones were twenty-four elders sitting, dressed in white garments, with crowns of gold on their heads.

Godundumunbu chairnga janay, nyungunku wuljaljiku workmaniji. Jana nyunguwunbu bayanba, wujurrdurr, wungaraburr, nyungu crowns ngulkurrduku kujilkuda. Kingabakuda majaman bunal, majaman bunal, majaman bunal, majaman bunal, majaman bunal, majaman bunal.

Chair ngulkurr yindu-yindu 24-bala yinyangkarr chairmundurr. Elder-elder 24-bala bundandan yinyaymba chairnga 24-balanga. Jananga kambi bingaji, wandu crown gold-one dukulbu.

ebible

1CO.9.17

<retrieved_example> <eng> For if I am an offender, or have committed any thing worthy of death, I refuse not to die: but if there is none of these things of which these accuse me, no man may deliver me to them. I appeal to Cesar. <gvn> Kaki ngayu buyun-damanyaku, yalada, ngayu wulayarrku. Kaki jananga kuku ngaykunku manubaja kari, bamangka nganya murruji jananda mambarril townbu Jerusalem. Ngayu kinganda townbu Rome dunganka courtcasemunku.” </retrieved_example> <bible_ref> 1CO.9.17 <eng> For if I do this thing willingly, I have a reward: but if against my will, a dispensation of the gospel is committed to me.

Ngayu paypal kari. Kaki ngala ngalayamunku buyun-damanyaku, ngayu wawu kari warrida. Kaki Godku nganya wawurr-wawurrduku kujinka, yinya kuku ngaykunku. Godku nganya mission kari workmanijika.

Kaki ngayu wawubu Godumu kuku balkanka, ngayu money maninka yurrandamun. Yamba yala kari. Godungkuku nganya manin, nyungu work balkanka. Ngayu must nyungun kuku nyaka.

ebible

MRK.10.37

<retrieved_example> <eng> Even so, let your light shine before men, that they may see your good works and glorify your Father who is in heaven. <gvn> Yurra yala light. Yurra junkayku bunday, bamangka yurranin nyajinkaku junkayku bundarinyakuda. Yinyaynka jana God jiringa buyay-maninka.” </retrieved_example> <bible_ref> MRK.10.37 <eng> They said to him, “Grant to us that we may sit, one at your right hand and one at your left hand, in your glory.”

Jana nyungundu yalaman, Nyulu Godundumunku yunu bama yunu jawun-karra yunu jawun-karra yunu jawun-karra yunu jawun-karra yunu jawun-karra.

Bula nyungundu yalaman, “Yalada ngalin yununji bundanka? Yundu majamal, ngalin yununji bundanka, yinyamundu majamanka yununji.”

ebible

MAT.18.19

<retrieved_example> <eng> And he said to them, What man shall there be among you, that shall have one sheep, and if it shall fall into a pit on the sabbath, will he not lay hold on it, and lift it out? <gvn> Jesusangka yalaman, “Nguba yunu sheep nyubunku. Nyulu sheep nguba bulkajimunbu daray Saturdaymunbu. Yundu wanjarrmalda? Yundu sheep karrbal, bulkajimun jarra-kulbal. </retrieved_example> <bible_ref> MAT.18.19 <eng> Again I say to you, That if two of you shall agree on earth, concerning any thing that they shall ask, it shall be done for them by my Father who is in heaven.

Ngayu wuljaljiku balkawan. Kaki yundu bulu-duray baja heavenmunbu ngaykundu balkan, Godungku yunu bama ngulkurrduku kujil. Godungku bama junkurrji bundanday, junkurrji bundanday.

“Ngayu yurranda balkal baja, kaki bama bula jambul junjuynku wawumal and bula jambulku Godundu babaji, Nganjanangka Godungku dajilkuda.

ebible

LUK.11.27

<retrieved_example> <eng> And I said to him, Sir, thou knowest. And he said to me, These are they who came out of great tribulation, and have washed their robes, and made them white in the blood of the Lamb. <gvn> Ngayu yalaman, “Ngayu binal kari. Wanjamun baja? Yunduku binal, wulman.” Nyulu ngaykundu yalaman, “Jana kulimun jirraymun kadarin. Jana jananga kambi julurrin mulabu sheepandamundu karrkayandamundu, bingaji-bungan. </retrieved_example> <bible_ref> LUK.11.27 <eng> It came to pass, as he said these things, a certain woman out of the multitude lifted up her voice and said to him, “Blessed is the womb that bore you, and the breasts which nursed you!”

Jesusangka yalaman baja, Ngayu jawun-karra wubulku binal kari. Wanjundumundu wawurr-wawurrduku balkan, Jana, Godungku junkurrji, yamba yundu ngulkurrduku kujil, yamba yundu ngulkurrduku kujil.

Jesusangka yinya kuku balkan, nyubunyinja jalbungku yalaman, “Yunu ngamu wawurr-wawurrmaniji, yundu nyunguku kangkal.”

Metrics

heldout bible direct chrf: 29.5
heldout bible ref chrf: 29.7

Resource profile

Average GPU: n/a
Max GPU: n/a
Mean VRAM: n/a
Max VRAM: n/a
Max power: n/a
Cost class: n/a
Estimated run cost: n/a

Version ladder

Every material training branch from the last two days is listed here so the model line is auditable. chrF is useful for comparison, but exact approved known resources stay lookup-first.

23 releases

Version	Role	Bible ref	Usage	Elder rows	Decision	Links
v24.3-joint-lexeme-dose29-s3598-20260715 v24.3-joint-lexeme-dose29-s3598-20260715	Current runtime-verified Kuku Yalanji research candidate for closed-set lexical reconstruction and separately labelled sentence drafts runtime verified research candidate	n/a	53.17	0/43	The governed training-overlapping lexical gate and every frozen development retention check passed. ADVANCE authorizes further research, not unrestricted deployment: the model remains 0/43 exact on the elder diagnostic and lacks speaker-diverse human evaluation.	eval
v21.2-claude-balanced-replay-guarded-20260714 v21.2-claude-balanced-replay-guarded-20260714	Current Kuku Yalanji research model of record: frozen v21.2 weights with validated guarded decoding; CPU inference is intentionally unloaded research only	43.84	47.96	0/43	Retain the exact published v21.2 weights and guarded decoder as reproducible research artifacts, but do not load them for public CPU inference while natural elder-register transfer remains unsolved. The homepage uses separately labelled dictionary-context prompting until a stronger RunPod candidate clears the frozen linguistic and safety gates and receives appropriate speaker review.	model eval
v23.0-attested-narrative-adaptation-failed v23.0-attested-narrative-adaptation-failed	Rejected three-seed attested-narrative adaptation retained as compact negative evidence negative result	n/a	n/a	n/a	Do not promote. Seed 73 improved the held-out Text 3 corpus chrF++ point estimate by 0.6414, below the preregistered +1.0 floor; its paired interval crossed zero, untagged synthetic repetition worsened, and isolated lexical reconstruction fell to 46/297.	eval
v22.0-step-matched-replay-3120-failed v22.0-step-matched-replay-3120-failed	Rejected one-variable replay-exposure experiment retained as negative evidence negative result	n/a	n/a	n/a	Do not promote. Step 3,120 failed the frozen greedy checkpoint gate because tagged synthetic repeated-segment rows were 37 against a maximum of 25, and paired comparison showed lower tagged synthetic, untagged synthetic, and dictionary-usage agreement than the final v21.2 step 4,155 checkpoint.	eval
v21.2-claude-balanced-replay-v2-candidate v21.2-claude-balanced-replay-v2-candidate	Historical v21.2 weights release; superseded for serving by the guarded-policy release without changing weights research only	43.91	47.14	0/43	Balanced replay preserves synthetic performance while materially recovering dictionary-usage and Bible retention relative to v21.1. Natural elder-register transfer remains unsolved, and synthetic morpheme loops remain a hard safety failure; serve only as a clearly labelled research draft pending speaker review.	model eval
v21.1-codex-synthetic-direct-v2-candidate v21.1-codex-synthetic-direct-v2-candidate	Kuku Yalanji translation model v2 candidate research only	28.76	34.40	0/43	The model learned the synthetic treatment and remained stable without the task tag, but elder-shared, dictionary-usage, and Bible controls regressed severely. Preserve as a research artifact; keep retrieval-first routing and require balanced replay plus speaker review before promotion.
v20.0-full-candidate-corpus-gvn v20.0-full-candidate-corpus-gvn	Full-candidate-corpus diagnostic artifact internal proof	39.39	43.26	0/43	Full-corpus v20 fixed the length-compression failure and completed the 35k-row diagnostic, but it is not a faithful general model: Bible heldout stays near 39 chrF, heldout usage is 43.26 chrF, and elder-shared sentence pairs regressed to 0/43 exact.	model eval
v19.0-balanced-replay-from-v12-gvn v19.0-balanced-replay-from-v12-gvn	Balanced replay artifact internal proof	44.29	53.03	29/43	Restores near-v12 Bible heldout while adding elder sentence-pair/usage signal, but does not replace v10 usage or v12 Bible routing.	model eval
v18.0-usage-elder-sentence-continuation-from-v10 v18.0-usage-elder-sentence-continuation-from-v10	Elder sentence-pair memorization proof artifact internal proof	41.25	54.52	43/43	Exactly memorized all elder-shared sentence pairs, but regressed Bible and did not beat v10 on heldout usage.	model eval
v15.0-soft-lexical-hint-bible-gvn-token v15.0-soft-lexical-hint-bible-gvn-token	Lexical-hint diagnostic negative result	43.72	n/a	n/a	Reproduced train rows strongly, but did not beat v12 on heldout Bible.	model eval
v13.0-retrieval-context-bible-gvn-token v13.0-retrieval-context-bible-gvn-token	Retrieval-prefix diagnostic negative result	29.71	n/a	n/a	Retrieval context as an NLLB source prefix was harmful; heldout Bible dropped sharply.	model eval
v12.0-tagged-direct-plus-reference-bible-gvn-token v12.0-tagged-direct-plus-reference-bible-gvn-token	Current Bible draft fallback route candidate	44.36	n/a	n/a	Best Bible draft heldout score before v19, but still zero exact heldout reproduction.	model eval
v11.0-byt5-bible-control-32row v11.0-byt5-bible-control-32row	Byte-model control negative result	5.71	n/a	n/a	Failed the tiny overfit/control path; NLLB/LoRA remains the proven memorizing path for now.
v10.0-tagged-bible-plus-glossary-usage-tpi v10.0-tagged-bible-plus-glossary-usage-tpi	Current usage/general draft fallback route candidate	43.29	56.18	n/a	Best heldout usage signal so far; weaker Bible than v12/v19.	model eval
v9.8-tagged-bible-plus-db-usage-tpi v9.8-tagged-bible-plus-db-usage-tpi	First DB-usage multitask diagnostic internal proof	44.04	39.47	n/a	Memorized DB train examples but did not generalize well to word-id heldout usage.	model eval
v9.7-tagged-direct-plus-reference-bible-tpi v9.7-tagged-direct-plus-reference-bible-tpi	Reference-conditioning baseline internal proof	44.34	n/a	n/a	Established tagged Bible direct/reference training as viable, but still zero exact heldout reproduction.	model eval
v8.0-diagnostic-gates-summary v8.0-diagnostic-gates-summary	Pipeline proof gate internal proof	n/a	n/a	n/a	Proved tokenizer/LoRA/training/merge path could memorize; shifted problem from plumbing to data/task/generalization.
0.7.0-full-tpi-proxy-1.3b 0.7.0-full-tpi-proxy-1.3b	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.4.0-full-gvn-token 0.4.0-full-gvn-token	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.1.0-mini-pilot 0.1.0-mini-pilot	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-smoke 0.1.0-smoke	kuku_yalanji_ebible_parallel_smoke_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-baseline 0.1.0-baseline	kuku_yalanji_ebible_parallel_v0.1.0 training ready	n/a	n/a	n/a	No verdict recorded.
1.0.0-rc1 1.0.0-rc1	First leakage-controlled Mi'gmaq dictionary-example research candidate research only	n/a	n/a	n/a	Machine-integrity gates pass, but linguistic-quality promotion does not: frozen-test chrF++ is 21.43 with 0/742 exact matches and material lexical/content failures. Permit only a visibly warned research preview pending qualified Mi'gmaq review.	model eval