Translate v2

negative result

Model evaluation bench for the Kuku Yalanji training run. This page tracks the full v8-v20 diagnostic ladder, saved outputs, downloads, and resource profiles from each RunPod run.

Version lab Model registry

Where the model is after 48 hours

The training pipeline works. v20 trained the full candidate corpus and fixed the length-compression failure, but it also proved that scale alone is not enough. The hard problem is now product routing, mixture control, and faithful generalization, not GPU plumbing.

Saved evals only

Exact known text

lookup-first

Bible references, DB usage examples, and elder-shared sentence pairs should come from the approved source table, not generation.

Bible draft

v12.0

Best current Bible-draft fallback by heldout Bible chrF; still not faithful canonical reproduction.

Usage draft

v10.0

Best current heldout DB/general usage signal. Newer balanced runs have not beaten it.

Latest research

v20.0

Full-candidate-corpus diagnostic; length compression is fixed, but faithful generalization is still unresolved.

Latest run

Run: v20 full candidate corpus
Rows: 35,394 train / 4,433 validation / 4,397 test
Corpus shape: 20,911 source pairs expanded into tagged tasks
Training: 1h52m48s trainer time, 0.98 steps/s
RunPod cost: about $3.51

Model releaseDirection

Evaluation release

Live inference is intentionally off. Each release is reviewed through saved evaluation artifacts, exact-match counts, chrF/BLEU scores, resource usage, and downloadable model files.

Selected release: v15.0-soft-lexical-hint-bible-gvn-token
Direction: eng-gvn
Saved eval rows: 325
Release role: Lexical-hint diagnostic

Current eval summary

Kuku Yalanji · v15.0-soft-lexical-hint-bible-gvn-token

BLEU: 12.5
chrF: 43.7
Rows: 325.0

This page shows reproducible eval artifacts only. Approved Bible text, database examples, and elder-shared sentence pairs should remain lookup-first in the product.

Dataset

v15.0 oracle lexical-hint Bible diagnostic

Training rows

2,048

GPU

NVIDIA RTX PRO 4500 Blackwell

Artifacts

model public · adapter public

Release verdict

Reproduced train rows strongly, but did not beat v12 on heldout Bible.

Soft lexical hints were not the missing ingredient; do not scale this branch.

Downloads and artifacts

Public links resolve through the model-artifact file server. Merged models are large safetensor directories.

Merged modelmodel · directory LoRA adapteradapter · directory Training manifestmetadata · json Post-eval analysismetadata · json Heldout Bible reference predictionsevaluation · json

Saved test outputs

Held-out examples from the saved evaluation artifacts for this model release.

Sources: ebible

BLEU: 12.5
chrF: 43.7
Rows: 325.0

SourceModel outputReference

ebible

MAT.4.18

<lexical_hints> makarrda ; manil-manin ; Galileemundurr </lexical_hints> <bible_ref> MAT.4.18 <eng> Walking by the sea of Galilee, he saw two brothers: Simon, who is called Peter, and Andrew, his brother, casting a net into the sea; for they were fishermen.

Jesus yikindarrda jalundurr Galileeburr dungan-dungan, nyulu bulanin jambul nyajin, bulal yaba-yabaju, Simon Peter, nyulu yindu Andrew. Bula makarr yilban bananga, minya kuyu maninka, sellim-bunganka.

Nyulu Jesus jalundurr Galileemundurr dungan-dungan, nyulu bulangan nyajin, yaba-yabaju. Nyubun burri Simon, nyungu burri yindu Peter. Yabaju burri Andrew. Bula makarrda kuyu manil-manin, yinya bulanga work.

ebible

REV.4.4

<lexical_hints> 24-balanga ; wandu ; Chair </lexical_hints> <bible_ref> REV.4.4 <eng> Around the throne were twenty-four thrones. On the thrones were twenty-four elders sitting, dressed in white garments, with crowns of gold on their heads.

Chair 24-bala yinyangkarr chairmundurr chairmundurr. Jana chairnga bundandan, elder-elder 24-bala, jananga kambi bingaji. Jananga dukulbu crown gold-one.

Chair ngulkurr yindu-yindu 24-bala yinyangkarr chairmundurr. Elder-elder 24-bala bundandan yinyaymba chairnga 24-balanga. Jananga kambi bingaji, wandu crown gold-one dukulbu.

ebible

1CO.9.17

<lexical_hints> yurrandamun ; Godungkuku ; must </lexical_hints> <bible_ref> 1CO.9.17 <eng> For if I do this thing willingly, I have a reward: but if against my will, a dispensation of the gospel is committed to me.

Kaki ngayu ngayku work ngulkurr balkal, Godungku nganya payim-bungal. Kari. Kaki ngayu work ngulkurr balkal, Godungku nganya yungal.

Kaki ngayu wawubu Godumu kuku balkanka, ngayu money maninka yurrandamun. Yamba yala kari. Godungkuku nganya manin, nyungu work balkanka. Ngayu must nyungun kuku nyaka.

ebible

MRK.10.37

<lexical_hints> majamal ; yununji ; majamanka </lexical_hints> <bible_ref> MRK.10.37 <eng> They said to him, “Grant to us that we may sit, one at your right hand and one at your left hand, in your glory.”

Bula yalaman, Yundu Godunji bunday, yundu maja-bunganka. Yundu nganjinin yaba-yabaju bunday, yubalanga maja jirray, yubalanga najil.

Bula nyungundu yalaman, “Yalada ngalin yununji bundanka? Yundu majamal, ngalin yununji bundanka, yinyamundu majamanka yununji.”

ebible

MAT.18.19

<lexical_hints> junjuynku ; wawumal ; jambulku </lexical_hints> <bible_ref> MAT.18.19 <eng> Again I say to you, That if two of you shall agree on earth, concerning any thing that they shall ask, it shall be done for them by my Father who is in heaven.

Jesusangka yalaman baja, Kaki yurra jambul bubungu agreemanijinka, yurra Godundu babajinka, ngaykuwundu nganjanangka heavenmun nyungundu balkal.

“Ngayu yurranda balkal baja, kaki bama bula jambul junjuynku wawumal and bula jambulku Godundu babaji, Nganjanangka Godungku dajilkuda.

ebible

LUK.11.27

<lexical_hints> nyunguku ; wawurr-wawurrmaniji ; jalbungku </lexical_hints> <bible_ref> LUK.11.27 <eng> It came to pass, as he said these things, a certain woman out of the multitude lifted up her voice and said to him, “Blessed is the womb that bore you, and the breasts which nursed you!”

Jesusangka yinya balkal-balkawan, nyubunyinja jalbungku bamandamun yaykarrku yalaman, Yinya jalbu wawurr-wawurrmal yunu banaka. Yundu banaka ngawa-ngawa balkajinkuda, yala banyuwundu ngamu.

Jesusangka yinya kuku balkan, nyubunyinja jalbungku yalaman, “Yunu ngamu wawurr-wawurrmaniji, yundu nyunguku kangkal.”

Metrics

heldout bible direct chrf: 44.1
heldout bible ref chrf: 43.7

Resource profile

Average GPU: n/a
Max GPU: n/a
Mean VRAM: n/a
Max VRAM: n/a
Max power: n/a
Cost class: n/a
Estimated run cost: n/a

Version ladder

Every material training branch from the last two days is listed here so the model line is auditable. chrF is useful for comparison, but exact approved known resources stay lookup-first.

23 releases

Version	Role	Bible ref	Usage	Elder rows	Decision	Links
v24.3-joint-lexeme-dose29-s3598-20260715 v24.3-joint-lexeme-dose29-s3598-20260715	Current runtime-verified Kuku Yalanji research candidate for closed-set lexical reconstruction and separately labelled sentence drafts runtime verified research candidate	n/a	53.17	0/43	The governed training-overlapping lexical gate and every frozen development retention check passed. ADVANCE authorizes further research, not unrestricted deployment: the model remains 0/43 exact on the elder diagnostic and lacks speaker-diverse human evaluation.	eval
v21.2-claude-balanced-replay-guarded-20260714 v21.2-claude-balanced-replay-guarded-20260714	Current Kuku Yalanji research model of record: frozen v21.2 weights with validated guarded decoding; CPU inference is intentionally unloaded research only	43.84	47.96	0/43	Retain the exact published v21.2 weights and guarded decoder as reproducible research artifacts, but do not load them for public CPU inference while natural elder-register transfer remains unsolved. The homepage uses separately labelled dictionary-context prompting until a stronger RunPod candidate clears the frozen linguistic and safety gates and receives appropriate speaker review.	model eval
v23.0-attested-narrative-adaptation-failed v23.0-attested-narrative-adaptation-failed	Rejected three-seed attested-narrative adaptation retained as compact negative evidence negative result	n/a	n/a	n/a	Do not promote. Seed 73 improved the held-out Text 3 corpus chrF++ point estimate by 0.6414, below the preregistered +1.0 floor; its paired interval crossed zero, untagged synthetic repetition worsened, and isolated lexical reconstruction fell to 46/297.	eval
v22.0-step-matched-replay-3120-failed v22.0-step-matched-replay-3120-failed	Rejected one-variable replay-exposure experiment retained as negative evidence negative result	n/a	n/a	n/a	Do not promote. Step 3,120 failed the frozen greedy checkpoint gate because tagged synthetic repeated-segment rows were 37 against a maximum of 25, and paired comparison showed lower tagged synthetic, untagged synthetic, and dictionary-usage agreement than the final v21.2 step 4,155 checkpoint.	eval
v21.2-claude-balanced-replay-v2-candidate v21.2-claude-balanced-replay-v2-candidate	Historical v21.2 weights release; superseded for serving by the guarded-policy release without changing weights research only	43.91	47.14	0/43	Balanced replay preserves synthetic performance while materially recovering dictionary-usage and Bible retention relative to v21.1. Natural elder-register transfer remains unsolved, and synthetic morpheme loops remain a hard safety failure; serve only as a clearly labelled research draft pending speaker review.	model eval
v21.1-codex-synthetic-direct-v2-candidate v21.1-codex-synthetic-direct-v2-candidate	Kuku Yalanji translation model v2 candidate research only	28.76	34.40	0/43	The model learned the synthetic treatment and remained stable without the task tag, but elder-shared, dictionary-usage, and Bible controls regressed severely. Preserve as a research artifact; keep retrieval-first routing and require balanced replay plus speaker review before promotion.
v20.0-full-candidate-corpus-gvn v20.0-full-candidate-corpus-gvn	Full-candidate-corpus diagnostic artifact internal proof	39.39	43.26	0/43	Full-corpus v20 fixed the length-compression failure and completed the 35k-row diagnostic, but it is not a faithful general model: Bible heldout stays near 39 chrF, heldout usage is 43.26 chrF, and elder-shared sentence pairs regressed to 0/43 exact.	model eval
v19.0-balanced-replay-from-v12-gvn v19.0-balanced-replay-from-v12-gvn	Balanced replay artifact internal proof	44.29	53.03	29/43	Restores near-v12 Bible heldout while adding elder sentence-pair/usage signal, but does not replace v10 usage or v12 Bible routing.	model eval
v18.0-usage-elder-sentence-continuation-from-v10 v18.0-usage-elder-sentence-continuation-from-v10	Elder sentence-pair memorization proof artifact internal proof	41.25	54.52	43/43	Exactly memorized all elder-shared sentence pairs, but regressed Bible and did not beat v10 on heldout usage.	model eval
v15.0-soft-lexical-hint-bible-gvn-token v15.0-soft-lexical-hint-bible-gvn-token	Lexical-hint diagnostic negative result	43.72	n/a	n/a	Reproduced train rows strongly, but did not beat v12 on heldout Bible.	model eval
v13.0-retrieval-context-bible-gvn-token v13.0-retrieval-context-bible-gvn-token	Retrieval-prefix diagnostic negative result	29.71	n/a	n/a	Retrieval context as an NLLB source prefix was harmful; heldout Bible dropped sharply.	model eval
v12.0-tagged-direct-plus-reference-bible-gvn-token v12.0-tagged-direct-plus-reference-bible-gvn-token	Current Bible draft fallback route candidate	44.36	n/a	n/a	Best Bible draft heldout score before v19, but still zero exact heldout reproduction.	model eval
v11.0-byt5-bible-control-32row v11.0-byt5-bible-control-32row	Byte-model control negative result	5.71	n/a	n/a	Failed the tiny overfit/control path; NLLB/LoRA remains the proven memorizing path for now.
v10.0-tagged-bible-plus-glossary-usage-tpi v10.0-tagged-bible-plus-glossary-usage-tpi	Current usage/general draft fallback route candidate	43.29	56.18	n/a	Best heldout usage signal so far; weaker Bible than v12/v19.	model eval
v9.8-tagged-bible-plus-db-usage-tpi v9.8-tagged-bible-plus-db-usage-tpi	First DB-usage multitask diagnostic internal proof	44.04	39.47	n/a	Memorized DB train examples but did not generalize well to word-id heldout usage.	model eval
v9.7-tagged-direct-plus-reference-bible-tpi v9.7-tagged-direct-plus-reference-bible-tpi	Reference-conditioning baseline internal proof	44.34	n/a	n/a	Established tagged Bible direct/reference training as viable, but still zero exact heldout reproduction.	model eval
v8.0-diagnostic-gates-summary v8.0-diagnostic-gates-summary	Pipeline proof gate internal proof	n/a	n/a	n/a	Proved tokenizer/LoRA/training/merge path could memorize; shifted problem from plumbing to data/task/generalization.
0.7.0-full-tpi-proxy-1.3b 0.7.0-full-tpi-proxy-1.3b	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.4.0-full-gvn-token 0.4.0-full-gvn-token	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.1.0-mini-pilot 0.1.0-mini-pilot	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-smoke 0.1.0-smoke	kuku_yalanji_ebible_parallel_smoke_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-baseline 0.1.0-baseline	kuku_yalanji_ebible_parallel_v0.1.0 training ready	n/a	n/a	n/a	No verdict recorded.
1.0.0-rc1 1.0.0-rc1	First leakage-controlled Mi'gmaq dictionary-example research candidate research only	n/a	n/a	n/a	Machine-integrity gates pass, but linguistic-quality promotion does not: frozen-test chrF++ is 21.43 with 0/742 exact matches and material lexical/content failures. Permit only a visibly warned research preview pending qualified Mi'gmaq review.	model eval