Translate v2

research only

Model evaluation bench for the Kuku Yalanji training run. This page tracks the full v8-v20 diagnostic ladder, saved outputs, downloads, and resource profiles from each RunPod run.

Version lab Model registry

Where the model is after 48 hours

The training pipeline works. v20 trained the full candidate corpus and fixed the length-compression failure, but it also proved that scale alone is not enough. The hard problem is now product routing, mixture control, and faithful generalization, not GPU plumbing.

Saved evals only

Exact known text

lookup-first

Bible references, DB usage examples, and elder-shared sentence pairs should come from the approved source table, not generation.

Bible draft

v12.0

Best current Bible-draft fallback by heldout Bible chrF; still not faithful canonical reproduction.

Usage draft

v10.0

Best current heldout DB/general usage signal. Newer balanced runs have not beaten it.

Latest research

v20.0

Full-candidate-corpus diagnostic; length compression is fixed, but faithful generalization is still unresolved.

Latest run

Run: v20 full candidate corpus
Rows: 35,394 train / 4,433 validation / 4,397 test
Corpus shape: 20,911 source pairs expanded into tagged tasks
Training: 1h52m48s trainer time, 0.98 steps/s
RunPod cost: about $3.51

Model releaseDirection

Evaluation release

Live inference is intentionally off. Each release is reviewed through saved evaluation artifacts, exact-match counts, chrF/BLEU scores, resource usage, and downloadable model files.

Selected release: v21.2-claude-balanced-replay-v2-candidate
Direction: eng-gvn
Saved eval rows: see artifacts
Release role: Historical v21.2 weights release; superseded for serving by the guarded-policy release without changing weights

Current eval summary

Kuku Yalanji · v21.2-claude-balanced-replay-v2-candidate

No compact saved-output preview is attached to this release yet. Use the artifacts and metrics sections below.

This page shows reproducible eval artifacts only. Approved Bible text, database examples, and elder-shared sentence pairs should remain lookup-first in the product.

Dataset

22,164-row balanced treatment per epoch: 16,642 leakage-audited synthetic rows, 2,047 Bible-direct replay rows, 2,047 Bible-reference replay rows, and 1,428 dictionary-usage replay rows; elder rows excluded

Training rows

22,164

GPU

NVIDIA A40

Artifacts

model public · adapter local

Release verdict

Balanced replay preserves synthetic performance while materially recovering dictionary-usage and Bible retention relative to v21.1. Natural elder-register transfer remains unsolved, and synthetic morpheme loops remain a hard safety failure; serve only as a clearly labelled research draft pending speaker review.

The independent comparison revalidated ordered IDs, inputs, references, decoder settings, and archive checksums before scoring.
Compared with v21.1, corpus chrF changed by -0.87 on tagged synthetic test, +12.74 on dictionary usage, +15.23 on Bible direct, and +15.15 on Bible reference.
The 43-row elder control remains 0 exact with severe under-translation; its paired confidence interval versus v21.1 includes zero.
Thirty-seven tagged synthetic rows contain a segment repeated at least ten times, so generation guards and a frozen-battery rerun are mandatory before broader routing.
The original release record preserves its greedy evaluation; current serving uses the separately validated guarded-policy release and remains visibly labelled as a research draft.

Downloads and artifacts

Public links resolve through the model-artifact file server. Merged models are large safetensor directories.

Comprehensive model, training, and hosting guidedocumentation · html Atlas dynamic LoRA hosting bundlebundle · tar.gz Standalone merged-model bundlemodel · tar.gz Hosting manifestmetadata · json Release checksumsmetadata · sha256 Exact v21.2 training splitsdataset · tar.gz Complete synthetic process corpus v2dataset · zip Independent v21 comparisonevaluation · html

Metrics

train loss: 1.199
synthetic dev bleu: 26.2
synthetic dev chrf: 52.8
synthetic test tagged bleu: 26.8
synthetic test tagged chrf: 53.8
synthetic test tagged exact: 426.0
synthetic test untagged bleu: 26.8
synthetic test untagged chrf: 53.8
synthetic test untagged exact: 429.0
elder shared bleu: 6.521
elder shared chrf: 29.4
elder shared exact: 0.0000
heldout usage bleu: 11.6
heldout usage chrf: 47.1
heldout usage exact: 3.000
bible direct bleu: 12.4
bible direct chrf: 43.8
bible ref bleu: 12.4
bible ref chrf: 43.9
empty outputs all sets: 0.0000
synthetic tagged segment loop rows: 37.0

Resource profile

Average GPU: n/a
Max GPU: n/a
Mean VRAM: n/a
Max VRAM: 30,383 MiB
Max power: n/a
Cost class: n/a
Estimated run cost: n/a

Version ladder

Every material training branch from the last two days is listed here so the model line is auditable. chrF is useful for comparison, but exact approved known resources stay lookup-first.

23 releases

Version	Role	Bible ref	Usage	Elder rows	Decision	Links
v24.3-joint-lexeme-dose29-s3598-20260715 v24.3-joint-lexeme-dose29-s3598-20260715	Current runtime-verified Kuku Yalanji research candidate for closed-set lexical reconstruction and separately labelled sentence drafts runtime verified research candidate	n/a	53.17	0/43	The governed training-overlapping lexical gate and every frozen development retention check passed. ADVANCE authorizes further research, not unrestricted deployment: the model remains 0/43 exact on the elder diagnostic and lacks speaker-diverse human evaluation.	eval
v21.2-claude-balanced-replay-guarded-20260714 v21.2-claude-balanced-replay-guarded-20260714	Current Kuku Yalanji research model of record: frozen v21.2 weights with validated guarded decoding; CPU inference is intentionally unloaded research only	43.84	47.96	0/43	Retain the exact published v21.2 weights and guarded decoder as reproducible research artifacts, but do not load them for public CPU inference while natural elder-register transfer remains unsolved. The homepage uses separately labelled dictionary-context prompting until a stronger RunPod candidate clears the frozen linguistic and safety gates and receives appropriate speaker review.	model eval
v23.0-attested-narrative-adaptation-failed v23.0-attested-narrative-adaptation-failed	Rejected three-seed attested-narrative adaptation retained as compact negative evidence negative result	n/a	n/a	n/a	Do not promote. Seed 73 improved the held-out Text 3 corpus chrF++ point estimate by 0.6414, below the preregistered +1.0 floor; its paired interval crossed zero, untagged synthetic repetition worsened, and isolated lexical reconstruction fell to 46/297.	eval
v22.0-step-matched-replay-3120-failed v22.0-step-matched-replay-3120-failed	Rejected one-variable replay-exposure experiment retained as negative evidence negative result	n/a	n/a	n/a	Do not promote. Step 3,120 failed the frozen greedy checkpoint gate because tagged synthetic repeated-segment rows were 37 against a maximum of 25, and paired comparison showed lower tagged synthetic, untagged synthetic, and dictionary-usage agreement than the final v21.2 step 4,155 checkpoint.	eval
v21.2-claude-balanced-replay-v2-candidate v21.2-claude-balanced-replay-v2-candidate	Historical v21.2 weights release; superseded for serving by the guarded-policy release without changing weights research only	43.91	47.14	0/43	Balanced replay preserves synthetic performance while materially recovering dictionary-usage and Bible retention relative to v21.1. Natural elder-register transfer remains unsolved, and synthetic morpheme loops remain a hard safety failure; serve only as a clearly labelled research draft pending speaker review.	model eval
v21.1-codex-synthetic-direct-v2-candidate v21.1-codex-synthetic-direct-v2-candidate	Kuku Yalanji translation model v2 candidate research only	28.76	34.40	0/43	The model learned the synthetic treatment and remained stable without the task tag, but elder-shared, dictionary-usage, and Bible controls regressed severely. Preserve as a research artifact; keep retrieval-first routing and require balanced replay plus speaker review before promotion.
v20.0-full-candidate-corpus-gvn v20.0-full-candidate-corpus-gvn	Full-candidate-corpus diagnostic artifact internal proof	39.39	43.26	0/43	Full-corpus v20 fixed the length-compression failure and completed the 35k-row diagnostic, but it is not a faithful general model: Bible heldout stays near 39 chrF, heldout usage is 43.26 chrF, and elder-shared sentence pairs regressed to 0/43 exact.	model eval
v19.0-balanced-replay-from-v12-gvn v19.0-balanced-replay-from-v12-gvn	Balanced replay artifact internal proof	44.29	53.03	29/43	Restores near-v12 Bible heldout while adding elder sentence-pair/usage signal, but does not replace v10 usage or v12 Bible routing.	model eval
v18.0-usage-elder-sentence-continuation-from-v10 v18.0-usage-elder-sentence-continuation-from-v10	Elder sentence-pair memorization proof artifact internal proof	41.25	54.52	43/43	Exactly memorized all elder-shared sentence pairs, but regressed Bible and did not beat v10 on heldout usage.	model eval
v15.0-soft-lexical-hint-bible-gvn-token v15.0-soft-lexical-hint-bible-gvn-token	Lexical-hint diagnostic negative result	43.72	n/a	n/a	Reproduced train rows strongly, but did not beat v12 on heldout Bible.	model eval
v13.0-retrieval-context-bible-gvn-token v13.0-retrieval-context-bible-gvn-token	Retrieval-prefix diagnostic negative result	29.71	n/a	n/a	Retrieval context as an NLLB source prefix was harmful; heldout Bible dropped sharply.	model eval
v12.0-tagged-direct-plus-reference-bible-gvn-token v12.0-tagged-direct-plus-reference-bible-gvn-token	Current Bible draft fallback route candidate	44.36	n/a	n/a	Best Bible draft heldout score before v19, but still zero exact heldout reproduction.	model eval
v11.0-byt5-bible-control-32row v11.0-byt5-bible-control-32row	Byte-model control negative result	5.71	n/a	n/a	Failed the tiny overfit/control path; NLLB/LoRA remains the proven memorizing path for now.
v10.0-tagged-bible-plus-glossary-usage-tpi v10.0-tagged-bible-plus-glossary-usage-tpi	Current usage/general draft fallback route candidate	43.29	56.18	n/a	Best heldout usage signal so far; weaker Bible than v12/v19.	model eval
v9.8-tagged-bible-plus-db-usage-tpi v9.8-tagged-bible-plus-db-usage-tpi	First DB-usage multitask diagnostic internal proof	44.04	39.47	n/a	Memorized DB train examples but did not generalize well to word-id heldout usage.	model eval
v9.7-tagged-direct-plus-reference-bible-tpi v9.7-tagged-direct-plus-reference-bible-tpi	Reference-conditioning baseline internal proof	44.34	n/a	n/a	Established tagged Bible direct/reference training as viable, but still zero exact heldout reproduction.	model eval
v8.0-diagnostic-gates-summary v8.0-diagnostic-gates-summary	Pipeline proof gate internal proof	n/a	n/a	n/a	Proved tokenizer/LoRA/training/merge path could memorize; shifted problem from plumbing to data/task/generalization.
0.7.0-full-tpi-proxy-1.3b 0.7.0-full-tpi-proxy-1.3b	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.4.0-full-gvn-token 0.4.0-full-gvn-token	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.1.0-mini-pilot 0.1.0-mini-pilot	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-smoke 0.1.0-smoke	kuku_yalanji_ebible_parallel_smoke_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-baseline 0.1.0-baseline	kuku_yalanji_ebible_parallel_v0.1.0 training ready	n/a	n/a	n/a	No verdict recorded.
1.0.0-rc1 1.0.0-rc1	First leakage-controlled Mi'gmaq dictionary-example research candidate research only	n/a	n/a	n/a	Machine-integrity gates pass, but linguistic-quality promotion does not: frozen-test chrF++ is 21.43 with 0/742 exact matches and material lexical/content failures. Permit only a visibly warned research preview pending qualified Mi'gmaq review.	model eval