Translate v2

runtime verified research candidate

Model evaluation bench for the Kuku Yalanji training run. This page tracks the full v8-v20 diagnostic ladder, saved outputs, downloads, and resource profiles from each RunPod run.

Version lab Model registry

Where the model is after 48 hours

The training pipeline works. v20 trained the full candidate corpus and fixed the length-compression failure, but it also proved that scale alone is not enough. The hard problem is now product routing, mixture control, and faithful generalization, not GPU plumbing.

Saved evals only

Exact known text

lookup-first

Bible references, DB usage examples, and elder-shared sentence pairs should come from the approved source table, not generation.

Bible draft

v12.0

Best current Bible-draft fallback by heldout Bible chrF; still not faithful canonical reproduction.

Usage draft

v10.0

Best current heldout DB/general usage signal. Newer balanced runs have not beaten it.

Latest research

v20.0

Full-candidate-corpus diagnostic; length compression is fixed, but faithful generalization is still unresolved.

Latest run

Run: v20 full candidate corpus
Rows: 35,394 train / 4,433 validation / 4,397 test
Corpus shape: 20,911 source pairs expanded into tagged tasks
Training: 1h52m48s trainer time, 0.98 steps/s
RunPod cost: about $3.51

Model releaseDirection

Evaluation release

Live inference is intentionally off. Each release is reviewed through saved evaluation artifacts, exact-match counts, chrF/BLEU scores, resource usage, and downloadable model files.

Selected release: v24.3-joint-lexeme-dose29-s3598-20260715
Direction: eng-gvn
Saved eval rows: see artifacts
Release role: Current runtime-verified Kuku Yalanji research candidate for closed-set lexical reconstruction and separately labelled sentence drafts

Current eval summary

Kuku Yalanji · v24.3-joint-lexeme-dose29-s3598-20260715

No compact saved-output preview is attached to this release yet. Use the artifacts and metrics sections below.

This page shows reproducible eval artifacts only. Approved Bible text, database examples, and elder-shared sentence pairs should remain lookup-first in the product.

Dataset

115,136 materialized rows: 2,724 governed lexemes x29 plus 18,070 non-Bible sentence rows x2; one-seed development screen

Training rows

115,136

GPU

NVIDIA RTX A6000 48GB

Artifacts

model n/a · adapter public

Release verdict

The governed training-overlapping lexical gate and every frozen development retention check passed. ADVANCE authorizes further research, not unrestricted deployment: the model remains 0/43 exact on the elder diagnostic and lacks speaker-diverse human evaluation.

The headline 98.9721% score is accepted-reference reconstruction on 2,724 governed dictionary records shown 29 times each during training. It is not unseen-word generalisation or sentence accuracy.
The selected step improved every reported development sentence corpus-chrF endpoint over untouched v21.2, but the run used one seed and development data; elder exact remained 0/43.
The older 297-prompt benchmark scored 91 exact because only 208 prompts overlap and the two resources have different accepted surfaces; 206/208 overlapping predictions were supported by the governed census.
All 28 remaining governed-census failures are published with token, part-of-speech, semantic-domain, closest-reference, and checkpoint-trajectory evidence.
Dynamic CPU loading was verified with both task prefixes. It peaked at 8.00 GiB cgroup memory, settled around 7.0-7.2 GiB RSS, and produced 4.9-23.2 second requests under variable shared-disk pressure.
The adapter is large because PEFT persisted full embedding and output matrices for the trainable <lexeme> row. Repacking or merging requires full adapter-parity evaluation before release.
Known dictionary queries should continue to use deterministic lookup. Every generated sentence must be visibly labelled as an unverified research draft.

Downloads and artifacts

Public links resolve through the model-artifact file server. Merged models are large safetensor directories.

Comprehensive v24.3 model, training, evaluation, and hosting guidedocumentation · html Comprehensive v24.3 guide sourcedocumentation · markdown Complete v24.3 dynamic hosting bundlebundle · tar.gz v24.3 adapter-only hosting bundleadapter · tar.gz v24.3 hosting manifestmetadata · json v24.3 CPU runtime verificationevaluation · json All 2,724 selected-checkpoint lexical predictionsevaluation · json Complete 28-row v24.3 lexical failure ledgerevaluation · jsonl v24.3 final research decisionevaluation · json Exact v24.3 training and evaluation datadataset · tar.gz v24.3 release metadatametadata · json v24.3 release checksumsmetadata · sha256

Metrics

optimizer steps: 3598.0
train loss: 0.9482
closed set lexical rows: 2724.0
closed set lexical accepted exact: 2696.0
closed set lexical accepted exact percent: 99.0
closed set lexical wilson 95 low percent: 98.5
closed set lexical failures: 28.0
historical lexicon rows: 297.0
historical lexicon exact: 91.0
historical lexicon exact percent: 30.6
synthetic dev rows: 1609.0
synthetic dev chrf: 58.2
synthetic dev exact: 857.0
natural dev rows: 53.0
natural dev chrf: 36.2
natural dev exact: 0.0000
usage diagnostic rows: 84.0
usage diagnostic chrf: 53.2
usage diagnostic exact: 6.000
elder diagnostic rows: 43.0
elder diagnostic chrf: 25.3
elder diagnostic exact: 0.0000
elder mean token length ratio: 0.5925
empty outputs selected suites: 0.0000
adapter sha256: dd61583a60df2d538989e963e104cb626d78965d300e4e473e9a82ef59c04502
base model sha256: 7f9d0fe325e9e4568e45f13179adb336b93bbd53e83ddab2826e999eba3c76f7

Resource profile

Average GPU: n/a
Max GPU: n/a
Mean VRAM: n/a
Max VRAM: 36,586 MiB
Max power: n/a
Cost class: n/a
Estimated run cost: n/a

Version ladder

Every material training branch from the last two days is listed here so the model line is auditable. chrF is useful for comparison, but exact approved known resources stay lookup-first.

23 releases

Version	Role	Bible ref	Usage	Elder rows	Decision	Links
v24.3-joint-lexeme-dose29-s3598-20260715 v24.3-joint-lexeme-dose29-s3598-20260715	Current runtime-verified Kuku Yalanji research candidate for closed-set lexical reconstruction and separately labelled sentence drafts runtime verified research candidate	n/a	53.17	0/43	The governed training-overlapping lexical gate and every frozen development retention check passed. ADVANCE authorizes further research, not unrestricted deployment: the model remains 0/43 exact on the elder diagnostic and lacks speaker-diverse human evaluation.	eval
v21.2-claude-balanced-replay-guarded-20260714 v21.2-claude-balanced-replay-guarded-20260714	Current Kuku Yalanji research model of record: frozen v21.2 weights with validated guarded decoding; CPU inference is intentionally unloaded research only	43.84	47.96	0/43	Retain the exact published v21.2 weights and guarded decoder as reproducible research artifacts, but do not load them for public CPU inference while natural elder-register transfer remains unsolved. The homepage uses separately labelled dictionary-context prompting until a stronger RunPod candidate clears the frozen linguistic and safety gates and receives appropriate speaker review.	model eval
v23.0-attested-narrative-adaptation-failed v23.0-attested-narrative-adaptation-failed	Rejected three-seed attested-narrative adaptation retained as compact negative evidence negative result	n/a	n/a	n/a	Do not promote. Seed 73 improved the held-out Text 3 corpus chrF++ point estimate by 0.6414, below the preregistered +1.0 floor; its paired interval crossed zero, untagged synthetic repetition worsened, and isolated lexical reconstruction fell to 46/297.	eval
v22.0-step-matched-replay-3120-failed v22.0-step-matched-replay-3120-failed	Rejected one-variable replay-exposure experiment retained as negative evidence negative result	n/a	n/a	n/a	Do not promote. Step 3,120 failed the frozen greedy checkpoint gate because tagged synthetic repeated-segment rows were 37 against a maximum of 25, and paired comparison showed lower tagged synthetic, untagged synthetic, and dictionary-usage agreement than the final v21.2 step 4,155 checkpoint.	eval
v21.2-claude-balanced-replay-v2-candidate v21.2-claude-balanced-replay-v2-candidate	Historical v21.2 weights release; superseded for serving by the guarded-policy release without changing weights research only	43.91	47.14	0/43	Balanced replay preserves synthetic performance while materially recovering dictionary-usage and Bible retention relative to v21.1. Natural elder-register transfer remains unsolved, and synthetic morpheme loops remain a hard safety failure; serve only as a clearly labelled research draft pending speaker review.	model eval
v21.1-codex-synthetic-direct-v2-candidate v21.1-codex-synthetic-direct-v2-candidate	Kuku Yalanji translation model v2 candidate research only	28.76	34.40	0/43	The model learned the synthetic treatment and remained stable without the task tag, but elder-shared, dictionary-usage, and Bible controls regressed severely. Preserve as a research artifact; keep retrieval-first routing and require balanced replay plus speaker review before promotion.
v20.0-full-candidate-corpus-gvn v20.0-full-candidate-corpus-gvn	Full-candidate-corpus diagnostic artifact internal proof	39.39	43.26	0/43	Full-corpus v20 fixed the length-compression failure and completed the 35k-row diagnostic, but it is not a faithful general model: Bible heldout stays near 39 chrF, heldout usage is 43.26 chrF, and elder-shared sentence pairs regressed to 0/43 exact.	model eval
v19.0-balanced-replay-from-v12-gvn v19.0-balanced-replay-from-v12-gvn	Balanced replay artifact internal proof	44.29	53.03	29/43	Restores near-v12 Bible heldout while adding elder sentence-pair/usage signal, but does not replace v10 usage or v12 Bible routing.	model eval
v18.0-usage-elder-sentence-continuation-from-v10 v18.0-usage-elder-sentence-continuation-from-v10	Elder sentence-pair memorization proof artifact internal proof	41.25	54.52	43/43	Exactly memorized all elder-shared sentence pairs, but regressed Bible and did not beat v10 on heldout usage.	model eval
v15.0-soft-lexical-hint-bible-gvn-token v15.0-soft-lexical-hint-bible-gvn-token	Lexical-hint diagnostic negative result	43.72	n/a	n/a	Reproduced train rows strongly, but did not beat v12 on heldout Bible.	model eval
v13.0-retrieval-context-bible-gvn-token v13.0-retrieval-context-bible-gvn-token	Retrieval-prefix diagnostic negative result	29.71	n/a	n/a	Retrieval context as an NLLB source prefix was harmful; heldout Bible dropped sharply.	model eval
v12.0-tagged-direct-plus-reference-bible-gvn-token v12.0-tagged-direct-plus-reference-bible-gvn-token	Current Bible draft fallback route candidate	44.36	n/a	n/a	Best Bible draft heldout score before v19, but still zero exact heldout reproduction.	model eval
v11.0-byt5-bible-control-32row v11.0-byt5-bible-control-32row	Byte-model control negative result	5.71	n/a	n/a	Failed the tiny overfit/control path; NLLB/LoRA remains the proven memorizing path for now.
v10.0-tagged-bible-plus-glossary-usage-tpi v10.0-tagged-bible-plus-glossary-usage-tpi	Current usage/general draft fallback route candidate	43.29	56.18	n/a	Best heldout usage signal so far; weaker Bible than v12/v19.	model eval
v9.8-tagged-bible-plus-db-usage-tpi v9.8-tagged-bible-plus-db-usage-tpi	First DB-usage multitask diagnostic internal proof	44.04	39.47	n/a	Memorized DB train examples but did not generalize well to word-id heldout usage.	model eval
v9.7-tagged-direct-plus-reference-bible-tpi v9.7-tagged-direct-plus-reference-bible-tpi	Reference-conditioning baseline internal proof	44.34	n/a	n/a	Established tagged Bible direct/reference training as viable, but still zero exact heldout reproduction.	model eval
v8.0-diagnostic-gates-summary v8.0-diagnostic-gates-summary	Pipeline proof gate internal proof	n/a	n/a	n/a	Proved tokenizer/LoRA/training/merge path could memorize; shifted problem from plumbing to data/task/generalization.
0.7.0-full-tpi-proxy-1.3b 0.7.0-full-tpi-proxy-1.3b	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.4.0-full-gvn-token 0.4.0-full-gvn-token	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.	model eval
0.1.0-mini-pilot 0.1.0-mini-pilot	kuku_yalanji_ebible_parallel_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-smoke 0.1.0-smoke	kuku_yalanji_ebible_parallel_smoke_v0.1.0 internal proof	n/a	n/a	n/a	No verdict recorded.
0.1.0-baseline 0.1.0-baseline	kuku_yalanji_ebible_parallel_v0.1.0 training ready	n/a	n/a	n/a	No verdict recorded.
1.0.0-rc1 1.0.0-rc1	First leakage-controlled Mi'gmaq dictionary-example research candidate research only	n/a	n/a	n/a	Machine-integrity gates pass, but linguistic-quality promotion does not: frozen-test chrF++ is 21.43 with 0/742 exact matches and material lexical/content failures. Permit only a visibly warned research preview pending qualified Mi'gmaq review.	model eval