# Kuku Yalanji Translation Model Technical Report For Third-Opinion Review

Date: 2026-07-02 UTC

Prepared for: MobTranslate Kuku Yalanji model-training review

Primary live review page: https://mobtranslate.com/translate/v2

Public v20 artifact directory:

```text
https://mobtranslate.com/model-artifacts/kuku-yalanji-nllb-lora/v20.0-full-candidate-corpus-gvn/
```

Local canonical research root:

```text
/mnt/donto-data/donto-resources/research/translation-training/kuku-yalanji-runpod-2026-06-30/
```

This report is deliberately long and self-contained. It is intended to be handed to another agent or researcher for a third opinion before defining the next formal goal.

## 1. Executive Summary

We are trying to build an English -> Kuku Yalanji translation model for MobTranslate. The current model line uses NLLB-200 as the base translation model and LoRA fine-tuning on available English/Kuku Yalanji parallel data. The system has gone through smoke tests, full Bible-domain training, diagnostic overfit gates, reference-conditioning experiments, DB usage-example ingestion, elder-shared sentence-pair ingestion, lexical-constraint diagnostics, candidate-router diagnostics, and finally a full-candidate-corpus v20 run.

The most important current result:

```text
v20 completed successfully, but v20 is not a faithful general model.
```

v20 trained on the largest current candidate set:

```text
35,394 train rows
4,433 validation rows
4,397 test rows
3 epochs
NVIDIA L40S
about $3.51 RunPod cost
```

v20 fixed the earlier systematic length-compression failure:

```text
test_all mean length ratio: 1.023
```

But v20 did not improve faithfulness:

```text
test_all chrF: 39.39
test_all exact: 135 / 4,397
Bible test direct chrF: 39.41
Bible test reference-conditioned chrF: 39.39
heldout DB usage chrF: 43.26
heldout DB usage exact: 3 / 84
elder-shared sentence-pair chrF: 36.24
elder-shared sentence-pair exact: 0 / 43
```

The v20 result is especially important because it falsifies a tempting simple hypothesis: "we just need to train on all 20k available sentence pairs." We did a full candidate-corpus continuation, and it did not solve faithfulness. It made length behavior healthier, but it washed out the small high-value elder-shared sentence-pair behavior and did not recover the best Bible/DB scores from earlier smaller runs.

The best current product routing decision remains:

```text
Exact known resources: lookup-first, not generation.
Bible draft fallback: v12.0 currently best stable Bible-draft branch.
Usage/general draft fallback: v10.0 remains best heldout DB usage branch.
Latest research artifact: v20.0, diagnostic only.
```

The main technical conclusion:

```text
The problem is no longer "can the pipeline train?"
The pipeline can train.
The problem is objective design: multiple incompatible tasks are being forced into one generator without enough task/domain control, and exact reproduction tasks are being treated as generation tasks when they should be retrieval/lookup.
```

## 2. What We Need A Third Opinion On

Please review the following questions:

1. Given v20's full-corpus failure, should v21 be another NLLB LoRA run, or should the next work be data/evaluation/routing before spending GPU?
2. If v21 is a GPU run, should it be route-specific, multi-adapter, curriculum-based, or a new architecture/control?
3. Is the current NLLB `gvn_Latn` added-token strategy still defensible, or should we revert to the `tpi_Latn` proxy for some branches?
4. Is the full Bible+sentence-candidate corpus too noisy for a single continuation objective?
5. Should DB usage examples and elder-shared sentence pairs be used primarily for training, primarily for exact lookup, or both?
6. Is chrF moving us in the wrong direction because Kuku Yalanji references sometimes contain approved interpretive expansions rather than literal direct translations?
7. Would a retrieval-augmented generator, candidate selector, or post-editing/reranking system be more rational than another monolithic generator run?
8. What is the cheapest next experiment that would genuinely reduce uncertainty?

## 3. Current Product And Safety Position

The public MobTranslate page at:

```text
https://mobtranslate.com/translate/v2
```

is currently an evaluation-only model page. It intentionally does not run live inference. The API confirms this:

```json
{
  "success": false,
  "status": "not_configured",
  "error": "No translate/v2 inference endpoint is configured for this deployment."
}
```

This is deliberate. None of the trained models should be presented as a faithful or community-approved translator. The page now shows saved evaluation artifacts, downloadable model files, metrics, resource usage, and version history.

The product logic we currently believe is correct:

```text
If the input maps to an approved known Bible reference:
  return the approved corpus text by lookup.

If the input maps to a known MobTranslate DB usage example:
  return the approved DB example by lookup.

If the input maps to an elder-shared sentence pair:
  return the approved sentence pair by lookup.

Only use neural generation for draft translation where no exact approved resource exists.
```

This is a central architectural point. A neural model should not be asked to memorize a database when the key is available. Approved known resources are evidence-backed records, not model outputs.

## 4. Language, Dataset, And Naming

Target language:

```text
Name: Kuku Yalanji
Internal code: gvn
ISO 639-3: gvn
NLLB native language token: no native gvn_Latn token
```

Important terminology:

```text
Use "elder-shared sentence pairs". Do not name the rows by the source medium.
```

The current high-value non-Bible data was transcribed from elder-shared source pages into aligned sentence pairs. The source medium is not the important unit; the supervised training unit is the sentence pair.

Current core data sources:

```text
1. eBible-derived English/Kuku Yalanji parallel corpus.
2. eBible-derived sentence-candidate expansion.
3. MobTranslate DB usage examples.
4. Elder-shared sentence pairs.
```

Rights status in the registry:

```text
Rights granted for elder-shared/DB examples.
Bible-derived rows are being used as research diagnostics.
Known approved resources remain lookup-first.
```

## 5. Training Harness Overview

Training code:

```text
/mnt/donto-data/workspace/mobtranslate.com/training/translation/train_nllb_lora.py
```

Evaluation code:

```text
/mnt/donto-data/workspace/mobtranslate.com/training/translation/evaluate_nllb_lora.py
```

v20 dataset builder:

```text
/mnt/donto-data/workspace/mobtranslate.com/training/translation/build_v20_full_candidate_corpus.py
```

v20 RunPod driver:

```text
/mnt/donto-data/workspace/mobtranslate.com/training/translation/run_v20_full_candidate_corpus.sh
```

### 5.1 NLLB Fine-Tuning Details

The trainer uses Hugging Face `Seq2SeqTrainer`, PEFT LoRA, and sacreBLEU/chrF metrics.

Important implementation details:

```text
Base model: usually facebook/nllb-200-distilled-1.3B for serious runs.
Task: seq2seq translation, English -> Kuku Yalanji.
Source language token: eng_Latn.
Target language token: either tpi_Latn proxy or added gvn_Latn.
Forced BOS token: set to target language token id.
LoRA task type: SEQ_2_SEQ_LM.
Metrics during trainer eval: BLEU and chrF with word_order=2.
Merged model: adapter is merged and saved as a full safetensors model after training.
```

The trainer explicitly handles added language codes:

```text
add_lang_code(tokenizer, model, lang_code, init_from=None)
```

For `gvn_Latn`, the trainer can add a new special token and optionally initialize it from an existing language code such as `tpi_Latn`. It also updates tokenizer language maps:

```text
lang_code_to_id
fairseq_tokens_to_ids
id_to_lang_code
fairseq_ids_to_tokens
```

Generation setup:

```text
model.config.forced_bos_token_id = target_lang_id
model.generation_config.forced_bos_token_id = target_lang_id
```

This was one of the key things v8 audited.

### 5.2 LoRA Configuration

Early runs used smaller LoRA target sets. Later serious runs expanded LoRA targets.

v20 LoRA configuration:

```text
lora_r: 64
lora_alpha: 128
lora_dropout: 0.0
target modules:
  q_proj
  k_proj
  v_proj
  out_proj
  fc1
  fc2
```

This is a high-capacity LoRA relative to the early smoke/pilot runs. v20's failure should not be dismissed as obviously just "LoRA too small." Capacity may still matter, but the result strongly points to objective/data mixture problems.

### 5.3 Decode Configuration For Diagnostics

For sanity runs and v20, decode constraints were deliberately disabled:

```text
num_beams: 1
no_repeat_ngram_size: 0
repetition_penalty: 1.0
length_penalty: 1.0
```

This matters because earlier under-generation could have been caused or worsened by decoding penalties. v20 removed that confound and still did not reach faithfulness.

### 5.4 Post-Training Evaluation

The evaluator:

```text
loads merged model
sets src_lang/tgt_lang
sets forced_bos_token_id
generates predictions
writes prediction JSON with references
computes BLEU and chrF
```

Post-eval analysis then computes:

```text
exact match count
exact rate
empty output count
length ratio
max token repeat share
reference token recall
task counts
tier counts
DB headword hit rate
elder-shared sentence-pair exact count
```

The page at `/translate/v2` reads the registry and saved prediction artifacts. It does not depend on a live model.

## 6. Dataset Inventory

### 6.1 Original Bible Corpus

The early full corpus used the eBible-derived high-confidence parallel data:

```text
dataset: kuku_yalanji_ebible_parallel_v0.1.0
high-confidence train rows in later v20 split: 13,270
```

The larger candidate corpus adds sentence-candidate rows:

```text
dataset: kuku_yalanji_ebible_parallel_with_sentence_candidates_v0.1.0
total source pairs referenced in app copy: 20,911
directional bidirectional rows in older planning docs: 41,822
```

v20 English -> Kuku source split:

```text
train source rows: 16,623
validation source rows: 2,153
test source rows: 2,135
```

v20 Bible tier counts:

```text
train:
  high_confidence_verse: 13,270
  sentence_candidate: 3,353

validation:
  high_confidence_verse: 1,670
  sentence_candidate: 483

test:
  high_confidence_verse: 1,657
  sentence_candidate: 478
```

### 6.2 DB Usage Examples

MobTranslate DB contained:

```text
449 translated usage examples
365 train usage rows
84 heldout usage rows
298 train word_ids
74 heldout word_ids
0 train/heldout word_id intersection
```

The word-id split is important. It avoids leakage where examples for the same dictionary headword appear in both train and heldout.

### 6.3 Elder-Shared Sentence Pairs

Current supervised elder-shared sentence-pair set:

```text
43 aligned English -> Kuku Yalanji rows
25 high-confidence
18 medium-confidence
```

These are small but high-value because they are non-Bible, community-relevant, and directly aligned.

### 6.4 v18 Dataset

v18 combined the v10 usage branch with elder-shared sentence pairs.

Manifest:

```text
prepared/v18.0-v10-plus-elder-sentence-pair/v18_elder_sentence_pair_manifest.json
```

Important counts:

```text
v10 train rows: 5,556
v10 eval train rows: 4,461
v10 validation rows: 128
v10 heldout all rows: 734
usage train rows: 365
usage train oversampled rows: 1,460
usage heldout rows: 84
elder-shared sentence-pair rows: 43
elder-shared oversampled rows: 516
train rows: 6,072
usage + elder continuation rows: 1,976
eval usage + elder rows: 408
```

v18 training note:

```text
Train from v10 merged model on usage + elder rows.
Evaluate DB heldout, elder-shared sentence pairs, and Bible heldout separately.
```

### 6.5 v19 Dataset

v19 balanced v12 Bible replay with DB usage and elder-shared sentence-pair rows.

Manifest:

```text
prepared/v19.0-balanced-replay-after-elder-sentence-pair-ingest/v19_balanced_replay_manifest.json
```

Important counts:

```text
bible train tagged rows: 4,096
bible train direct rows: 2,048
bible train reference rows: 2,048
bible heldout direct rows: 325
bible heldout reference rows: 325
usage train rows: 365
usage heldout rows: 84
elder-shared sentence-pair rows: 43
usage oversampled rows: 1,460
elder-shared sentence-pair oversampled rows: 344
train rows: 5,900
validation rows: 235
heldout all rows: 734
heldout all plus elder rows: 777
```

v19 was trained from v12 with target `gvn_Latn`, low LR, and no target token reinitialization.

### 6.6 v20 Dataset

v20 was the first full candidate-corpus run after v18/v19.

Manifest:

```text
prepared/v20.0-full-candidate-corpus-gvn/v20_full_candidate_corpus_manifest.json
```

Purpose:

```text
All English -> Kuku rows from the 20,911-pair Bible+sentence-candidate export,
with tagged direct and reference-conditioned Bible tasks plus DB usage and
elder-shared sentence pairs.
```

v20 output counts:

```text
train rows: 35,394
validation rows: 4,433
test rows: 4,397
bible train direct rows: 16,623
bible train reference rows: 16,623
bible validation direct rows: 2,153
bible validation reference rows: 2,153
bible test direct rows: 2,135
bible test reference rows: 2,135
usage oversampled rows: 1,460
elder-shared sentence-pair oversampled rows: 688
eval train sample rows: 2,456
```

v20 oversampling:

```text
DB usage train rows: 365 * 4 = 1,460
elder-shared sentence pairs: 43 * 16 = 688
```

Even with 16x oversampling, the elder-shared sentence pairs made up only:

```text
688 / 35,394 = 1.94% of train rows
```

DB usage made up:

```text
1,460 / 35,394 = 4.13% of train rows
```

Bible direct/reference rows dominated:

```text
33,246 / 35,394 = 93.94% of train rows
```

This ratio is probably one of the key reasons v20 washed out the small non-Bible behavior.

## 7. Version Ladder And What Each Version Proved

The registry currently lists 16 releases. The important ones are summarized below.

| Version | Role | Rows / Setup | Key Result | Interpretation |
|---|---:|---|---|---|
| `0.1.0-smoke` | End-to-end proof | 256 train, 1 epoch | chrF about 5.45 | Proved RunPod, LoRA, merge, eval pipeline. Not a translator. |
| `0.1.0-mini-pilot` | Budget-safe pilot | 2,048 train, 1 epoch | chrF about 4.18 | Still not trained enough. |
| `0.4.0-full-gvn-token` | First full high-confidence `gvn_Latn` | 13,270 train, 8 epochs | test chrF 32.66, length ratio 0.752 | Better than pilots but under-generating and weak. |
| `0.7.0-full-tpi-proxy-1.3b` | Full high-confidence tpi proxy | 13,270 train, 8 epochs | test chrF 37.09, standalone chrF 34.49, length ratio 0.757 | tpi proxy beat custom gvn early, but still compressed output. |
| `v8` | Diagnostic gates | 1 -> 8 -> 32 -> 256 rows | pipeline could memorize | Training/tokenization/LoRA mostly alive. |
| `v9.7` | Tagged direct/reference Bible tpi | 4,096 train, 25 epochs | Bible ref chrF 44.34, exact 0 | Reference conditioning helps score but not exact reproduction. |
| `v10` | Bible + DB usage multitask tpi | 5,556 train, 20 epochs | usage chrF 56.18, Bible ref chrF 43.29 | Best current usage/general draft fallback. |
| `v11` | ByT5 small control | 32 rows | heldout chrF about 5.9 | Not useful as tested; does not justify switching yet. |
| `v12` | Tagged direct/reference Bible `gvn_Latn` | 4,096 train, 25 epochs | Bible direct/ref chrF 44.37/44.36, exact 0 | Best current Bible draft fallback. |
| `v13` | Retrieval-prefix Bible diagnostic | 512 train, 20 epochs | Bible chrF about 29.7 | This NLLB prefix format hurt. Retrieval still useful for product lookup/evidence. |
| `v14` | Decode-time lexical constraints | decode-only | oracle constraints improved chrF but caused over-generation | Hard `force_words_ids` is too brittle. |
| `v15` | Soft lexical hint training | 2,048 train, 15 epochs | Bible direct/ref chrF 44.15/43.72 | Soft oracle hints did not beat v12. |
| `v16` | Resource router/candidate ceiling | no GPU | lookup exact 734/734; oracle candidates still far from exact | Exact known resources should be lookup-first. |
| `v18` | DB usage + elder continuation from v10 | 1,976 continuation rows, 10 epochs | elder exact 43/43; usage chrF 54.52; Bible regressed | Proved elder rows can be memorized in a route-specific continuation. |
| `v19` | Balanced replay from v12 | 5,900 train, 8 epochs | Bible ref chrF 44.29; usage chrF 53.03; elder exact 29/43 | Best broad post-elder artifact, but not universal. |
| `v20` | Full candidate corpus from v12 | 35,394 train, 3 epochs | test chrF 39.39; elder exact 0/43 | Full-corpus scale alone failed. Diagnostic artifact only. |

## 8. Detailed Run History And Findings

### 8.1 Smoke And Early Full Runs

The early goal was simply to prove that we could:

```text
create a deterministic dataset
launch RunPod
install dependencies
train LoRA on CUDA
merge adapter into full model
evaluate heldout rows
pull artifacts back
publish model registry entries
```

This succeeded.

The early full runs showed two important behaviors:

1. The model could learn Kuku-looking generation better than smoke runs.
2. It systematically compressed long references and dropped content.

The strongest early full run before diagnostics was:

```text
0.7.0-full-tpi-proxy-1.3b
target: tpi_Latn proxy
test chrF: 37.09
standalone chrF: 34.49
mean length ratio: 0.7566
```

That run was not faithful enough. It generated semantic skeletons, not complete approved references.

### 8.2 v8 Gated Diagnostic Suite

The 2026-07-01 third-party review recommended not doing another large ambiguous run until we proved the pipeline could memorize tiny clean subsets. v8 implemented that.

v8 gates:

```text
v8.0 audit
v8.1 1-row overfit
v8.2 8-row overfit
v8.3 32-row overfit
v8.4 256-row overfit
v8.5 decode sweep
v8.6 32-row target-token A/B with gvn_Latn
v8.7 32-row reference-conditioning A/B
```

v8.0 audit checked:

```text
base model
source_lang
target_lang
forced_bos_token_id
tokenizer language code id
label tokenization/truncation
LoRA module matching
trainable parameter count
```

v8.1 demonstrated that 1-row overfit could exact-match with BLEU/chrF 100 after training. This ruled out catastrophic tokenization/trainer failure.

v8.6 demonstrated that a custom added `gvn_Latn` token initialized from `tpi_Latn` could memorize 32 rows:

```text
target_lang: gvn_Latn
target_lang_init_from: tpi_Latn
eval_bleu: 100
eval_chrf: 100
```

v8.7 demonstrated that reference-conditioned input could also memorize 32 rows:

```text
input format: <bible_ref> REF <eng> English text
target_lang: tpi_Latn
eval_bleu: 100
eval_chrf: 100
```

Conclusion:

```text
The basic pipeline is alive.
NLLB + LoRA can memorize clean Kuku Yalanji rows.
The failure of later models is not simply "training is broken."
```

### 8.3 v9 Bible Curriculum And Reference Conditioning

v9 tested progressively cleaner Bible curricula and reference-conditioned inputs.

The most important v9 release in the registry:

```text
v9.7-tagged-direct-plus-reference-bible-tpi
train rows: 4,096
epochs: 25
batch size: 16
heldout Bible direct chrF: 44.13
heldout Bible ref chrF: 44.34
heldout exact: 0
```

Interpretation:

```text
Reference conditioning improved or stabilized heldout chrF.
Exact canonical reproduction still failed.
Bible exact reproduction should not rely on generation.
```

### 8.4 v10 DB Usage Multitask

v10 introduced DB usage examples into the multitask setup.

Registry metrics:

```text
v10.0-tagged-bible-plus-glossary-usage-tpi
train rows: 5,556
epochs: 20
batch size: 16
heldout usage chrF: 56.18
heldout usage exact: 3
heldout Bible direct chrF: 43.20
heldout Bible ref chrF: 43.29
heldout all chrF: 43.60
```

Interpretation:

```text
v10 remains the best heldout DB usage/general draft branch.
It did not solve exact usage output, but it is better than later broad runs on usage chrF.
```

### 8.5 v11 ByT5 Control

v11 tested a byte-level model control:

```text
v11.0-byt5-bible-control-32row
heldout Bible direct chrF: 6.09
heldout Bible ref chrF: 5.71
heldout all chrF: 5.90
```

This was not a strong enough result to switch away from NLLB. It also may not be a fair full test of byte-level models because it was small, but it does not justify spending more there before resolving clearer NLLB/data issues.

### 8.6 v12 Best Bible Draft Branch

v12 used `gvn_Latn` and tagged direct/reference Bible data:

```text
v12.0-tagged-direct-plus-reference-bible-gvn-token
train rows: 4,096
epochs: 25
batch size: 16
heldout Bible direct chrF: 44.37
heldout Bible ref chrF: 44.36
heldout exact: 0
```

v12 is currently the best Bible draft fallback by heldout Bible chrF. It still cannot reproduce approved Bible references exactly.

### 8.7 v13 Retrieval-Prefix Diagnostic

v13 tried retrieved Bible context as a prefix to NLLB:

```text
v13.0-retrieval-context-bible-gvn-token
train rows: 512
epochs: 20
batch size: 8
heldout Bible direct chrF: 29.49
heldout Bible ref chrF: 29.71
```

This NLLB prefix format was harmful. It does not mean retrieval is bad. It means "stuff retrieved examples into NLLB source text" was not a good path in this branch.

Retrieval remains strongly recommended for:

```text
exact known-resource lookup
evidence display
nearest-example suggestions
candidate construction
possibly an LLM reranker/post-editor
```

### 8.8 v14 Lexical Constraints

v14 was a decode-only diagnostic against v12. It tested whether target-side lexical constraints could force missing reference words.

First 64 heldout direct rows:

| Decode | BLEU | chrF | Exact | Constraint Hits | Mean Length Ratio | Max Length Ratio | Rows > 2x Length |
|---|---:|---:|---:|---:|---:|---:|---:|
| v12 greedy first64 | 13.66 | 44.60 | 0 | n/a | 1.009 | 1.43 | 0 |
| beam4 unconstrained | 13.27 | 44.29 | 0 | n/a | 1.014 | 1.50 | 0 |
| oracle constraints, 3 words | 8.21 | 47.73 | 1 | 168/183 | 1.622 | 6.67 | 14 |
| oracle constraints, 1 word | 9.25 | 44.43 | 0 | 53/63 | 1.534 | 6.67 | 10 |

Interpretation:

```text
The model can mechanically obey target-side constraints.
The constraints contain useful signal.
Hard constrained beam search causes severe over-generation.
Do not use force_words_ids as product decode path yet.
Convert lexical hints into supervised formatting or soft reranking instead.
```

### 8.9 v15 Soft Lexical Hints

v15 trained soft lexical hints rather than hard decode constraints.

Registry metrics:

```text
v15.0-soft-lexical-hint-bible-gvn-token
train rows: 2,048
epochs: 15
batch size: 16
heldout Bible direct chrF: 44.15
heldout Bible ref chrF: 43.72
```

Interpretation:

```text
Soft hints did not beat v12.
The missing ingredient is probably not a simple lexical-hint prefix.
Do not scale this branch.
```

### 8.10 v16 Resource Router And Candidate Ceiling

v16 was local/no-GPU and is central to product strategy.

v16.0 exact resource router:

```text
known heldout rows: 734
Bible exact rows: 650
DB usage exact rows: 84
router exact: 734 / 734
mean char-F: 100
token recall: 1.0
length ratio: 1.0
```

This proves that known resources can be served exactly by lookup.

v16.1 Bible candidate ceiling:

```text
best single Bible candidate: 52.99 mean sentence char-F
oracle over candidates: 58.76 mean sentence char-F
oracle exact: 1 / 325
```

v16.1 usage candidate ceiling:

```text
best single usage candidate: 65.36 mean sentence char-F
oracle over candidates: 69.54 mean sentence char-F
oracle exact: 8 / 84
```

v16.2 simple selectors:

```text
Bible cross-validated grid selector: 53.49 mean sentence char-F, 0 exact
Usage cross-validated grid selector: 65.01 mean sentence char-F, 3 exact
```

Interpretation:

```text
Known resources should be lookup-first.
Candidate selection has some headroom, especially usage, but simple selectors do not approach oracle enough yet.
Another generator-only run is unlikely to solve exact known-resource reproduction.
```

### 8.11 v18 Elder-Shared Sentence-Pair Continuation

v18 trained from v10 on DB usage plus elder-shared sentence-pair rows.

Registry metrics:

```text
v18.0-usage-elder-sentence-continuation-from-v10
train rows: 1,976
validation rows: 408
epochs: 10
batch size: 16
train loss: 0.2773
train usage+elder chrF: 100
train usage+elder exact: 408
elder-shared sentence-pair chrF: 100
elder-shared sentence-pair exact: 43 / 43
heldout usage chrF: 54.52
heldout usage exact: 3 / 84
heldout Bible direct chrF: 41.42
heldout Bible ref chrF: 41.25
heldout all chrF: 41.67
```

Interpretation:

```text
The elder-shared rows are learnable.
The model can memorize all 43 when the branch focuses on them.
But Bible quality regresses, so this is not a universal model.
It is a route-specific proof artifact.
```

### 8.12 v19 Balanced Replay

v19 trained from v12 on a balanced replay set:

```text
v12 Bible replay
DB usage rows
elder-shared sentence pairs
```

Registry metrics:

```text
v19.0-balanced-replay-from-v12-gvn
train rows: 5,900
validation rows: 235
epochs: 8
batch size: 16
train loss: 0.2129
validation BLEU: 95.75
validation chrF: 97.18
heldout usage chrF: 53.03
heldout usage exact: 6 / 84
heldout Bible direct chrF: 44.16
heldout Bible ref chrF: 44.29
heldout all chrF: 44.46
heldout all plus elder chrF: 47.47
elder-shared sentence-pair chrF: 93.86
elder-shared sentence-pair exact: 29 / 43
length ratio mean heldout all: 0.976
```

Interpretation:

```text
v19 is the best broad post-elder artifact.
It nearly preserves v12 Bible chrF.
It recovers useful elder-shared sentence-pair behavior, but not full exact memorization.
It still does not beat v10 on usage chrF.
It is not a universal replacement.
```

### 8.13 v20 Full Candidate Corpus

v20 trained on the full candidate corpus from v12.

Run ID:

```text
v20.0-full-candidate-corpus-gvn-3epoch-lr2e-5
```

RunPod:

```text
pod id: wesa6w06ovb8ag
GPU: NVIDIA L40S, 46 GB
price: $0.99/hr
pod created: 2026-07-02 13:30 UTC
training started: 2026-07-02 13:39 UTC
pod stopped: 2026-07-02 17:03 UTC
estimated wall billing time: about 3.55h
estimated cost: about $3.51
```

Training config:

```text
base model: v12 merged model
source_lang: eng_Latn
target_lang: gvn_Latn
LoRA r/alpha: 64 / 128
LoRA dropout: 0.0
target modules: q_proj,k_proj,v_proj,out_proj,fc1,fc2
epochs: 3
batch size: 8
gradient accumulation: 2
effective batch: 16
learning rate: 2e-5
max source length: 320
max target/new tokens: 256
eval/save steps: 1100
logging steps: 220
generation: greedy, no anti-repeat constraints
```

Trainer metrics:

```text
train_runtime: 6767.6911s
train_samples_per_second: 15.69
train_steps_per_second: 0.981
train_loss: 1.4851415328335662
trainer validation BLEU: 13.4068
trainer validation chrF: 39.8821
trainer test BLEU: 13.1574
trainer test chrF: 40.2506
```

Checkpoint progression:

| Step | eval loss | BLEU | chrF | Note |
|---:|---:|---:|---:|---|
| 1100 | 1.7448 | 11.82 | 37.60 | first scheduled eval |
| 2200 | 1.6535 | 12.92 | 39.15 | improvement |
| 3300 | 1.6113 | 12.92 | 39.12 | flat |
| 4400 | 1.5919 | 13.04 | 39.50 | modest gain |
| 5500 | 1.5804 | 13.41 | 39.88 | best scheduled capped chrF |
| 6600 | 1.5765 | 13.33 | 39.72 | final scheduled eval slightly lower chrF |

Resource profile:

```text
resource samples: 2,174
monitor window: 2026-07-02T13:39:22Z -> 2026-07-02T16:44:14Z
mean GPU utilization: 42.95%
max GPU utilization: 100%
mean VRAM used: 17,287 MiB
max VRAM used: 21,914 MiB
mean GPU power: 191.50 W
max GPU power: 290.49 W
mean Python CPU: 116.66%
max Python CPU: 537.0%
max Python RSS: 7,537.6 MiB
workspace used: 67%
```

v20 post-train eval suite:

| Eval split | Rows | BLEU | chrF | Exact | Mean length ratio | Notes |
|---|---:|---:|---:|---:|---:|---|
| train sample | 2,456 | 23.35 | 47.10 | 170 | 1.007 | includes sampled train Bible + usage + elder |
| validation Bible direct | 2,153 | 11.98 | 39.30 | 52 | 1.001 | one empty output |
| validation Bible ref | 2,153 | 11.93 | 39.09 | 46 | 1.011 | reference-conditioned |
| test Bible direct | 2,135 | 11.90 | 39.41 | 65 | 1.016 | heldout Bible direct |
| test Bible ref | 2,135 | 11.93 | 39.39 | 67 | 1.018 | heldout Bible ref |
| heldout usage | 84 | 7.84 | 43.26 | 3 | 1.440 | DB headword hit 59/84 |
| elder-shared sentence pairs | 43 | 12.64 | 36.24 | 0 | 0.834 | severe regression |
| test all | 4,397 | 11.92 | 39.39 | 135 | 1.023 | full saved test |

v20 test-all composition:

```text
bible_ref: 2,135
translate: 2,178
usage_example_with_glossary: 84
high_confidence_verse: 3,314
sentence_candidate: 956
none: 127
```

v20 reference-token recall:

```text
test_all mean reference token recall: 0.3766
test_all median reference token recall: 0.3333
elder-shared mean reference token recall: 0.2771
heldout usage mean reference token recall: 0.4364
```

This is the major hidden failure behind a reasonable length ratio. v20 often outputs the right amount of text, but not the right target content.

v20 final diagnosis:

```text
Full-corpus training succeeded mechanically.
Length compression was fixed.
Faithfulness did not improve.
Elder-shared sentence-pair behavior collapsed from 29/43 exact in v19 and 43/43 in v18 to 0/43 in v20.
Heldout usage regressed from v10/v18/v19.
Bible heldout regressed from v12/v19.
```

## 9. Cross-Version Metrics Table

The key scorecard values:

| Version | Bible direct chrF | Bible ref chrF | Usage chrF | Elder chrF | Elder exact | Heldout all chrF | Decision |
|---|---:|---:|---:|---:|---:|---:|---|
| `v9.7` | 44.13 | 44.34 | n/a | n/a | n/a | n/a | reference-conditioned baseline |
| `v10` | 43.20 | 43.29 | 56.18 | n/a | n/a | 43.60 | best usage/general fallback |
| `v12` | 44.37 | 44.36 | n/a | n/a | n/a | n/a | best Bible draft fallback |
| `v13` | 29.49 | 29.71 | n/a | n/a | n/a | n/a | do not scale retrieval-prefix NLLB |
| `v15` | 44.15 | 43.72 | n/a | n/a | n/a | n/a | lexical hints not enough |
| `v18` | 41.42 | 41.25 | 54.52 | 100.00 | 43/43 | 41.67 | route-specific elder proof |
| `v19` | 44.16 | 44.29 | 53.03 | 93.86 | 29/43 | 44.46 | best broad post-elder artifact |
| `v20` | 39.41 | 39.39 | 43.26 | 36.24 | 0/43 | 39.39 | full corpus failed |

The trend is clear:

```text
v12/v19 are best for Bible draft.
v10 is best for usage/general heldout chrF.
v18 proves elder rows can be learned when focused.
v20 is worse than v19 on every route-specific score except length ratio.
```

## 10. Why v20 Matters

v20 is not a failure of execution. It is a useful negative result.

It proves:

```text
The training infrastructure can handle the full candidate corpus.
The model can train for several hours on L40S without resource failure.
The model can output length-balanced Kuku-looking text.
The current objective does not yield faithful Kuku Yalanji.
```

It falsifies:

```text
"Just train on the full 20k sentence-pair set."
"Just add sentence candidates."
"Just oversample elder rows inside the giant corpus."
"Length ratio was the main problem."
```

Length ratio was a problem in early runs, but after v20 it is no longer sufficient as a progress indicator. A model can have near-1.0 length ratio and still be lexically/semantically wrong.

## 11. Primary Failure Modes

### 11.1 Mixed Objective Interference

v20's objective mixes:

```text
<translate> English Bible source -> approved Kuku Bible text
<bible_ref> reference + English Bible source -> approved Kuku Bible text
usage_example_with_glossary DB examples
elder-shared sentence-pair rows
sentence-candidate rows
```

The train distribution is still overwhelmingly Bible:

```text
Bible tagged rows: 93.94%
DB usage rows: 4.13%
elder-shared sentence-pair rows: 1.94%
```

The model appears to learn a generalized Bible-domain Kuku-like generation mode rather than preserving small route-specific exact examples.

### 11.2 Exact Reproduction Is The Wrong Job For A Generator

For known Bible references, DB examples, and elder-shared sentence pairs, exact output can be retrieved. Generation should be used for unknown draft translation, not database reproduction.

v16 proves exact lookup:

```text
734 / 734 known resource heldout exact by router
```

No generator has come close to that.

### 11.3 Bible Target Text Is Often Not A Literal English Translation

Many Bible target references are approved renderings that may expand, interpret, or restructure the English. English-only input is underspecified for exact approved text. Even reference-conditioned generation did not solve exact reproduction.

This means:

```text
For exact Bible reference output, retrieval should be canonical.
For draft translation, Bible references may be useful training data but can distort ordinary translation behavior.
```

### 11.4 Sentence Candidates May Introduce Noise

v20 included sentence candidates:

```text
train sentence candidates: 3,353 source rows, doubled into direct/ref tasks
validation sentence candidates: 483
test sentence candidates: 478
```

These may be lower confidence than high-confidence verse rows. They expand scale but may blur the mapping the model is supposed to learn.

v20 is not enough to prove sentence candidates are harmful, but it shows that adding them silently does not solve the problem.

### 11.5 DB And Elder Rows Are Too Small To Survive A Bible-Dominated Mix

v18:

```text
elder exact: 43/43
```

v19:

```text
elder exact: 29/43
```

v20:

```text
elder exact: 0/43
```

The row count and loss distribution in v20 almost certainly drowned the 43 rows even with 16x oversampling.

### 11.6 chrF Alone Can Reward Partial Kuku-Looking Output

chrF is useful for regression checks, but a 39-44 chrF model can still:

```text
drop clauses
substitute generic religious phrasing
copy only high-frequency Kuku patterns
miss named entities or participants
miss target morphology
produce a correct-looking length with wrong lexical content
```

The report should not treat chrF as "faithfulness."

### 11.7 Target Token Strategy Is Still Not Fully Settled

Observed:

```text
early 0.7 tpi_Latn proxy beat early custom gvn_Latn run
v8 proved added gvn_Latn can memorize 32 rows
v12 gvn_Latn became best Bible draft branch
v20 gvn_Latn full-corpus failed
```

This does not prove `gvn_Latn` is bad. It proves token strategy is not the only variable.

## 12. Literature-Informed Context Already Considered

Relevant notes are in:

```text
docs/kuku-yalanji-next-model-research-2026-07-01.md
docs/aycock-low-resource-mt-notes-2026-07-01.md
```

Prior takeaways:

1. Parallel examples are the main fuel for low-resource MT. Grammar prose is useful primarily when converted into supervised examples, gloss tasks, contrastive tests, or evaluation cases.
2. More reasoning or chain-of-thought prompting is not a free translation fix.
3. chrF/BLEU need qualitative error analysis and human/community review.
4. NLLB is defensible but cannot invent competence for an unsupported language without representative parallel examples.
5. Indigenous MT progress is often modest; domain-specific error analysis matters.
6. Retrieval and dictionary entries can help, but only with domain-specific evaluation.

This aligns with the local results.

## 13. App And Artifact State

The model page:

```text
https://mobtranslate.com/translate/v2
```

now shows:

```text
v20 latest diagnostic release
saved evals only
no live model box required
v8-v20 diagnostic ladder
download links for merged models/adapters/evals
resource profiles and costs
```

The v20 public artifacts include:

```text
merged/
adapter/
RUN_REPORT.md
LIVE_RUN_NOTE.md
model_manifest.json
post_eval_analysis.json
resource_summary.json
source_length_preflight.json
eval_test_all_predictions_greedy.json
eval_test_bible_direct_predictions_greedy.json
eval_test_bible_ref_predictions_greedy.json
eval_validation_bible_direct_predictions_greedy.json
eval_validation_bible_ref_predictions_greedy.json
eval_heldout_usage_predictions_greedy.json
eval_elder_sentence_pair_all_predictions_greedy.json
eval_train_sample_predictions_greedy.json
```

v20 artifact checks:

```text
post_eval_analysis.json: HTTP 200
RUN_REPORT.md: HTTP 200
eval_test_all_predictions_greedy.json: HTTP 200
```

Live inference:

```text
disabled
/api/translate/v2 returns not_configured
```

RunPod:

```text
no active pods after v20
```

## 14. Cost And Resource Observations

Relevant RunPod costs:

```text
v19 balanced replay:
  RTX PRO 4500 Blackwell
  about $0.74/hr
  17m15s trainer runtime
  estimated cost about $0.68

v20 full candidate corpus:
  L40S
  $0.99/hr
  about 3.55h wall
  estimated cost about $3.51
```

The cost is not the main blocker yet. The main blocker is experimental ambiguity. Another v20-scale run is affordable in isolation, but wasteful if it does not reduce uncertainty.

Approximate cost classes:

| Experiment type | Likely cost | Notes |
|---|---:|---|
| local/no-GPU data/eval/router analysis | $0 | Should be done before next GPU run. |
| v19-sized continuation | <$1-$2 | Good for route-specific or mixture probes. |
| v20-sized full 3-epoch continuation | ~$3-$5 | Affordable but only justified with a better objective. |
| multiple hyperparameter sweep over full corpus | $15-$50+ | Not justified yet. |
| large architecture switch / full fine-tune | unknown, likely higher | Should wait until NLLB objective/data issues are clearer. |

## 15. Hypotheses Ranked After v20

### H1: Mixed-objective interference is the dominant problem

Status: strongly supported.

Evidence:

```text
v18 focused elder continuation: 43/43 elder exact.
v19 balanced replay: 29/43 elder exact.
v20 full corpus: 0/43 elder exact.
```

As Bible/candidate rows dominate, small route-specific behavior collapses.

### H2: Exact known-resource reproduction should be retrieval, not generation

Status: proven for known rows.

Evidence:

```text
v16 resource router: 734/734 exact.
Best generator: nowhere near exact lookup.
```

### H3: More full-corpus data alone is not enough

Status: strongly supported.

Evidence:

```text
v20 full candidate corpus underperforms v12/v19 Bible and v10 usage.
```

### H4: Length compression was a symptom, not the root cause

Status: strongly supported.

Evidence:

```text
v20 length ratio mean: 1.023
v20 reference token recall mean: 0.377
v20 still not faithful.
```

### H5: Sentence-candidate noise may hurt

Status: plausible, not proven.

Evidence:

```text
v20 adds sentence candidates and regresses.
But v20 also changes many other variables.
Need controlled high-confidence-only vs candidate-inclusive comparison.
```

### H6: `gvn_Latn` target token is not the main blocker

Status: likely.

Evidence:

```text
v8 gvn token memorized 32 rows.
v12 gvn token is best Bible draft branch.
v20 gvn failed, but due to objective/data likely.
```

### H7: LoRA capacity is not obviously the main blocker

Status: likely but not fully proven.

Evidence:

```text
v20 used r=64/alpha=128 on q/k/v/out/fc1/fc2.
v8/v18 memorization proves capacity for small exact sets.
```

Capacity may still matter for full generalization, but not before data/objective clarity.

### H8: NLLB may be structurally limited for this unsupported language

Status: plausible, not decisive.

Evidence:

```text
NLLB was not built with gvn_Latn.
However NLLB learns some Kuku patterns and can memorize.
ByT5-small control was weak, but not a definitive architecture comparison.
```

### H9: The evaluation target itself is ambiguous

Status: strongly supported.

Evidence:

```text
Bible exact reproduction, general translation, DB usage examples, and elder-shared sentence pairs are different tasks.
One unqualified "English -> Kuku Yalanji" objective hides these differences.
```

## 16. What Not To Do Next

Do not define the next goal as:

```text
"Run v21 on all data for more epochs."
```

Why:

```text
v20 already tested "all data" and regressed.
More epochs could overfit the Bible-heavy distribution further.
It would not address objective ambiguity.
```

Do not define the next goal as:

```text
"Make v20 live on the website."
```

Why:

```text
v20 is not faithful.
The page should remain eval-only.
Exact known resources should be lookup-first.
```

Do not define the next goal as:

```text
"Use hard lexical constraints in production."
```

Why:

```text
v14 showed hard constraints cause severe over-generation.
```

Do not define the next goal as:

```text
"Switch architecture immediately."
```

Why:

```text
ByT5-small control was weak.
The NLLB pipeline still has unresolved objective/data issues.
A model switch may be useful later, but it is not the cheapest uncertainty reducer right now.
```

## 17. Candidate Next Goals

The next goal should be chosen after third-party review. Below are technically plausible options.

### Option A: No-GPU v21 Planning Gate Before Training

Goal:

```text
Build a route-specific evaluation harness and data audit before any v21 GPU run.
```

Tasks:

```text
1. Create one canonical eval suite:
   - Bible exact known-resource lookup
   - Bible draft heldout
   - DB usage heldout by word_id
   - elder-shared sentence-pair exact
   - general short sentence probes
   - length/repetition/reference-token recall diagnostics

2. Create a per-row model comparison table across v10/v12/v18/v19/v20.

3. Cluster rows where:
   - v19 beats v20
   - v20 beats v19
   - v10 beats all on usage
   - v18 exact-memorizes but v20 fails

4. Decide whether v21 should optimize a composite score or a single route.
```

Cost:

```text
$0 RunPod
```

Pros:

```text
Cheapest.
Reduces ambiguity before spending GPU.
Likely improves next run definition.
```

Cons:

```text
Does not train a new model immediately.
```

My view:

```text
This is the most rational immediate next goal.
```

### Option B: v21 Route-Specific Usage/Elder Adapter

Goal:

```text
Train a route-specific usage/elder adapter from v10 or v18, not a universal model.
```

Possible setup:

```text
base: v10 merged or v18 merged
target: tpi_Latn if continuing v10/v18 branch, or gvn_Latn only if carefully staged
data:
  DB usage train rows
  elder-shared sentence pairs
  maybe small anti-forgetting Bible replay
validation:
  DB heldout by word_id
  all 43 elder-shared sentence pairs
  Bible heldout only as regression monitor
selection metric:
  composite usage chrF + elder exact, with Bible regression cap
```

Pros:

```text
Directly attacks the non-Bible route.
Cheap, v18-sized.
Already proven learnable.
```

Cons:

```text
Does not solve Bible/general universal model.
May overfit exact elder rows unless separated from lookup.
```

Cost:

```text
Probably under $1-$2.
```

### Option C: v21 Balanced Multi-Task With Composite Checkpoint Selection

Goal:

```text
Train a broad model like v19, but with explicit composite eval/checkpoint selection across Bible, usage, and elder gates.
```

Key change from v20:

```text
Do not select checkpoint on Bible-heavy capped validation only.
Do not let the full corpus distribution dominate the loss.
Use a balanced validation metric:
  Bible draft chrF
  DB usage chrF
  elder exact / chrF
  length ratio constraints
  repeat constraints
```

Possible data ratio:

```text
Bible high-confidence tagged rows: controlled sample, not all candidates.
DB usage oversampled to a substantial share.
Elder-shared rows oversampled or batch-balanced.
Sentence candidates excluded initially or separately tagged with lower weight.
```

Pros:

```text
Most likely to improve on v19 without v20's washout.
Still one model artifact.
```

Cons:

```text
Requires modifying trainer/data loader or doing careful physical balancing.
Could still compromise all routes.
```

Cost:

```text
v19-sized to v20-sized, about $1-$5 depending row count/epochs.
```

### Option D: v21 High-Confidence-Only Full Bible Control

Goal:

```text
Test whether v20's sentence-candidate rows caused regression.
```

Setup:

```text
base: v12 merged
data:
  high-confidence Bible rows only
  direct/ref tags
  optionally DB/elder excluded
same v20 hyperparameters:
  lr 2e-5
  r=64
  3 epochs
evaluation:
  same v20 Bible direct/ref heldout
```

Pros:

```text
Isolates sentence-candidate noise.
Cheap if capped.
```

Cons:

```text
Does not solve DB/elder.
May simply reproduce v12-ish results.
```

Cost:

```text
Likely under v20 cost.
```

### Option E: v21 Retrieval/Lookup + Draft Model Product System

Goal:

```text
Stop trying to make one model do exact known-resource output.
Build product routing as the model system.
```

Architecture:

```text
input
-> exact canonical-resource matcher
-> DB usage/example matcher
-> elder-shared sentence-pair matcher
-> if exact known resource found, return lookup result with evidence
-> else route to draft model:
   Bible-like -> v12/v19
   usage/general -> v10
   latest research -> optional v20 display only
```

Pros:

```text
Immediate product correctness for known resources.
Uses the strongest branch per domain.
Avoids pretending v20 is faithful.
```

Cons:

```text
Does not create one clean downloadable universal model.
Requires product engineering/routing work.
```

Cost:

```text
Mostly local app/database work, no GPU.
```

### Option F: v21 Architecture Control

Goal:

```text
Run a fairer architecture control only after data/eval is stable.
```

Candidates:

```text
ByT5 base/large or mT5 control
character/byte-level seq2seq
small instruction model with retrieval examples
NLLB distilled 1.3B full fine-tune if budget allows
```

Pros:

```text
Could solve unsupported-token/tokenizer issues.
```

Cons:

```text
More expensive and less grounded until objective is fixed.
v11 ByT5-small was weak.
```

My view:

```text
Do not choose this as the immediate next goal unless another reviewer sees a strong reason.
```

## 18. My Recommended Next Goal

I would not start v21 GPU training immediately.

Recommended next goal:

```text
Build a no-GPU route-specific evaluation and selection harness, then run one cheap v21 balanced/checkpointed training run only after the harness can score candidate checkpoints by route.
```

Concrete wording:

```text
Goal: Prepare v21 by building a route-specific evaluation harness over all existing v10/v12/v18/v19/v20 outputs, define a composite gate for Bible draft, DB usage, and elder-shared sentence pairs, then run one budget-capped v21 training job that optimizes that composite gate instead of a Bible-dominated validation loss.
```

The gate should require:

```text
Bible direct/ref chrF >= v19 minus small tolerance
DB usage chrF >= v10 or clear qualitative improvement
elder-shared exact >= v19, ideally back toward v18
test_all length ratio between 0.85 and 1.20
no repeat-collapse outliers
no live product promotion unless human/community review approves
```

If the user wants an immediate training run anyway, choose a v19-sized run, not a v20-sized all-corpus run:

```text
v21.0-balanced-composite-gate-from-v12
base: v12 merged
data: high-confidence Bible tagged subset + DB usage + elder-shared sentence pairs
exclude sentence candidates in first v21 pass
use stronger DB/elder balance than v19
train 8-12 epochs
lr: 2e-5 to 3e-5
LoRA: r=64, alpha=128, same modules as v20
eval every small interval on separate Bible/usage/elder files
select by external composite score, not trainer's single validation chrF
```

## 19. Specific Implementation Recommendations For v21

### 19.1 Use External Composite Checkpoint Selection

The trainer currently selects best model by:

```text
metric_for_best_model = "chrf"
eval_dataset = tokenized["validation"]
```

In v20, validation was dominated by Bible direct/ref rows. This makes trainer checkpoint selection poorly aligned with the product routes.

Recommended:

```text
save checkpoints
run external eval on:
  heldout Bible direct
  heldout Bible ref
  heldout usage
  elder-shared sentence pairs
  general probe set
score composite
select checkpoint after training
```

Composite example:

```text
score =
  0.30 * normalized Bible ref chrF
  0.20 * normalized Bible direct chrF
  0.25 * normalized DB usage chrF
  0.15 * elder exact rate
  0.10 * elder chrF
  - penalties for length ratio outliers and repeats
```

This is just a starting point. The exact weights should reflect product priorities.

### 19.2 Stop Treating Physical Row Duplication As The Only Mixture Control

v20 used physical oversampling:

```text
usage 4x
elder 16x
```

This still left Bible at about 94% of train rows.

Better:

```text
implement domain-balanced batch sampling
or build explicitly balanced epoch shards
or train route-specific adapters
```

If physical duplication remains simplest, make the ratios explicit:

```text
Bible 50-60%
DB usage 20-25%
elder/shared + general probes 15-25%
```

Then validate by route.

### 19.3 Separate High-Confidence Bible From Sentence Candidates

Do not silently mix sentence candidates into the same training objective. Use tags:

```text
<translate>
<bible_ref>
<sentence_candidate>
<usage_example_with_glossary>
<elder_sentence_pair>
```

Or exclude sentence candidates until high-confidence rows are fully understood.

### 19.4 Preserve Exact Known Resources By Lookup

Training should not be evaluated as if it must exactly reproduce known resources when the app can look them up. The app should show:

```text
approved source text
model draft, if generated
nearest examples / evidence
```

This lets the model be useful without pretending it is authoritative.

### 19.5 Add Human-Readable Error Taxonomy

Add a qualitative report for each release with buckets such as:

```text
dropped clause
wrong participant/person
wrong polarity
wrong tense/aspect
generic Bible-style paraphrase
English leakage
named entity error
over-generation
under-generation
repetition
wrong headword usage
good draft / needs minor edit
exact approved match
```

This can be partly LLM-assisted, but final correctness needs human/community review.

### 19.6 Create A General Probe Set

The current heldout usage set is only 84 rows. Create a fixed small probe suite:

```text
20 everyday movement/action sentences
20 family/social sentences
20 object/place sentences
20 tense/aspect/polarity probes
20 DB-headword probes
20 elder/community-review probes
```

Even if references are incomplete at first, the probe set can record expectations and human corrections over time.

## 20. Data Paths And Artifacts For Review

Repository:

```text
/mnt/donto-data/workspace/mobtranslate.com
```

Registry:

```text
/mnt/donto-data/workspace/mobtranslate.com/apps/web/public/models/registry.json
```

Training code:

```text
/mnt/donto-data/workspace/mobtranslate.com/training/translation/
```

v20 prepared dataset:

```text
/mnt/donto-data/donto-resources/research/translation-training/kuku-yalanji-runpod-2026-06-30/prepared/v20.0-full-candidate-corpus-gvn/
```

v20 run archive:

```text
/mnt/donto-data/donto-resources/research/translation-training/kuku-yalanji-runpod-2026-06-30/runpod/v20.0-full-candidate-corpus-20260702T133916Z/
```

v20 public artifacts:

```text
/mnt/donto-data/donto-resources/research/translation-training/kuku-yalanji-runpod-2026-06-30/public-model-artifacts/kuku-yalanji-nllb-lora/v20.0-full-candidate-corpus-gvn/
```

v20 key files:

```text
model_manifest.json
post_eval_analysis.json
resource_summary.json
RUN_REPORT.md
LIVE_RUN_NOTE.md
eval_test_all_predictions_greedy.json
eval_heldout_usage_predictions_greedy.json
eval_elder_sentence_pair_all_predictions_greedy.json
```

v19 manifest:

```text
prepared/v19.0-balanced-replay-after-elder-sentence-pair-ingest/v19_balanced_replay_manifest.json
```

v18 manifest:

```text
prepared/v18.0-v10-plus-elder-sentence-pair/v18_elder_sentence_pair_manifest.json
```

v16 diagnostics:

```text
analysis/v16-router-candidate-diagnostic-2026-07-02/
analysis/v16.2-candidate-selector-diagnostic-2026-07-02/
```

v14 diagnostic:

```text
runpod/v14-lexical-constraints-ete777zxs8gl9l/v14-lexical-constraint-gpu-report-2026-07-02.md
```

## 21. Reviewer's Suggested Reading Order

For another agent with limited time:

1. Read this report sections 1, 7, 8.13, 9, 15, 17, 18.
2. Inspect:

```text
public-model-artifacts/kuku-yalanji-nllb-lora/v20.0-full-candidate-corpus-gvn/post_eval_analysis.json
apps/web/public/models/registry.json
prepared/v20.0-full-candidate-corpus-gvn/v20_full_candidate_corpus_manifest.json
analysis/v16-router-candidate-diagnostic-2026-07-02/v16-summary.json
```

3. Compare saved outputs:

```text
v20 eval_elder_sentence_pair_all_predictions_greedy.json
v19 eval_elder_sentence_pair_all_predictions_greedy.json
v18 eval elder sentence-pair predictions, if reviewing local archive
v10 heldout usage predictions
v12 heldout Bible predictions
```

4. Decide whether v21 should be:

```text
no-GPU eval harness first
route-specific adapter
balanced composite-gated model
data cleanup/control run
architecture switch
```

## 22. Bottom-Line Assessment

The current state is technically much better than at the start:

```text
RunPod setup works.
Training harness works.
Resource monitoring works.
Model artifacts are published.
The eval-only page works.
The pipeline can memorize clean rows.
The system has versioned reports and download links.
```

But the model quality problem is unsolved:

```text
v20 is not faithful.
v20 should not be used as product translator.
v20 should not be the base assumption that "more data solved it."
```

The strongest evidence points to:

```text
objective ambiguity
domain mixture imbalance
lookup-vs-generation confusion
Bible-domain overdominance
insufficient route-specific validation
```

The next goal should therefore not be a blind larger run. It should either:

```text
1. Build route-specific evaluation and composite checkpoint selection first, then train v21 cheaply; or
2. Train a small route-specific adapter with a very clear scorecard.
```

My preferred next goal:

```text
Build the route-specific v21 evaluation/selection harness, then run one budget-capped v21 balanced-composite training job only if the harness can prove what "better" means across Bible draft, DB usage, and elder-shared sentence pairs.
```