# v20.0 Full Candidate Corpus Live Run Note

Run ID: `v20.0-full-candidate-corpus-gvn-3epoch-lr2e-5`

Started: 2026-07-02 13:39 UTC on RunPod pod `wesa6w06ovb8ag`

GPU: NVIDIA L40S, 46 GB VRAM, `$0.99/hr`

Dataset: `v20.0-full-candidate-corpus-gvn`

- Train rows: 35,394
- Validation rows: 4,433
- Test rows: 4,397
- Bible rows are doubled into direct translation and reference-conditioned tasks.
- DB usage examples are split by word id with no train/heldout word-id overlap.
- Elder-shared sentence pairs are included as replay rows for training and as a separate post-train eval set.

Training configuration:

- Base model: `v12.0-tagged-direct-plus-reference-bible-gvn-token-4096row-25epoch-batch16/merged`
- Target language token: `gvn_Latn`
- LoRA rank/alpha: `64/128`
- Effective batch size: `16`
- Epochs: `3`
- Learning rate: `2e-5`
- Max source/target length: `320/256`
- Decode during eval: greedy, no repetition penalty, no no-repeat ngram block

Live status:

- Training is confirmed active, not just launched.
- GPU memory is stable around 21.5 GB / 46 GB.
- First observed training loss sequence through step 1100: `2.2798 -> 1.7971 -> 1.7111 -> 1.6641 -> 1.6185`.
- Step 1100 validation generation completed in `654.5s`.
- Step 1100 capped validation metrics: `eval_loss=1.7448`, `BLEU=11.82`, `chrF=37.60`.
- `checkpoint-1100` was written and training resumed.
- Step 2200 validation generation completed in `587.1s`.
- Step 2200 capped validation metrics: `eval_loss=1.6535`, `BLEU=12.92`, `chrF=39.15`.
- `checkpoint-2200` was written and training resumed.
- Step 3300 validation generation completed in `604.0s`.
- Step 3300 capped validation metrics: `eval_loss=1.6113`, `BLEU=12.92`, `chrF=39.12`.
- `checkpoint-3300` was written and training resumed. The capped validation chrF is flat from step 2200 while training loss continues to fall.
- Step 4400 validation generation completed in `550.7s`.
- Step 4400 capped validation metrics: `eval_loss=1.5919`, `BLEU=13.04`, `chrF=39.50`.
- `checkpoint-4400` was written and training resumed. This is the best capped validation chrF so far, but only a modest gain.
- Step 5500 validation generation completed in `569.7s`.
- Step 5500 capped validation metrics: `eval_loss=1.5804`, `BLEU=13.41`, `chrF=39.88`.
- `checkpoint-5500` was written and training resumed into the final training segment.
- Raw training completed at `6636/6636` after about `1h52m47s` trainer time.
- Final trainer metrics logged `train_runtime=6767.7s`, `train_samples_per_second=15.69`, `train_steps_per_second=0.981`, and `train_loss=1.4851`.
- Final scheduled validation around step 6600 completed in `542.6s`.
- Step 6600 capped validation metrics: `eval_loss=1.5765`, `BLEU=13.33`, `chrF=39.72`.
- The step 5500 checkpoint remains the best capped validation chrF so far. The trainer is now running post-training validation/test evaluation before writing the manifest and merged model.
- Trainer capped test evaluation completed with `test_loss=1.5613`.
- `model_manifest.json`, `adapter/`, and `merged/` were written.
- The driver started the saved post-train eval suite on the merged model, beginning with `eval_train_sample`.
- Saved eval `train_sample` completed: `2456` rows, `BLEU=23.35`, `chrF=47.10`.
- The driver moved to `validation_bible_direct`.
- Saved eval `validation_bible_direct` completed: `2153` rows, `BLEU=11.98`, `chrF=39.30`.
- Saved eval `validation_bible_ref` completed: `2153` rows, `BLEU=11.93`, `chrF=39.09`.
- The driver moved to `test_bible_direct`.
- Saved eval `test_bible_direct` completed: `2135` rows, `chrF=39.41`.
- Saved eval `test_bible_ref` completed: `2135` rows, `chrF=39.39`.
- Saved eval `heldout_usage` completed: `84` rows, `chrF=43.26`.
- The driver moved to `elder_sentence_pair_all`.
- Saved eval `elder_sentence_pair_all` completed: `43` rows, `BLEU=12.64`, `chrF=36.24`.
- This is a regression from the smaller v18/v19 elder-shared sentence-pair memorization behavior.
- The driver moved to the final large `test_all` eval.
- Saved eval `test_all` completed: `4397` rows, `BLEU=11.92`, `chrF=39.39`, `135` exact rows, mean length ratio `1.023`.
- Final diagnosis: v20 fixes the systematic under-generation problem and proves the full-candidate-corpus training path works, but it is not a faithful general model. Bible heldout stays near `39` chrF, heldout DB usage is `43.26` chrF, and elder-shared sentence pairs regress to `0/43` exact.
- Artifacts were fetched to the local archive and public model-artifact directory.
- RunPod pod `wesa6w06ovb8ag` was stopped after artifact fetch. Estimated wall time billed: about `3.55h`; estimated cost: about `$3.51`.
- `https://mobtranslate.com/translate/v2` was rebuilt and restarted as an eval-only page with v20 registered as the latest diagnostic release.
- Live inference remains disabled: `/api/translate/v2` returns `not_configured` unless a future endpoint is explicitly configured.

Completed actions:

1. Checkpoint/eval completion confirmed.
2. GPU utilization, memory, power, and RunPod cost monitored.
3. Model/eval/log artifacts fetched.
4. Pod stopped promptly.
5. Saved eval outputs published to the eval-only model page and registry.

Next training implication:

- Do not spend the next run on another larger undifferentiated full-corpus pass. v20 shows that the mixed objective is washing out the elder-shared sentence-pair behavior and not improving Bible faithfulness. The next run should either be route-specific or should add stronger mixture control/checkpoint selection against separate Bible, DB usage, and elder-shared sentence-pair gates.