Mi'gmaq Translation Model v1: Training Record and Operator Handoff

Program: MobTranslate Mi'gmaq translation research
Started: 2026-07-12
Current phase: run complete; frozen test, artifact verification, and local serving checks complete
Deployment status: guarded noncommercial research preview only; unrestricted promotion blocked
Intended direction for v1: English to Mi'gmaq (eng_Latn -> custom mic_Latn)
Research root: /mnt/donto-data/donto-resources/research/translation-training/migmaq-runpod-2026-07-12/

1. Purpose and non-claims

This program builds a reproducible first neural translation candidate from the sentence examples in the Mi'gmaq/Mi'kmaq Online Talking Dictionary. The candidate is a research instrument. It is not a speaker, authority, normative orthography, or replacement for review by Mi'gmaq language experts.

The first candidate will be trained and evaluated on attested dictionary example pairs. The public interface must label generated text as model output and must not imply community endorsement. A model is not promoted merely because its training loss decreases.

2. Source, rights, dialect, and orthography

The source site says the project began in Listuguj and that its entries use Listuguj spellings. That statement defines the written-data scope of this run. Source-supplied speaker codes are retained as opaque provenance only. They are not expanded into identities, dialect labels, or consent claims.

The NLLB base model is also noncommercially licensed. Every dataset, adapter, merged checkpoint, model card, download page, and endpoint created by this program must therefore remain research-only and noncommercial unless the relevant rights holders grant broader permission.

3. Frozen source inventory

The source snapshot contains:

• 6,974 parsed dictionary entries;
• 7,235 example instances;
• 7,234 structurally complete Mi'gmaq-English example pairs;
• 7,227 unique exact example pairs before normalization audit;
• 8 repeated pair instances;
• 6,733 headwords with a non-empty English translation;
• 7,234 sentence recordings linked to examples.

Headword translations are lexical records, not sentence translations. They will be exported as a separate auxiliary dataset and will not be mixed into the sentence test score. The sentence candidate will initially use only complete, source-attested example pairs.

3.1 Frozen dataset release

Two independent disposable builds produced the same archive digest before the canonical release was written. The release builder refuses to overwrite this version.

One example at entries/u/utanmji'jmuowa/utanmji'jmuowa.html#example-0 has no English translation and is excluded with a structural-rejection record. Eight repeated bilingual instances collapse to their unique pairs while retaining every contributing source record. One U+0092 C1 control inherited from the source HTML is deterministically interpreted as its Windows-1252 punctuation value and recorded in the quality audit. No lexical content is silently corrected.

4. Corpus construction contract

The dataset builder must:

1. read the immutable manifests/entries.jsonl source snapshot;
2. validate every input record against a typed schema;
3. preserve Unicode, capitalization, punctuation, apostrophes, source URLs, entry IDs, example indexes, source checksums, and recording provenance;
4. reject only structurally unusable rows, while reporting suspicious rows for review instead of silently rewriting them;
5. normalize to Unicode NFC and collapse transport whitespace without lowercasing or stripping diacritics;
6. deduplicate exact normalized bilingual pairs while retaining every contributing source reference;
7. construct connected leakage groups for entries sharing a normalized headword, Mi'gmaq sentence, English sentence, or bilingual pair;
8. assign whole connected groups deterministically to train, validation, or test;
9. assert that no grouped entry, normalized source sentence, normalized target sentence, or normalized bilingual pair crosses a split;
10. publish JSONL data, manifests, rights records, leakage reports, checksums, and a reproducibility README.

The held-out test set is frozen before model selection. It must not be used to choose epoch count, learning rate, token initialization, decoding settings, or architecture. Selection uses training diagnostics and the validation split only. The test set is read once for the final candidate report.

5. Model design

The first conservative candidate reuses the proven MobTranslate NLLB-LoRA pipeline:

• base: facebook/nllb-200-distilled-600M;
• task: English to Mi'gmaq;
• source token: built-in eng_Latn;
• target token: added mic_Latn;
• adaptation: LoRA plus trainable shared embeddings and language-model head so the new token can be learned;
• precision: BF16 where the selected GPU supports it;
• tokenizer/model artifact: saved with the custom language token and merged checkpoint;
• platform: an on-demand RunPod GPU, deleted only after artifacts and logs are pulled and checksums match.

Mi'gmaq is not one of NLLB-200's built-in language tokens. Any existing token used to initialize mic_Latn is only a numerical starting point. It is not a claim of genealogical relationship, dialect identity, or linguistic equivalence. The choice and its validation evidence must be recorded.

6. Mandatory gates

6.1 Data gates

• source snapshot checksum recorded;
• typed parse succeeds for every entry;
• rights status is explicit on every row;
• structural rejection count and reasons published;
• exact duplicate policy published;
• train/validation/test leakage audit has zero violations;
• source-entry and normalized-text overlap have zero violations;
• split counts and length distributions published;
• archive hashes verify after packaging.

6.2 Plumbing gate

Before the full run, a small training-only subset must demonstrate that:

• the custom mic_Latn token is added and survives reload;
• gradients reach LoRA, shared embeddings, and the language-model head;
• the candidate can deliberately overfit a small training subset;
• forced target-language decoding produces non-empty Mi'gmaq-shaped output;
• save, merge, reload, and inference all succeed;
• no held-out test examples are present in the gate.

6.3 Candidate gates

The final report must include, at minimum:

• SacreBLEU signature and score;
• chrF++ score;
• exact-match rate, reported but not treated as the main linguistic metric;
• validation and frozen-test predictions with stable row IDs;
• bootstrap confidence intervals;
• length-ratio and empty-output diagnostics;
• repetition, copying, and degeneration checks;
• seen-headword versus unseen-headword slices;
• short/medium/long sentence slices;
• punctuation and apostrophe preservation diagnostics;
• manual error analysis covering lexical choice, person/number, animacy, transitivity, negation, word order, and hallucinated material;
• a written promotion ruling with limitations.

Automatic metrics alone cannot establish grammatical or cultural adequacy. Public promotion requires review by qualified Mi'gmaq speakers or language workers. Until that review exists, the model remains visibly labelled as a research candidate.

7. RunPod operating procedure

1. Create a versioned local run contract and checksum it.
2. Select a 48 GB GPU when custom-token embedding training requires it.
3. Create an on-demand pod with sufficient local workspace.
4. Bootstrap the exact Python environment and record package, CUDA, driver, and GPU versions.
5. Upload only the checksummed code bundle and frozen dataset bundle.
6. Verify remote checksums before execution.
7. Run the plumbing gate and inspect real GPU utilization, memory, loss movement, and artifact creation.
8. Stop if the plumbing gate fails; do not spend the full run budget on a broken path.
9. Run the full candidate with resource monitoring and persistent logs.
10. Evaluate and merge on the pod, then create hashes for every artifact.
11. Pull model, tokenizer, adapter, predictions, metrics, logs, environment lock, and run metadata.
12. Compare remote and local hashes.
13. Confirm all required artifacts open and the merged model performs inference locally or in a clean verification environment.
14. Delete the pod so billing stops.

No pod is considered working merely because it exists. Progress must be verified from changing step counts/loss, nonzero GPU utilization, bounded memory, and growing output/log files.

8. Artifact map

Operational code lives in the MobTranslate repository under apps/web/scripts/ and training/translation/. Research outputs stay on the mounted data disk.

9. Recovery and reproducibility

A later operator must be able to reproduce the exact dataset and determine whether a checkpoint came from it without querying production state. Therefore every run records:

• git commit plus dirty-worktree patch digest;
• source archive and manifest digests;
• dataset release ID and file hashes;
• deterministic split algorithm and seed namespace;
• model base revision;
• tokenizer special-token changes;
• all training and decoding arguments;
• package versions, Python, CUDA, driver, and GPU;
• RunPod pod ID, GPU type, start/end times, and billed duration when available;
• random seeds;
• metrics and per-row predictions;
• adapter and merged-model hashes;
• exact deployment and rollback commands.

If a model is later superseded, its files and evaluation remain immutable. Registry state changes by adding a new version and marking the former version superseded; artifacts are not overwritten.

10. Progress ledger

2026-07-12T09:00Z - program opened

• Confirmed the Kuku Yalanji NLLB-LoRA/RunPod stack is present and reusable.
• Confirmed the configured RunPod account is operational and has no active pod or network volume.
• Confirmed Mi'gmaq is not a built-in NLLB-200 language token; a custom token is required.
• Counted 7,234 complete source example pairs and separated lexical headword material from sentence material.
• Froze the noncommercial licensing and Listuguj-orthography scope.
• Began the deterministic corpus builder and leakage audit.

2026-07-12T09:16Z - dataset v1 frozen and published

• Added a typed, checksum-pinned TypeScript dataset builder and focused leakage-control tests.
• Built connected split components from normalized headword, Mi'gmaq sentence, and English sentence overlap.
• Froze 5,798 train, 686 validation, and 742 test sentence pairs.
• Exported 6,733 headword-gloss rows separately as lexical auxiliary material excluded from v1 training.
• Passed zero-overlap assertions for leakage group, entry, headword, Mi'gmaq text, English text, and bilingual pair.
• Verified every release-file checksum and the complete archive checksum.
• Rebuilt independently twice and obtained identical archive SHA-256 values.
• Published the noncommercial research dataset under /datasets/migmaq/.

2026-07-12T09:25Z - RunPod A40 provisioned

• Provisioned secure-cloud pod 8fyl89yeeesbyu: NVIDIA A40, 46,068 MiB VRAM, $0.44/hour.
• Set a hard automatic termination time of 2026-07-12T15:25:03Z.
• Verified the GPU, CUDA visibility, Python, memory, and workspace before transfer.
• Matched the code, run-contract, and dataset archive hashes after upload.
• Reverified every internal dataset checksum after remote extraction.
• Installed and recorded the resolved environment: Python 3.11.10, torch 2.4.1+cu124, Transformers 4.48.3, PEFT 0.19.1, Datasets 4.8.5, Accelerate 1.14.0, SacreBLEU 2.6.0.
• Started a five-second GPU/CPU/RAM/disk resource monitor and verified that its row count advanced.

2026-07-12T09:40Z - plumbing gate v1 failed; full run remained blocked

• Trained only the 128-row training-split overfit gate for 30 epochs.
• The in-memory adapter learned: validation chrF++ rose from 6.44 at epoch 5 to 40.60 at epoch 30, and the final logged training loss fell to 1.7091.
• The independently reloaded adapter exactly reproduced the trainer's chrF++ 40.60 and produced no empty outputs.
• The first merged checkpoint failed reload: 92 of 128 outputs were empty and the rest included pathological repetition. The machine verdict was fail; no full-corpus training started.
• Preserved the failed verdict, predictions, logs, checkpoints, and resource trace.

2026-07-12T10:00Z - PEFT/NLLB merge defect isolated and fixed

• Compared adapter, in-memory merge, serialized weights, and clean reload at tensor and prediction level.
• Found that PEFT replaced model.shared while NLLB's encoder and decoder retained the original runtime embedding object.
• Safe serialization then tied the reloaded decoder to the replacement matrix, producing a maximum decoder-weight divergence of 1.5062255859375 and output collapse.
• Fixed serialization by making the actual runtime decoder embedding canonical, retaining the independently trained output head, and materializing all three NLLB embedding alias keys.
• The corrected diagnostic reports zero state-dict differences and identical in-memory-versus-reloaded merged predictions.
• Did not relax the predeclared gate thresholds. Prepared a continuation from checkpoint 480 to 100 total epochs so the memorization criterion is tested as written.

2026-07-12T10:14Z - plumbing gate v2 passed

• Continued the same 128 training rows to 100 total epochs from the preserved checkpoint.
• Achieved 128/128 exact outputs: SacreBLEU 100.00, chrF++ 100.00, exact match 1.00.
• Reduced the logged loss from 7.7135 to 0.0982.
• Reloaded the corrected merged checkpoint and observed zero empty outputs and zero source-copy outputs.
• On 64 disjoint rows drawn only from the training split, produced 64 non-empty, distinct outputs. This sanity result is not reported as translation quality.
• Passed every predeclared machine gate without accessing validation or frozen test data.
• Pulled the verdict, failed-v1 evidence, predictions, model manifest, logs, package lock, and resource-monitor snapshot to the mounted research disk and checksummed them.
• Authorized the full 5,798-row run. The 742-row frozen test remains untouched.

2026-07-12T10:18Z - full candidate run started

• Started the eight-epoch full run with 5,798 training rows and 686 validation rows.
• The trainer command has no test-file argument. Checkpoint selection is validation chrF++ only; the 742-row frozen test remains inaccessible to training and selection.
• At step 180 (epoch 0.99), validation loss was 4.3357, SacreBLEU was 0.1677, and chrF++ was 12.4634. The untuned base proxy had validation chrF++ 11.3090.
• At step 360 (epoch 1.98), validation loss improved to 3.8356 and chrF++ improved to 15.3597. The best checkpoint therefore advanced from step 180 to step 360.
• At step 540 (epoch 2.97), validation loss improved again to 3.6013 and chrF++ to 16.4716. Checkpoint 540 became the new validation-selected best.
• At step 720 (epoch 3.96), validation loss was 3.4399 and chrF++ was 18.4789. At step 900 (epoch 4.95), loss was 3.3459 and chrF++ was 19.6308. Every scheduled checkpoint through step 900 improved the validation selection metric; checkpoint 900 is the current best.
• Observed training loss moved from 7.8123 near startup to 3.6888 by epoch 2.97. This is a process-health signal, not a translation-quality claim.
• Direct process and monitor checks showed the trainer alive, a changing optimizer step, nonzero A40 utilization, about 8.5 GB VRAM use, about 2.1 GB resident host memory, bounded temperature around 52 C, and a growing run directory. No empty-response or stalled-process behavior has been observed.
• Uploaded checksum-matched validation-selection and bootstrap-analysis utilities to the pod. The selector rejects empty files, duplicate IDs, non-validation rows, mismatched row sets, and unexpected row counts before ranking decoding configurations.
• Froze the complete post-training protocol as migmaq-v1-evaluation-code-20260712T1033Z.tar.gz; local and remote SHA-256 both equal 37286e683247aecbfc2e2573c1dee089770110349bbb7f32b8e337efffe70b6c.

2026-07-12T10:52Z - training and clean merged reload completed

• Completed 1,448 optimizer steps in 1,623.3 seconds. The training manifest contains 5,798 train rows, 686 validation rows, no test split, and no test-file digest.
• Validation chrF++ improved at every scheduled checkpoint. Step 1440 was selected with chrF++ 20.9027, SacreBLEU 1.1713, and validation loss 3.2163.
• Saved a 3.1 GB adapter and a 2.7 GB standalone merged model. The merge manifest records the corrected shared/encoder/decoder embedding materialization and an intentionally untied output head.
• Independently reloaded the merged directory and evaluated all 686 validation rows: chrF++ 20.9299, SacreBLEU 1.1681, zero exact matches, zero empty outputs, and zero source-copy outputs.
• Found that the original run inventory hashed full.log before the final timestamp was appended. Every other file matched. Preserved that failed check, fixed the future script order, generated SHA256SUMS.post-close, and verified every post-close file successfully.
• Re-ran the merged audit with the correct pinned 600M base-model metadata. The first audit remains preserved rather than overwritten.

2026-07-12T11:06Z - validation-only decoding selection frozen

• Compared three decoding configurations on the same ordered 686 validation IDs. All row counts and split labels were machine-checked; the frozen test was not read.
• Selected the balanced configuration: four beams, no-repeat trigram constraint, repetition penalty 1.10, and length penalty 1.0.
• Selected validation scores: chrF++ 21.0434 (95% bootstrap interval 20.4083 to 21.6518), SacreBLEU 1.3867 (0.9756 to 2.1040), exact match 0/686.
• The paired chrF++ gain over the zero-shot base proxy is 9.7344, with a paired 95% bootstrap interval of 9.1063 to 10.3009.
• Surface checks: zero empty outputs, zero source copies, zero repeated-trigram outputs, 676 unique outputs, and mean evaluation-summary character-length ratio 0.8758.
• The unconstrained configuration produced repeated trigrams in 45/686 outputs and failed the predeclared degeneration gate.
• By construction all 686 validation rows belong to headword groups absent from training; there is no seen-headword validation slice. This is evidence of strict lexical grouping, not evidence of broad linguistic competence.
• Verified all validation-selection files against selection/SHA256SUMS. The one-shot frozen-test runner is now authorized.

2026-07-12T11:11Z - one-shot frozen test completed

• Reserved exclusive access in the frozen-test ledger at 11:08:40Z and completed the single authorized run at 11:11:15Z.
• Evaluated exactly 742 unique split=test rows with the validation-selected balanced decoding configuration.
• Obtained chrF++ 21.4298 (95% bootstrap interval 20.8128 to 22.0332) and SacreBLEU 1.7795 (0.9308 to 2.6029).
• Observed 0 exact matches, 0 empty outputs, 0 source copies, 0 outputs equal to a training target, and 0 repeated-trigram outputs.
• Preserved every row-level prediction and the complete analysis. Prediction SHA-256 is 5d294244e21e160a8d9e8ae35eb385a93b3608cd51df534505088778767c7566; analysis SHA-256 is 2e02ba0250c38cddbc54392872c7d1bd3d0ae73291eecce46b2339a0f6b50fb7.
• Machine integrity passed, but linguistic promotion did not: manual reference comparison found material lexical substitution, omission, and participant-structure failures.

2026-07-12T11:36Z - complete artifact pull independently verified

• Pulled the adapter, standalone merged model, validation-selected resumable checkpoint, full logs, reports, selection ledger, frozen-test ledger, and resource trace to the mounted research disk.
• Matched the local 3,217,507,488-byte adapter to SHA-256 d85184d614f614815910442cbc5324f440c4030cdebd6b5cff2d3d7046381c67.
• Matched the local 2,804,330,488-byte merged weights to SHA-256 8df8467e96ba480e790951f86cdc30919cd1d441faf7d1c5c5521cef1c78eb01.
• Matched the recovery checkpoint adapter to the selected adapter and verified its optimizer, scheduler, RNG, trainer-state, tokenizer, and configuration files against the pod's post-close inventory.
• Rechecked the validation-selection, frozen-test, and manifest-of-manifests checksum ledgers successfully.

2026-07-12T11:39Z - local CPU serving gate passed

• Replaced the obsolete v21.1 Kuku Yalanji process while retaining the current v21.2 Kuku Yalanji service.
• Loaded the verified Mi'gmaq merged model in mobtranslate-migmaq-v1-infer.service with CPU float32, the selected decoding configuration, a loopback-only socket, a 400-character request cap, bounded admission, MemoryMax=6G, and MemorySwapMax=0.
• The service warmed up, returned a complete health record, answered two held-out requests in 5.042 and 4.011 seconds, used no cgroup swap, and had zero restarts. Peak memory was 5,371,129,856 bytes.
• Frozen evaluation used CUDA bfloat16. CPU float32 beam search selected a one-character variant on one smoke row and a different later clause on another. This expected numerical-backend sensitivity is documented; byte-identical predictions across backends are not claimed.

2026-07-12T11:41Z - RunPod deleted

• Saved final pod metadata after all remote/local checksum comparisons and local inference checks passed.
• Deleted pod 8fyl89yeeesbyu and confirmed the active-pod list is empty.
• Elapsed pod time was 8,186 seconds (2.2739 hours), for an estimated $1.0005 at $0.44/hour. The first billing-ledger snapshot had posted $0.5734 but had not yet posted the final hour bucket; both values are retained rather than treating the lagging snapshot as final.

2026-07-12T12:12Z - guarded research preview deployed

• Built the production Next application with the production environment, installed the loopback Mi'gmaq endpoint in the web-service unit, and restarted the web service cleanly.
• Published the guarded translator, model registry, versioned artifacts, and the complete 2,247,094,395-byte merged-model archive.
• Verified desktop and mobile layouts, a browser-submitted translation, no page-level mobile overflow, public 400/413 request guards, model archive byte ranges, and the Kuku Yalanji sibling translator.
• The complete archive SHA-256 is 2abe9602ee2b47cee8037b6839ce8faa052f867280591d0d917b2eea4260d591.
• A final RunPod billing query posted $1.020713 for 8,191,142 billed milliseconds.

2026-07-12T12:57Z - dictionary browse duplicates consolidated without deletion

• Browser verification exposed 6,581 headwords duplicated by two historical imports using equivalent word-class taxonomies.
• Exported and checksummed all 6,581 mappings plus 10,754 definitions, 9,053 translations, and 6,713 examples before migration.
• Copied 3 legacy-only definitions, 2,467 definition-linked translations, and 111 legacy-only examples onto canonical managed rows with explicit provenance notes.
• Marked the older word representations as browse-superseded without deleting any of the 13,643 stored word rows.
• The live browse view now contains 7,062 entries and zero duplicate normalized-headword + part-of-speech keys. Distinct homographs such as the noun/verb Winpegijuig records remain separate.
• Verified 28,323 active recordings remain attached; a real dictionary audio request returned HTTP 206.
• Full method, checksums, postconditions, and recovery are in Mi'gmaq Dictionary Browse Consolidation.

11. Current status

Training, one-shot frozen evaluation, artifact transfer, independent checksum verification, local CPU serving, public deployment, dictionary browse consolidation, and pod deletion are complete. The guarded preview is live at mobtranslate.com/labs/migmaq, with the canonical 7,062-entry dictionary and complete downloads linked beside it. The candidate remains research-only: frozen-test chrF++ is 21.43, exact match is 0/742, and material lexical/content failures remain. Unrestricted routing remains blocked pending qualified Mi'gmaq review and materially stronger evidence.