Mi'gmaq v1 Evaluation and Promotion Report

Candidate: mobtranslate/[email protected]
Evaluation completed: 2026-07-12T11:11:15Z
Decision: preserve and expose only as a guarded noncommercial research preview

1. Evaluation question

The experiment asks whether an NLLB-LoRA candidate trained on 5,798 dictionary example pairs can produce stable English-to-Mi'gmaq drafts on source-entry groups that are wholly absent from training. It does not ask whether automatic metrics can certify Mi'gmaq grammar. They cannot.

The relevant success hierarchy is:

1. data, rights, split, and serialization integrity;
2. nondegenerate generation on every held-out row;
3. measurable improvement over the untuned numerical proxy;
4. lexical and morphosyntactic adequacy under qualified human review;
5. only then, consideration of broader routing.

The candidate passes levels 1-3 and has not passed level 4.

2. Controls against optimistic bias

• The source snapshot and dataset archive were frozen and checksummed before training.
• Connected leakage groups joined shared entry IDs, normalized headwords, English strings, Mi'gmaq strings, and bilingual pairs.
• No connected group crosses train, validation, or test.
• The full trainer was invoked without a test file; the model manifest records only train and validation.
• Checkpoint 1440 was selected by validation chrF++ under the predeclared schedule.
• Three decoding configurations were compared on the same ordered 686 validation IDs.
• The selected decoding record states frozen_test_accessed: false and is checksummed.
• A separate runner created an exclusive access ledger before opening the 742-row test.
• The test was generated once with the frozen model and decoding configuration.
• Evaluation row count, unique IDs, and split == test were asserted before analysis.

3. Pipeline integrity

The first 128-row plumbing gate exposed a real PEFT/NLLB merge defect: the saved merged model did not preserve the runtime decoder embedding alias, causing 92/128 empty outputs after reload. The failed verdict blocked full training.

The correction made the runtime decoder embedding canonical, kept the independently trained output head untied, and materialized model.shared.weight, encoder embedding, and decoder embedding separately during serialization. A resumed gate reached 128/128 exact and reproduced predictions after merge/reload. The full candidate uses that corrected path.

The completed run contains three recovery checkpoints, the selected adapter, a standalone merged model, TensorBoard events, manifests, audits, full logs, and a five-second resource trace. The post-close checksum inventory passes. The original inventory's single full.log mismatch is preserved and explained: its final timestamp was appended after hashing. The script order is corrected for future runs.

4. Training curve

The selection metric improves monotonically, although gains flatten after step 1080. BLEU is not monotonic. No post hoc epoch or metric change was made.

5. Decoding selection

All three produced 686 non-empty, non-copy outputs. The unconstrained run's 45 repeated-trigram outputs demonstrate a real degeneration tendency hidden by aggregate length ratio.

6. Final held-out results

Validation chrF++ is 21.0434 with a 1,000-resample percentile interval of 20.4083-21.6518. The paired gain over the base proxy is +9.7344 chrF++, interval +9.1063 to +10.3009.

Frozen-test chrF++ is 21.4298, interval 20.8128-22.0332. SacreBLEU is 1.7795, interval 0.9308-2.6029. Exact match is 0/742.

Test diagnostics:

• 742 rows, all source-headword groups unseen in training;
• 734 unique outputs, maximum frequency 3;
• 0 empty outputs and 0 English source copies;
• 0 outputs exactly equal to any training target;
• 6 outputs with a repeated bigram and 0 with a repeated trigram;
• mean character-length ratio 0.9063; 5th-95th percentile 0.6129-1.3058;
• terminal-punctuation agreement 98.38%;
• apostrophe-presence agreement 89.76%.

Length slices do not reveal a simple collapse: chrF++ is 22.09 for 46 short sources, 21.09 for 374 medium sources, and 21.66 for 322 long sources. This does not imply long-clause adequacy; individual long examples show major content omission.

7. Linguistic interpretation

The outputs often have source-like orthographic shape and frequent Mi'gmaq character sequences. That explains part of the chrF++ gain. It does not establish correct lexical heads, participant indexing, number, animacy, transitivity, negation, or discourse appropriateness.

Reference comparison exposes at least four recurring risks:

1. Lexical replacement: held-out headword material is replaced by frequent, more generic strings.
2. Content omission: longer English clauses can be compressed to a short output that omits predicates and participants.
3. Participant/possessive loss: kinship, body-part, and child/parent relations are often absent from the output relative to the source reference.
4. Degeneration without constraints: repeated trigrams rise sharply when repetition controls are removed.

The dataset's English side also contains source typos and editorial conventions such as he/she, parenthetical clarifications, and occasional malformed forms. These are preserved for provenance but may shape the model's behavior.

Because each item has one source reference, a low sentence score can include legitimate variation, and a higher score can still be morphologically wrong. The report therefore does not label any generated sentence “correct” without speaker review.

8. Resource record

The A40 training window contains 401 five-second monitor samples, 359 with allocated VRAM. During active GPU samples, mean utilization was 30.06%, mean VRAM was 8,172 MiB, and mean power was 144.46 W. Peaks were 56% utilization, 10,011 MiB VRAM, 172.49 W, 59 C, and 5,702.5 MiB Python RSS. The trainer completed in 27.1 minutes. Gate diagnosis, final evaluation, bootstrap analysis, checksumming, and artifact transfer extended total elapsed pod time to 8,186 seconds (2.2739 hours). RunPod's final ledger posted $1.020713 for 8,191,142 billed milliseconds. The pod was deleted after local verification.

8.1 Serving-backend check

Frozen validation and test generation used CUDA bfloat16. The guarded live service uses CPU float32 because the production host has no GPU. It loaded the same merged-weight hash and tokenizer with the same four-beam, no-repeat-3, repetition-1.10, length-1.0 configuration. Health and two held-out requests passed in 5.042 and 4.011 seconds, with no cgroup swap or restarts and a 5,371,129,856-byte peak.

Beam search is sensitive to near-tied floating-point scores. One CPU smoke output differed by one character from the archived GPU prediction, while another selected a different later clause. The frozen CUDA scores therefore describe the archived evaluation backend, not a promise of byte-identical CPU strings. This does not change the promotion ruling: both backends produce unreviewed research drafts.

9. Decision

The candidate is reproducible and nondegenerate, and it substantially improves over the untuned proxy. Its absolute held-out quality remains low and exact match is zero. The observed lexical and content failures are incompatible with authoritative use.

The permitted release is therefore limited to:

• a non-indexed or clearly labelled research preview;
• an always-visible “not speaker-reviewed” warning;
• source and CC BY-NC attribution;
• no commercial use;
• no claim of dialect-general or community-approved output;
• a direct path to the source dictionary and model documentation;
• bounded inference, input limits, and clean failure states.

Unrestricted public translation routing remains blocked. The next scientific step is structured review by qualified Mi'gmaq language workers, followed by an error-driven data and training revision rather than score-only tuning.