Mi'gmaq Online Recording Integration: Archive, Mapping, and Handoff

Last updated: 2026-07-12 05:19 UTC
Status: complete, independently verified, deployed, and live
Public dictionary: MobTranslate Mi'gmaq dictionary
Source: Mi'gmaq/Mi'kmaq Online Talking Dictionary
Source word index: All Words

1. Purpose and non-negotiable constraints

This integration archives the publicly available word and example-sentence recordings from the Mi'gmaq/Mi'kmaq Online Talking Dictionary, maps them to the MobTranslate Mi'gmaq dictionary, and serves local, checksum-verified copies through MobTranslate's existing recording infrastructure.

The implementation follows five constraints:

1. No source data is silently discarded. Raw HTML, every downloaded MP3, parser output, errors, mapping plans, checksums, import events, and verification reports are retained.
2. No historical MobTranslate row is deleted or collapsed. The existing Mi'gmaq dictionary contains two historical imports for most headwords. Both remain intact.
3. Only exact orthographic or guarded normalized headword matches are automatic. Case is preserved because it can distinguish a place name from a common word. There is no fuzzy auto-linking.
4. Imported recordings are explicitly attributed. The original entry, collection, speaker code supplied by the source, and license are visible in the dictionary UI.
5. The noncommercial license boundary stays machine-readable and visible. These source-derived assets must not be represented as unrestricted MobTranslate data.

2. Rights and attribution

The source site states that its content is available under the Creative Commons Attribution-NonCommercial 4.0 International license: CC BY-NC 4.0.

The database records:

• source name and source URL;
• attribution text;
• license name and URL;
• commercial_use_allowed = false;
• the date on which the terms were checked;
• the original entry and audio URLs for every recording.

MobTranslate's player links back to the source entry and displays the CC BY-NC license. The local copy exists for durability and same-origin playback; it does not erase or replace the source attribution.

3. Source audit

The source index contained 6,974 unique entry pages when captured on 2026-07-12. A typical entry contains:

• one Mi'gmaq headword;
• part of speech;
• one translation and one or more meanings;
• a pronunciation guide;
• alternate grammatical forms;
• normally three word recordings identified by source speaker codes;
• an example sentence, English translation, and sentence recording.

The source is static HTML. The importer uses a real HTML parser, not regular-expression scraping. It resolves every media link against the entry URL and assigns stable source-entry and source-recording identities.

The completed structural audit found 28,323 distinct audio links on entry pages and classified 28,330 recording references. The reference count is seven higher because a small number of source URLs are intentionally reused in multiple source positions. These references resolve to 28,304 unique audio URLs. The audit now reports zero unclassified audio links.

All 28,304 unique URLs were downloaded successfully, producing 335,103,692 bytes. Every retained file passed an MP3 signature check and was independently decoded with ffprobe. The decoded corpus contains 40,679,974 milliseconds (approximately 11.3 hours) of audio; 28,303 files are usable speech assets. One 864-byte source file decodes to only 0.344 milliseconds and is an apparent source-side placeholder. It is retained, checksummed, and marked unusable_too_short in the archive, but its references are deliberately excluded from production playback.

The source-entry quality audit also preserves structural exceptions rather than silently normalizing them: three entries have no word recording, one example has no recording, and a small minority of entries contain two or three example sections. Speaker codes remain exactly as supplied by the source and are not expanded into unverified speaker identities or consent claims.

Source-supplied speaker-code frequencies across all recording references are retained as follows. These are opaque provenance labels, not asserted names:

The two punctuation-only placeholders remain verbatim in provenance but are presented to readers as an unspecified source speaker rather than as a person identifier.

4. Durable archive layout

Canonical research root:

/mnt/donto-data/donto-resources/research/migmaq-online-talking-dictionary-2026-07-12/

Layout:

raw/index/                 captured index, home page, and representative entry
raw/entries/               one raw HTML file per source entry, keyed by URL SHA-256
raw/audio/                 source MP3s in content-address-like prefix directories
manifests/index.json       source inventory and selected-run scope
manifests/entries.jsonl    parsed linguistic and recording payload per entry
manifests/audio-references.jsonl
manifests/audio-downloads.jsonl
manifests/*-errors.jsonl   explicit failures; a clean import requires zero rows
manifests/import-plan.jsonl
reports/                   mapping, import, and verification summaries
history/                   prior raw/manifest/report versions retained on refresh
logs/                      long-run operator logs
SHA256SUMS                 verified digests for canonical evidence artifacts

Individual audio content hashes are recorded in manifests/audio-downloads.jsonl; individual raw-entry HTML hashes are recorded in manifests/entries.jsonl. SHA256SUMS therefore authenticates the compact manifests and final evidence without redundantly listing more than 35,000 payload files.

Production playback files are hard-linked, when the filesystem permits, into:

/mnt/donto-data/mobtranslate-storage/recordings/imports/mikmaq-online/<prefix>/<content-sha256>.mp3

This avoids storing duplicate bytes on the same data disk while keeping the research archive and production namespace independently intelligible. A cross-device deployment falls back to an exclusive copy.

5. Database provenance model

Migration:

apps/web/supabase/migrations/20260712010000_recording_source_provenance.sql

Tables:

The imported audio itself remains in the existing recordings table, so all normal storage routes, players, status rules, and sentence/word relationships continue to work.

6. Mapping policy

6.1 Text normalization

The first matching key preserves source case, performs Unicode compatibility composition, normalizes typographic apostrophes to ASCII apostrophes, and collapses whitespace. This prevents the common word winpegijuig from being collapsed into the proper place name Winpegijuig.

A lowercased, combining-mark-insensitive key is used only as a fallback when the source contains exactly one orthographic form for that normalized key. Part of speech is preferred when it is available. Mi'gmaq letters and apostrophes remain meaningful content, and no semantic or fuzzy match is accepted automatically.

6.2 Existing duplicates

When multiple MobTranslate rows have the same eligible exact headword, the canonical target is selected dynamically by:

1. verified status;
2. total attached definitions, translations, and examples;
3. count of populated linguistic enrichment fields;
4. most recent creation time;
5. UUID as a deterministic final tie-break.

The plan records every candidate and every score. The public word page applies the same content-completeness principle so the richer historical row is displayed. Source recordings attached to the canonical row are also discoverable from its exact duplicate.

6.3 Missing words

If a source headword has no exact MobTranslate match and the normalized source headword is unique, the importer creates an unverified, source-attributed dictionary row. Meanings, translation, and example sentence remain attributed to the original source. The word-class ID is inferred from the most frequent existing word_type to word_class_id relationship in the live Mi'gmaq data; there is no hand-maintained part-of-speech lookup table.

If multiple source entries collapse to the same normalized headword, automatic creation stops and the entries are marked review_required.

6.4 Example sentences

Sentence recordings attach only to an exact, whitespace-normalized, case-sensitive orthographic example under the selected word. If that example is absent, a source-attributed usage example is added. No case-folded or semantically similar sentence is auto-linked.

7. Reproducible commands

Archive commands run from apps/web:

pnpm exec tsx scripts/archive-mikmaq-online.ts inventory
pnpm exec tsx scripts/archive-mikmaq-online.ts crawl --concurrency 8 --requests-per-second 10
pnpm exec tsx scripts/archive-mikmaq-online.ts download --concurrency 8 --requests-per-second 10
pnpm exec tsx scripts/archive-mikmaq-online.ts probe --concurrency 8
pnpm exec tsx scripts/archive-mikmaq-online.ts audit

all runs those phases in order. Existing raw files are resumed. --refresh preserves changed prior versions under history/ before promoting new content.

Database commands require /opt/mobtranslate/web.env:

set -a
source /opt/mobtranslate/web.env
set +a
pnpm exec tsx scripts/import-mikmaq-online-recordings.ts plan
pnpm exec tsx scripts/import-mikmaq-online-recordings.ts import
pnpm exec tsx scripts/import-mikmaq-online-recordings.ts verify
pnpm --filter web build

The full importer refuses partial archives, any entry errors, any audio errors, or any missing audio manifest rows. --allow-partial exists only for an explicitly labelled integration pilot.

8. Backup and pilot evidence

Pre-mutation database backup:

/mnt/donto-data/backups/mobtranslate-pre-mikmaq-audio-20260712T023100Z.dump
SHA-256 3d4d7a9f346ce29d2de72d63bf30c5669f5c9089b9609ec9c84681bef47d7e0f

The custom-format dump passed pg_restore --list before the migration was applied.

Immediately before the full import, a second custom-format checkpoint was created and independently checked:

/mnt/donto-data/backups/mobtranslate-pre-full-mikmaq-audio-20260712T033328Z.dump
SHA-256 f5b6a14632775044f8806b6307ea8c0aadaaf794ea92f0f5c27b236c29520178

Its checksum and pg_restore --list both passed before the full run began.

After the full import and first successful verification, a third checkpoint was taken before correcting one ambiguous source-speaker attribution:

/mnt/donto-data/backups/mobtranslate-post-mikmaq-import-pre-speaker-ambiguity-20260712T042051Z.dump
SHA-256 b46f147c6a5e7eb162261dd9498a2b100b4c854a29ab043d18c6360c62ad97ff

This custom-format dump also passed its checksum and pg_restore --list gate.

Pilot import run:

run_id: 4dd76b1a-9ff7-4aa7-8966-9f2c73b33d39
entries processed: 3
new source-attributed words: 3
source examples added: 3
word recordings: 9
sentence recordings: 3
errors: 0

Pilot verification checked all 12 production files and found:

• 0 checksum mismatches;
• 0 missing files;
• 0 inactive imported recordings;
• 0 recordings without words;
• 0 sentence recordings without examples.

9. Full-run status

The complete entry-page crawl finished with 6,974/6,974 pages parsed and zero fetch or parser failures. A first structural audit exposed 244 example recordings in repeated example sections that a first-section-only parser would have omitted. The parser was corrected, a regression fixture was added, and all local pages were reparsed. The second audit reconciled every discovered audio link with zero unclassified links.

Current archive and mapping counts:

The archive and independent media probe are complete. The production database import ran from these fixed manifests from 2026-07-12 03:46 to 04:07 UTC:

run_id: c06f7d6d-668a-46e8-8bf7-91e22d5012db
status: completed
entries processed: 6,974
existing-word mappings: 6,496
new words created in the full run: 478
new words created across pilot + full run: 481
source examples added in the full run: 835
recordings created in the full run: 28,310
recordings reused by stable identity: 14
usable recording references processed: 28,324
unique stable recording identities: 28,322
unusable source-placeholder references skipped: 6
review-required mappings: 0
entry errors: 0

The 28,324 usable references plus six excluded placeholder references reconcile exactly to the 28,330 classified source references. Two usable references are repeated source identities, leaving 28,322 unique database recordings. The 14 reuse events comprise 12 pilot identities plus those two repeated references; this is expected idempotent behavior, not missing data.

The first full verification run (f5748ee4-ab45-4f9a-ba4e-630590e72053) passed all then-defined gates: 6,974 entry mappings, 28,322 expected and actual recording identities, 28,303 distinct production files rehashed, and zero file, checksum, metadata, activity, word-target, sentence-target, or identity violations.

A subsequent source-label audit found that the tewa'lutewei page assigns the dmm, ewm, and jnw labels to one identical URL and stable audio identity. Treating the first code as the speaker would be unjustified. The raw labels remain in the manifests and source payload; the database row now has no singular speaker_code, retains all three codes in metadata.sourceSpeakerCodes, and carries speakerAttributionStatus = ambiguous_source_labels. An append-only speaker_ambiguity_corrected event records the ruling. The importer and verifier now derive this condition from source identity groups, so future runs preserve and enforce it without a hand-maintained exception.

The strengthened final verification run (0db87a16-3d75-4fbf-a16e-8b1230e2fc20) passed after the correction. It rechecked all 6,974 entry mappings, all 28,322 expected and actual recording identities, all 28,303 production files and checksums, all required audio metadata and target relationships, and the source-speaker ambiguity rule. Every violation count was zero.

Final live database state:

The final production-environment Next.js build passed TypeScript and generated all 350 static pages while retaining the dynamic dictionary, recording, and storage routes. The terminal shared-workspace build ID is 6seAOMiyHQVtnO8c3wtw7. mobtranslate-web.service last restarted at 2026-07-12 05:18 UTC under PID 3818312 and reported ready in 267 ms; post-restart dictionary and API checks passed.

Live browser verification covered a newly created source word (guntewi'ganmit), two separately recorded example sections (qasqamgeg), the ambiguous source-speaker case (tewa'lutewei), and the case-separated winpegijuig / Winpegijuig entries. Chromium requested the local production MP3, playback began, source and license links were visible, and desktop 1440x900 plus mobile 390x844 layouts had no error overlay, browser errors, overlap, or horizontal overflow. Light and dark themes both rendered correctly. The only console notice was the pre-existing self-hosted Vercel Web Analytics script warning; it does not affect dictionary or audio behavior.

Machine-readable deployment evidence is in reports/live-deployment-verification.json; screenshots are retained beside it. The public entry point is mobtranslate.com/dictionaries/migmaq.

10. Recovery and idempotence

• Raw source files make crawl restarts cheap; completed files are parsed locally rather than fetched again.
• Downloads are keyed by source URL and verified by SHA-256.
• Production audio paths are keyed by content SHA-256 and are never overwritten with different bytes.
• Recording client_id values include the source recording identity and content checksum, making reruns idempotent.
• Import-entry upserts preserve first_seen_at; every run and event remains append-only in the run ledger.
• A failed entry is recorded and the run is marked failed. A later rerun can repair it without deleting successful rows.
• Verification recomputes production-file checksums and checks all target relationships.

11. Operator checklist

1. Confirm both error manifests are empty.
2. Confirm parsed entries equal the source index count.
3. Confirm every unique referenced audio URL has one download-manifest row.
4. Review import-plan-summary.json, especially source collisions and review-required entries.
5. Run the additive importer.
6. Run verification and require zero violations.
7. Build and restart mobtranslate-web.service.
8. Test a source word page, a source sentence recording, an exact historical duplicate, a newly created word, and a mobile viewport.
9. Confirm the browser requests local /api/storage/recordings/... URLs rather than hotlinking source MP3s.
10. Re-render this living book with render-docs.sh after every material run.

12. Known boundary

The imported collection is noncommercially licensed. Any future commercial deployment, unrestricted aggregate dataset download, or model-training release that includes these source-derived audio files requires a separate rights review and potentially direct permission from the source project. MobTranslate must keep this subset distinguishable from material under more permissive terms.