MobTranslate Next 16 Build Incident and Deployment Runbook

Incident date: 2026-07-14 UTC
Application: mobtranslate.com
Repository: /mnt/donto-data/workspace/mobtranslate.com
Web application: apps/web
Framework: Next.js 16.2.9, React 19.2.7, Tailwind CSS 4.3.1, pnpm workspace
Final deployed build: ZS7-SuKDwqjovoz_tBnyE
Status: resolved and live
Scope: build, output tracing, deployment, service readiness, and browser verification; no model weights or linguistic data were changed by this incident response.

1. Executive ruling

The reported build problem was real, but it was not one isolated Next.js compiler bug. Four conditions compounded:

1. The application explicitly forced the legacy Webpack path on Next 16 and retained a custom webpack alias hook that duplicated supported monorepo configuration.
2. Retained .next.* release and rollback trees lived under the source application. Tailwind v4's automatic source detection could traverse them because the pattern was not ignored.
3. Several runtime filesystem boundaries were too broad for Next's output-file tracer. One route trace reached 6,572 files, including 4,547 files from the live .next tree.
4. The shared /dev/sdb data disk was saturated by concurrent repository and research scans. This made both builds and the first post-deploy request appear hung even after the code-level causes were fixed.

The final remedy was to restore Next 16's native Turbopack path, use supported monorepo settings, move retained build trees outside the source repository, scope every runtime filesystem read, build with the complete production environment into an isolated distDir, and verify the exact artifact on a separate port before an atomic swap.

The deployed application now builds and serves correctly. The custom Kuku Yalanji and Mi'gmaq CPU model services remain intentionally unloaded; homepage translation uses the dictionary-guided prompt route.

2. User-visible and operator-visible symptoms

2.1 Original build behavior

• pnpm --filter web build expanded to next build --webpack.
• The first observed production build remained in compilation for about 38 minutes.
• That process read approximately 13 GB from disk but never wrote a valid BUILD_ID.
• It did not produce a deployable release and was terminated without replacing the live tree.
• PostCSS/Tailwind worker file descriptors showed reads inside retained .next.pre-* build output.

2.2 Intermediate failures that were correctly rejected

• A corrected isolated build started without /opt/mobtranslate/web.env failed because DATABASE_URL and BETTER_AUTH_SECRET were absent during prerendering.
• A candidate runtime initially exposed implementation metadata that could reveal a private inference origin. That candidate was not deployed; public endpoint formatting was corrected and retested.
• An initial readiness check targeted port 3000. That port belonged to an unrelated Next process and therefore returned a misleading 200. Caddy and mobtranslate-web.service both correctly use port 3300.

2.3 Brief post-swap 502 period

After the first corrected release was swapped in, Caddy returned 502 for dynamic Next routes while static /docs/ continued to return 200. Inspection established:

• Caddy's upstream was already 127.0.0.1:3300.
• mobtranslate-web.service was active and listening on 3300.
• The Next process had not crashed and had zero restarts.
• /dev/sdb was at 100% utilization with 116-129 ms read latency.
• Host I/O wait reached 56-76%, with load average approximately 35.
• Two broad find commands and two broad rg commands from the investigation were still reading the mounted disk, alongside independent workloads.

Those investigation scans were stopped. The same direct root route then responded in 14 ms and the public root in 100 ms. No Caddy change was required. This was a storage-starvation readiness failure, not a port mismatch or a failed Next build.

3. Evidence-led root cause analysis

3.1 Forced Webpack on a Turbopack-default framework

Next 16 uses Turbopack by default for next dev and next build. The application instead declared:

{
  "dev": "next dev --webpack",
  "build": "next build --webpack"
}

The application also had a custom webpack(config) callback to provide aliases. Code search found no imports using those aliases. The actual shared package is imported as @mobtranslate/ui, which Next can transpile through transpilePackages.

The fix was:

{
  "dev": "next dev",
  "build": "next build"
}

and:

transpilePackages: ['@mobtranslate/ui'],
turbopack: {
  root: path.resolve(__dirname, '../..'),
},
outputFileTracingRoot: path.resolve(__dirname, '../..'),

This follows Next's documented Next 16 default, Turbopack root, package transpilation, and monorepo tracing paths.

3.2 Retained build trees inside Tailwind's discovery boundary

The repository kept rollback trees such as .next.pre-* and .next.release-* beside the live .next directory. They were neither source files nor valid inputs to a new build, but Tailwind v4's automatic source detection could still discover them. A live PostCSS worker had open files under a retained build's static tree.

Two changes close this path:

1. Root .gitignore now includes .next.*/.
2. Retained and failed builds now live under:

/mnt/donto-data/workspace/.build-archive/mobtranslate.com/YYYYMMDD/

The live source tree contains only .next plus the one isolated candidate being built or tested. A candidate is moved to .next at deployment; a superseded live tree is moved out to the archive.

3.3 Over-broad output-file tracing

Next's output-file tracing records files that each server route may need at runtime. Broad dynamic filesystem reads, computed paths spanning repository roots, and an evaluated require('sharp') caused traces to include unrelated project files and prior build output.

Measured before the fix:

The runtime boundaries were corrected as follows:

• Created lib/word-image-cache.ts so cache constants and path construction are separate from image generation.
• Replaced the evaluated sharp require with a static import.
• Limited word-image, WOTD, storage, TTS, dictionary-sync, saved-model-result, and Atlas-method filesystem roots to their actual directories.
• Added /* turbopackIgnore: true */ only at intentional runtime filesystem boundaries using Next's supported placement.
• Replaced the Atlas methods runtime JSON read with a static JSON import.
• Removed the dictionary loader's candidate-root search list and used the known monorepo root.

The corrected output has 207 .nft.json route manifests. The audit found zero traces into a prior .next.* tree and zero unscoped project-root traces.

3.4 Production environment is part of the build contract

This application prerenders routes that initialize authentication and database-backed modules. A production build without the production environment is not representative and must not be deployed.

The accepted build loads /opt/mobtranslate/web.env without printing it. Custom model endpoint variables are then explicitly blanked so prerendering and runtime agree that CPU inference is paused.

Required secrets are never copied into this document, logs, model metadata, or public build output.

3.5 Disk contention was an independent operational cause

The fixed build path was run twice:

The code and route count were effectively unchanged except for the self-hosted analytics guard. The timing increase coincided with sustained 100% /dev/sdb utilization, so it must not be interpreted as a Turbopack regression.

Build monitoring must distinguish a silent but CPU-active TypeScript phase from a true hang. Check process state, CPU, output-tree modification time, and disk latency before terminating a quiet build.

4. Source changes

4.1 Build and workspace configuration

• .gitignore
• apps/web/package.json
• apps/web/next.config.js

4.2 Filesystem and output-tracing boundaries

• apps/web/lib/word-image-cache.ts (new)
• apps/web/lib/word-image.ts
• apps/web/lib/image-queue.ts
• apps/web/lib/storage.ts
• apps/web/lib/dictionary-sync/engine.ts
• apps/web/lib/models/results.ts
• apps/web/app/api/word-image/route.ts
• apps/web/app/api/word-images/route.ts
• apps/web/app/api/wotd/route.ts
• apps/web/app/wotd/[file]/route.ts
• apps/web/app/api/tts/route.ts
• apps/web/app/api/v2/admin/explore/audio/route.ts
• apps/web/app/atlas/methods/page.tsx

4.3 Live UX and self-hosting corrections

• apps/web/app/components/SharedLayout.tsx: full desktop navigation starts at 1360 px; the compact menu remains available below that width; mobile menu capacity covers all current destinations.
• apps/web/app/layout.tsx: Vercel Analytics renders only in a Vercel environment, eliminating the failed /_vercel/insights/script.js request on this host.

This list excludes unrelated pre-existing worktree changes.

5. Canonical build procedure

Run from the repository root. Choose a unique candidate name. Do not build into live .next.

cd /mnt/donto-data/workspace/mobtranslate.com

set -a
source /opt/mobtranslate/web.env
set +a

export MOBTRANSLATE_TRANSLATE_V2_ENDPOINT=
export MOBTRANSLATE_LABS_V2_ENDPOINT=
export MOBTRANSLATE_LABS_MIGMAQ_ENDPOINT=
export NEXT_DIST_DIR=.next.release-YYYYMMDDTHHMMZ
export NEXT_BUILD_CPUS=2

pnpm --filter web build

Mandatory build assertions:

test -s "apps/web/$NEXT_DIST_DIR/BUILD_ID"
test -d "apps/web/$NEXT_DIST_DIR/server"
test -d "apps/web/$NEXT_DIST_DIR/static"
cat "apps/web/$NEXT_DIST_DIR/BUILD_ID"

The worker cap is deliberate on this shared host. It limits peak pressure without disabling strict TypeScript or reducing the static-generation surface.

6. Candidate runtime procedure

Start the exact candidate on an unused port. The same NEXT_DIST_DIR must be present at runtime because Next reads it from next.config.js.

set -a
source /opt/mobtranslate/web.env
set +a

export MOBTRANSLATE_TRANSLATE_V2_ENDPOINT=
export MOBTRANSLATE_LABS_V2_ENDPOINT=
export MOBTRANSLATE_LABS_MIGMAQ_ENDPOINT=
export NEXT_DIST_DIR=.next.release-YYYYMMDDTHHMMZ
export PORT=3310

pnpm --filter web start

Verify real progress, not only process existence:

curl --fail --max-time 10 http://127.0.0.1:3310/api/version
curl --fail --max-time 30 http://127.0.0.1:3310/api/v1/models
curl --fail --max-time 30 http://127.0.0.1:3310/

Also test:

• one homepage prompt translation;
• Kuku Yalanji and Mi'gmaq lab 503 guards;
• a valid registered translate/v2 request returning 503 not_configured;
• the model catalog and OpenAPI;
• Chromium at desktop and mobile widths;
• browser console and page errors.

Stop the candidate process before moving its directory.

7. Atomic deployment and rollback

The production service runs with the default .next directory. Caddy proxies dynamic traffic to 127.0.0.1:3300.

Deployment sequence:

1. Read and record both old and candidate BUILD_ID values.
2. Stop mobtranslate-web.service.
3. Move live .next to the external build archive; do not delete it.
4. Move the verified candidate to .next.
5. Start mobtranslate-web.service.
6. Probe port 3300, then probe the public domain.
7. Keep the rollback tree until a later explicitly verified retention cleanup.

If the candidate move fails, move the rollback tree back to .next before restarting the service. If runtime smoke fails after deployment, inspect service logs and storage pressure before deciding to roll back; a saturated disk can make a valid build appear unavailable.

Final deployment record:

8. Final verification matrix

8.1 Build and tests

8.2 Live service and APIs

8.3 Registry and benchmark evidence

8.4 Browser verification

Real Chromium was used after the final public deployment:

• 1280x720: no horizontal overflow; compact navigation remained available; no application console errors.
• 390x844: no horizontal overflow; menu opened and exposed all nine navigation destinations; no overlaps obscured the translator controls.
• Vercel Analytics no longer attempted to load on the self-hosted deployment.

Screenshots are retained in /opencode/logs/:

mobtranslate-live-final-1280-20260714.png
mobtranslate-live-final-mobile-menu-20260714.png

9. Monitoring and recurrence prevention

9.1 Before every build

• Confirm no previous candidate remains under apps/web/.next.* except the one being built.
• Confirm rollback builds are outside the repository.
• Confirm required environment variables are loaded without printing values.
• Check df -h / /mnt/donto-data, free -h, and iostat -xz 2 3.
• Do not start a production build while broad find, rg, bfs, backup, or corpus scans saturate /dev/sdb.

9.2 During every build

• Record wall time and preserve the full log under /opencode/logs.
• Check process CPU, RSS, state, candidate modification time, and actual output growth.
• Treat an absent BUILD_ID as incomplete regardless of elapsed time.
• Do not infer failure from a quiet TypeScript phase while CPU and output are advancing.

9.3 Before every swap

• Start the exact candidate on an unused port.
• Verify root, translation, model API, guarded 503 routes, and static assets.
• Run a browser session and inspect both errors and console.
• Record old and new build IDs.
• Preserve the old live tree before moving the candidate.

9.4 After every swap

• Probe 127.0.0.1:3300, not a generic or remembered Next port.
• Probe the public domain with a cache-busting query.
• Verify systemctl show mobtranslate-web -p ActiveState -p SubState -p NRestarts.
• Verify custom inference ports remain closed unless an explicit model-serving deployment is underway.
• Rerender the living books whenever build or model status changes.

10. Known non-blocking notices

1. Next 16 warns that middleware.ts is deprecated in favor of the proxy convention. It does not block this build, but the migration should be scheduled as a focused follow-up with auth and routing regression tests.
2. Node reports that its SQLite API is experimental during prerender and runtime initialization. No SQLite failure was observed.
3. The shared data disk remains a cross-application bottleneck. A future CI/build worker or cgroup/I/O scheduling policy would reduce variance, but the current build is reproducible and correct.

11. Official technical references

• Next.js 16 upgrade guide: Turbopack is the default
• Turbopack configuration and root
• Next.js build memory guidance and worker behavior
• transpilePackages configuration
• Custom webpack configuration and compatibility caution
• Next.js output-file tracing and monorepo root
• Tailwind CSS automatic source detection and ignore behavior
• Next source example for turbopackIgnore

12. Audit artifacts

/opencode/logs/mobtranslate-build-turbopack-deploy-20260714T1326Z.log
/opencode/logs/mobtranslate-build-turbopack-deploy-20260714T1326Z.meta
/opencode/logs/mobtranslate-build-turbopack-final-20260714T1347Z.log
/opencode/logs/mobtranslate-build-turbopack-final-20260714T1347Z.meta
/opencode/logs/mobtranslate-live-final-1280-20260714.png
/opencode/logs/mobtranslate-live-final-mobile-menu-20260714.png

The source document is the canonical incident record. The rendered HTML and raw Markdown are published together by render-docs.sh; updating one without rerendering is incomplete.