# MobTranslate Next 16 Build Incident and Deployment Runbook

**Incident date:** 2026-07-14 UTC  
**Application:** `mobtranslate.com`  
**Repository:** `/mnt/donto-data/workspace/mobtranslate.com`  
**Web application:** `apps/web`  
**Framework:** Next.js 16.2.9, React 19.2.7, Tailwind CSS 4.3.1, pnpm workspace  
**Final deployed build:** `ZS7-SuKDwqjovoz_tBnyE`  
**Status:** resolved and live  
**Scope:** build, output tracing, deployment, service readiness, and browser verification; no model weights or
linguistic data were changed by this incident response.

## 1. Executive ruling

The reported build problem was real, but it was not one isolated Next.js compiler bug. Four conditions compounded:

1. The application explicitly forced the legacy Webpack path on Next 16 and retained a custom webpack alias hook
   that duplicated supported monorepo configuration.
2. Retained `.next.*` release and rollback trees lived under the source application. Tailwind v4's automatic source
   detection could traverse them because the pattern was not ignored.
3. Several runtime filesystem boundaries were too broad for Next's output-file tracer. One route trace reached 6,572
   files, including 4,547 files from the live `.next` tree.
4. The shared `/dev/sdb` data disk was saturated by concurrent repository and research scans. This made both builds
   and the first post-deploy request appear hung even after the code-level causes were fixed.

The final remedy was to restore Next 16's native Turbopack path, use supported monorepo settings, move retained build
trees outside the source repository, scope every runtime filesystem read, build with the complete production
environment into an isolated `distDir`, and verify the exact artifact on a separate port before an atomic swap.

The deployed application now builds and serves correctly. The custom Kuku Yalanji and Mi'gmaq CPU model services
remain intentionally unloaded; homepage translation uses the dictionary-guided prompt route.

## 2. User-visible and operator-visible symptoms

### 2.1 Original build behavior

- `pnpm --filter web build` expanded to `next build --webpack`.
- The first observed production build remained in compilation for about 38 minutes.
- That process read approximately 13 GB from disk but never wrote a valid `BUILD_ID`.
- It did not produce a deployable release and was terminated without replacing the live tree.
- PostCSS/Tailwind worker file descriptors showed reads inside retained `.next.pre-*` build output.

### 2.2 Intermediate failures that were correctly rejected

- A corrected isolated build started without `/opt/mobtranslate/web.env` failed because `DATABASE_URL` and
  `BETTER_AUTH_SECRET` were absent during prerendering.
- A candidate runtime initially exposed implementation metadata that could reveal a private inference origin. That
  candidate was not deployed; public endpoint formatting was corrected and retested.
- An initial readiness check targeted port 3000. That port belonged to an unrelated Next process and therefore
  returned a misleading 200. Caddy and `mobtranslate-web.service` both correctly use port 3300.

### 2.3 Brief post-swap 502 period

After the first corrected release was swapped in, Caddy returned 502 for dynamic Next routes while static `/docs/`
continued to return 200. Inspection established:

- Caddy's upstream was already `127.0.0.1:3300`.
- `mobtranslate-web.service` was active and listening on 3300.
- The Next process had not crashed and had zero restarts.
- `/dev/sdb` was at 100% utilization with 116-129 ms read latency.
- Host I/O wait reached 56-76%, with load average approximately 35.
- Two broad `find` commands and two broad `rg` commands from the investigation were still reading the mounted disk,
  alongside independent workloads.

Those investigation scans were stopped. The same direct root route then responded in 14 ms and the public root in
100 ms. No Caddy change was required. This was a storage-starvation readiness failure, not a port mismatch or a
failed Next build.

## 3. Evidence-led root cause analysis

### 3.1 Forced Webpack on a Turbopack-default framework

Next 16 uses Turbopack by default for `next dev` and `next build`. The application instead declared:

```json
{
  "dev": "next dev --webpack",
  "build": "next build --webpack"
}
```

The application also had a custom `webpack(config)` callback to provide aliases. Code search found no imports using
those aliases. The actual shared package is imported as `@mobtranslate/ui`, which Next can transpile through
`transpilePackages`.

The fix was:

```json
{
  "dev": "next dev",
  "build": "next build"
}
```

and:

```js
transpilePackages: ['@mobtranslate/ui'],
turbopack: {
  root: path.resolve(__dirname, '../..'),
},
outputFileTracingRoot: path.resolve(__dirname, '../..'),
```

This follows Next's documented Next 16 default, Turbopack root, package transpilation, and monorepo tracing paths.

### 3.2 Retained build trees inside Tailwind's discovery boundary

The repository kept rollback trees such as `.next.pre-*` and `.next.release-*` beside the live `.next` directory.
They were neither source files nor valid inputs to a new build, but Tailwind v4's automatic source detection could
still discover them. A live PostCSS worker had open files under a retained build's `static` tree.

Two changes close this path:

1. Root `.gitignore` now includes `.next.*/`.
2. Retained and failed builds now live under:

```text
/mnt/donto-data/workspace/.build-archive/mobtranslate.com/YYYYMMDD/
```

The live source tree contains only `.next` plus the one isolated candidate being built or tested. A candidate is
moved to `.next` at deployment; a superseded live tree is moved out to the archive.

### 3.3 Over-broad output-file tracing

Next's output-file tracing records files that each server route may need at runtime. Broad dynamic filesystem reads,
computed paths spanning repository roots, and an evaluated `require('sharp')` caused traces to include unrelated
project files and prior build output.

Measured before the fix:

| Trace observation | Count |
|---|---:|
| Files in the sampled image-route trace | 6,572 |
| Files from the live `.next` tree | 4,547 |

The runtime boundaries were corrected as follows:

- Created `lib/word-image-cache.ts` so cache constants and path construction are separate from image generation.
- Replaced the evaluated `sharp` require with a static import.
- Limited word-image, WOTD, storage, TTS, dictionary-sync, saved-model-result, and Atlas-method filesystem roots to
  their actual directories.
- Added `/* turbopackIgnore: true */` only at intentional runtime filesystem boundaries using Next's supported
  placement.
- Replaced the Atlas methods runtime JSON read with a static JSON import.
- Removed the dictionary loader's candidate-root search list and used the known monorepo root.

The corrected output has 207 `.nft.json` route manifests. The audit found zero traces into a prior `.next.*` tree
and zero unscoped project-root traces.

### 3.4 Production environment is part of the build contract

This application prerenders routes that initialize authentication and database-backed modules. A production build
without the production environment is not representative and must not be deployed.

The accepted build loads `/opt/mobtranslate/web.env` without printing it. Custom model endpoint variables are then
explicitly blanked so prerendering and runtime agree that CPU inference is paused.

Required secrets are never copied into this document, logs, model metadata, or public build output.

### 3.5 Disk contention was an independent operational cause

The fixed build path was run twice:

| Build | Compile | TypeScript | Static pages | Wall time | Result |
|---|---:|---:|---:|---:|---|
| Deployment candidate | 37.1 s | 77 s | 25.0 s | 173 s | passed |
| Final analytics-clean build under contention | 74 s | 3.1 min | 55 s | 402 s | passed |

The code and route count were effectively unchanged except for the self-hosted analytics guard. The timing increase
coincided with sustained 100% `/dev/sdb` utilization, so it must not be interpreted as a Turbopack regression.

Build monitoring must distinguish a silent but CPU-active TypeScript phase from a true hang. Check process state,
CPU, output-tree modification time, and disk latency before terminating a quiet build.

## 4. Source changes

### 4.1 Build and workspace configuration

- `.gitignore`
- `apps/web/package.json`
- `apps/web/next.config.js`

### 4.2 Filesystem and output-tracing boundaries

- `apps/web/lib/word-image-cache.ts` (new)
- `apps/web/lib/word-image.ts`
- `apps/web/lib/image-queue.ts`
- `apps/web/lib/storage.ts`
- `apps/web/lib/dictionary-sync/engine.ts`
- `apps/web/lib/models/results.ts`
- `apps/web/app/api/word-image/route.ts`
- `apps/web/app/api/word-images/route.ts`
- `apps/web/app/api/wotd/route.ts`
- `apps/web/app/wotd/[file]/route.ts`
- `apps/web/app/api/tts/route.ts`
- `apps/web/app/api/v2/admin/explore/audio/route.ts`
- `apps/web/app/atlas/methods/page.tsx`

### 4.3 Live UX and self-hosting corrections

- `apps/web/app/components/SharedLayout.tsx`: full desktop navigation starts at 1360 px; the compact menu remains
  available below that width; mobile menu capacity covers all current destinations.
- `apps/web/app/layout.tsx`: Vercel Analytics renders only in a Vercel environment, eliminating the failed
  `/_vercel/insights/script.js` request on this host.

This list excludes unrelated pre-existing worktree changes.

## 5. Canonical build procedure

Run from the repository root. Choose a unique candidate name. Do not build into live `.next`.

```bash
cd /mnt/donto-data/workspace/mobtranslate.com

set -a
source /opt/mobtranslate/web.env
set +a

export MOBTRANSLATE_TRANSLATE_V2_ENDPOINT=
export MOBTRANSLATE_LABS_V2_ENDPOINT=
export MOBTRANSLATE_LABS_MIGMAQ_ENDPOINT=
export NEXT_DIST_DIR=.next.release-YYYYMMDDTHHMMZ
export NEXT_BUILD_CPUS=2

pnpm --filter web build
```

Mandatory build assertions:

```bash
test -s "apps/web/$NEXT_DIST_DIR/BUILD_ID"
test -d "apps/web/$NEXT_DIST_DIR/server"
test -d "apps/web/$NEXT_DIST_DIR/static"
cat "apps/web/$NEXT_DIST_DIR/BUILD_ID"
```

The worker cap is deliberate on this shared host. It limits peak pressure without disabling strict TypeScript or
reducing the static-generation surface.

## 6. Candidate runtime procedure

Start the exact candidate on an unused port. The same `NEXT_DIST_DIR` must be present at runtime because Next reads
it from `next.config.js`.

```bash
set -a
source /opt/mobtranslate/web.env
set +a

export MOBTRANSLATE_TRANSLATE_V2_ENDPOINT=
export MOBTRANSLATE_LABS_V2_ENDPOINT=
export MOBTRANSLATE_LABS_MIGMAQ_ENDPOINT=
export NEXT_DIST_DIR=.next.release-YYYYMMDDTHHMMZ
export PORT=3310

pnpm --filter web start
```

Verify real progress, not only process existence:

```bash
curl --fail --max-time 10 http://127.0.0.1:3310/api/version
curl --fail --max-time 30 http://127.0.0.1:3310/api/v1/models
curl --fail --max-time 30 http://127.0.0.1:3310/
```

Also test:

- one homepage prompt translation;
- Kuku Yalanji and Mi'gmaq lab 503 guards;
- a valid registered `translate/v2` request returning 503 `not_configured`;
- the model catalog and OpenAPI;
- Chromium at desktop and mobile widths;
- browser console and page errors.

Stop the candidate process before moving its directory.

## 7. Atomic deployment and rollback

The production service runs with the default `.next` directory. Caddy proxies dynamic traffic to
`127.0.0.1:3300`.

Deployment sequence:

1. Read and record both old and candidate `BUILD_ID` values.
2. Stop `mobtranslate-web.service`.
3. Move live `.next` to the external build archive; do not delete it.
4. Move the verified candidate to `.next`.
5. Start `mobtranslate-web.service`.
6. Probe **port 3300**, then probe the public domain.
7. Keep the rollback tree until a later explicitly verified retention cleanup.

If the candidate move fails, move the rollback tree back to `.next` before restarting the service. If runtime smoke
fails after deployment, inspect service logs and storage pressure before deciding to roll back; a saturated disk can
make a valid build appear unavailable.

Final deployment record:

| Field | Value |
|---|---|
| Deployed at | 2026-07-14 13:55:44 UTC |
| New build ID | `ZS7-SuKDwqjovoz_tBnyE` |
| Replaced build ID | `LUCL6-Xz-yvnyYPvmgJJa` |
| Rollback tree | `.build-archive/mobtranslate.com/20260714/.next.rollback-LUCL6-Xz-yvnyYPvmgJJa-20260714T135544Z` |
| Production upstream | `127.0.0.1:3300` |
| Service readiness | 299 ms; zero restarts |

## 8. Final verification matrix

### 8.1 Build and tests

| Gate | Result |
|---|---|
| Next version and bundler | Next 16.2.9, Turbopack |
| Compile | passed in 74 s |
| Strict TypeScript | passed in 3.1 min under disk contention |
| Static generation | 361/361 in 55 s |
| Total build wall time | 402 s |
| Focused Vitest files | 4 passed |
| Focused tests | 13 passed |
| Route trace manifests | 207 |
| Unscoped prior-build traces | 0 |

### 8.2 Live service and APIs

| Gate | Result |
|---|---|
| `mobtranslate-web.service` | active, zero restarts |
| `kuku-v2-infer.service` | inactive and disabled |
| `mobtranslate-migmaq-v1-infer.service` | inactive and disabled |
| Ports 7955 and 8765 | no listeners |
| `GET /` | 200 |
| Homepage dictionary-guided translation | 200 with translation and gloss |
| `GET /api/v1/models` | 200; two model families |
| Registered model releases | 21 |
| `POST /api/labs/v2` | intentional 503 |
| `POST /api/labs/migmaq` | intentional 503 |
| valid `POST /api/translate/v2` | intentional 503 `not_configured` |
| `GET /docs/` | 200 |

### 8.3 Registry and benchmark evidence

| Measure | Count |
|---|---:|
| Registry artifacts | 147 |
| Available artifact entries | 129 |
| Unique available URLs | 125 |
| Current v21.2 release artifacts | 33 |
| Current evaluation artifacts | 16 |
| Frozen benchmark rows per checkpoint | 340 |
| Three-checkpoint predictions | 1,020 |
| A40 run-to-run exact parity | 1,020/1,020 |
| A40-to-frozen-CPU exact parity | 1,020/1,020 |

### 8.4 Browser verification

Real Chromium was used after the final public deployment:

- 1280x720: no horizontal overflow; compact navigation remained available; no application console errors.
- 390x844: no horizontal overflow; menu opened and exposed all nine navigation destinations; no overlaps obscured
  the translator controls.
- Vercel Analytics no longer attempted to load on the self-hosted deployment.

Screenshots are retained in `/opencode/logs/`:

```text
mobtranslate-live-final-1280-20260714.png
mobtranslate-live-final-mobile-menu-20260714.png
```

## 9. Monitoring and recurrence prevention

### 9.1 Before every build

- Confirm no previous candidate remains under `apps/web/.next.*` except the one being built.
- Confirm rollback builds are outside the repository.
- Confirm required environment variables are loaded without printing values.
- Check `df -h / /mnt/donto-data`, `free -h`, and `iostat -xz 2 3`.
- Do not start a production build while broad `find`, `rg`, `bfs`, backup, or corpus scans saturate `/dev/sdb`.

### 9.2 During every build

- Record wall time and preserve the full log under `/opencode/logs`.
- Check process CPU, RSS, state, candidate modification time, and actual output growth.
- Treat an absent `BUILD_ID` as incomplete regardless of elapsed time.
- Do not infer failure from a quiet TypeScript phase while CPU and output are advancing.

### 9.3 Before every swap

- Start the exact candidate on an unused port.
- Verify root, translation, model API, guarded 503 routes, and static assets.
- Run a browser session and inspect both `errors` and `console`.
- Record old and new build IDs.
- Preserve the old live tree before moving the candidate.

### 9.4 After every swap

- Probe `127.0.0.1:3300`, not a generic or remembered Next port.
- Probe the public domain with a cache-busting query.
- Verify `systemctl show mobtranslate-web -p ActiveState -p SubState -p NRestarts`.
- Verify custom inference ports remain closed unless an explicit model-serving deployment is underway.
- Rerender the living books whenever build or model status changes.

## 10. Known non-blocking notices

1. Next 16 warns that `middleware.ts` is deprecated in favor of the `proxy` convention. It does not block this build,
   but the migration should be scheduled as a focused follow-up with auth and routing regression tests.
2. Node reports that its SQLite API is experimental during prerender and runtime initialization. No SQLite failure was
   observed.
3. The shared data disk remains a cross-application bottleneck. A future CI/build worker or cgroup/I/O scheduling
   policy would reduce variance, but the current build is reproducible and correct.

## 11. Official technical references

- [Next.js 16 upgrade guide: Turbopack is the default](https://nextjs.org/docs/app/guides/upgrading/version-16)
- [Turbopack configuration and root](https://nextjs.org/docs/app/api-reference/turbopack)
- [Next.js build memory guidance and worker behavior](https://nextjs.org/docs/app/guides/memory-usage)
- [`transpilePackages` configuration](https://nextjs.org/docs/app/api-reference/config/next-config-js/transpilePackages)
- [Custom webpack configuration and compatibility caution](https://nextjs.org/docs/app/api-reference/config/next-config-js/webpack)
- [Next.js output-file tracing and monorepo root](https://nextjs.org/docs/15/app/api-reference/config/next-config-js/output)
- [Tailwind CSS automatic source detection and ignore behavior](https://tailwindcss.com/docs/detecting-classes-in-source-files)
- [Next source example for `turbopackIgnore`](https://github.com/vercel/next.js/blob/canary/packages/next/src/server/next-server.ts)

## 12. Audit artifacts

```text
/opencode/logs/mobtranslate-build-turbopack-deploy-20260714T1326Z.log
/opencode/logs/mobtranslate-build-turbopack-deploy-20260714T1326Z.meta
/opencode/logs/mobtranslate-build-turbopack-final-20260714T1347Z.log
/opencode/logs/mobtranslate-build-turbopack-final-20260714T1347Z.meta
/opencode/logs/mobtranslate-live-final-1280-20260714.png
/opencode/logs/mobtranslate-live-final-mobile-menu-20260714.png
```

The source document is the canonical incident record. The rendered HTML and raw Markdown are published together by
`render-docs.sh`; updating one without rerendering is incomplete.