# Kuku Yalanji RunPod Model-Version Evaluation Kit v1

This is a sealed, portable GPU benchmark for English to Kuku Yalanji model checkpoints. It evaluates every model
listed in `benchmark_manifest.json` on one frozen 340-row input, reports dictionary and elder results separately,
computes paired row-level bootstrap intervals, records the complete CUDA/software environment, and emits checksums
for every result.

The kit is an engineering and regression instrument. It is not speaker certification.

## Frozen benchmark

- 297 unique isolated English dictionary glosses, scored against every canonical headword attached to the gloss.
- 43 rights-cleared elder sentence pairs, scored separately and never pooled with dictionary words.
- Decoder: one beam, four-token no-repeat guard, repetition penalty 1.10, length penalty 1.0.
- Numeric type: float32 on an NVIDIA A40, with TF32 and sampling disabled.
- Determinism: seed 0, deterministic PyTorch algorithms, and `CUBLAS_WORKSPACE_CONFIG=:4096:8`.
- Paired bootstrap: 10,000 resamples with the frozen seed in the manifest.

The dictionary informed earlier corpus development, and the elder rows were seen in earlier evaluation. This is a
version-comparison and coverage probe, not an untouched generalization test.

## Expected model layout

```text
models/
  step2770/
    model.safetensors
    config.json
    tokenizer_config.json
    sentencepiece.bpe.model
    ...
  step3120/
    ...
  step4155_guarded/
    ...
```

The preflight hashes `model.safetensors` for every model and refuses to run if any byte differs from the manifest.
The input, row counts, pair-kind counts, direction, unique row IDs, tokenizer files, model type, GPU, CUDA version,
Torch version, and scoring-library versions are also fail-closed.

## One-command RunPod run

Use the image named in `benchmark_manifest.json`, attach at least 40 GB of container disk and 40 GB at
`/workspace`, stage this kit plus the three model directories, then run:

```bash
cd /workspace/kuku-eval-kit
./bootstrap_and_run.sh
```

The command creates a system-site-packages virtual environment so it keeps the image's CUDA-enabled Torch, installs
the exact scoring stack, verifies all inputs, evaluates each model sequentially, and writes a new immutable
`runs/kuku-version-benchmark-<UTC>/` directory.

## Outputs

Each successful run contains:

- `benchmark_manifest.json`, `requirements.lock`, and `SHA256SUMS.kit`: the sealed execution contract;
- `preflight.json`: input and model checksum verification;
- `environment.json`: Python, Torch, CUDA, GPU, and package identities;
- `predictions/<label>.predictions.json`: all 340 row-level outputs and corpus diagnostics;
- `version_probe_analysis.json`: exact retrieval, exposure strata, elder metrics, and paired intervals;
- `VERSION-PROBE-RESULTS.md`: concise human-readable tables and interpretation limits;
- `gpu-local-parity.json`: row-level GPU versus frozen local baseline comparison;
- `logs/gpu-resource.csv` and `resource_summary.json`: measured GPU use;
- `OUTPUT_SHA256SUMS` and `RUN_COMPLETE`: result integrity and completion gate.

Do not treat a launched process as a completed run. Require `RUN_COMPLETE`, verify `OUTPUT_SHA256SUMS`, confirm
all three reports contain 340 rows and zero empty outputs, and inspect real GPU/resource progress before deleting
the pod.

## Fast post-training feedback loop

Keep this kit on the training pod. After a new checkpoint is produced:

1. Merge or export it into a Transformers model directory.
2. Compute its `model.safetensors` SHA-256 and add a new manifest entry with a unique label and directory.
3. Run `./bootstrap_and_run.sh` before downloading multi-gigabyte weights.
4. Compare the new row-level metrics and paired intervals against all baseline entries.
5. Download the small result directory first. Download and publish model weights only if frozen promotion gates pass.
6. Preserve failed results as negative evidence, then terminate the pod.

New checkpoints must not change the frozen input, decoder, dtype, metric code, or baseline outputs. Any benchmark
change requires a new kit version.