# Kuku Yalanji Lexicon and Elder Version Benchmark Run Contract

Status: **locked before inference**  
Locked: `2026-07-14T07:24:31Z`

## Models

| Label | Checkpoint | Merged `model.safetensors` SHA-256 |
|---|---|---|
| `step2770` | reconstructed v21.2 step 2,770 | `1f2a31593a350f373f2c4345f1bd4f16a29ece18b53fa552ba1828acff7c84f7` |
| `step3120` | v22.0 continuation step 3,120 | `8ce31236b051195fbbd01df8d7149ce729249a30959592e9762b2dabf404c2aa` |
| `step4155_guarded` | exact v21.2 final step 4,155 weights | `7f9d0fe325e9e4568e45f13179adb336b93bbd53e83ddab2826e999eba3c76f7` |

## Inputs

- Combined 340-row inference file SHA-256: `6e525f16eb7475c6bc5450b978d70db0fbf1a0d71f2af9f143fb80c4bf60c01f`
- Isolated 297-row lexicon probe SHA-256: `38ce15e36a60018d1e9c057f359a68f4c3bf7f8f09b4d027fa7ca82abd88cce7`
- Elder 43-row source SHA-256: `29a9bfbfecbe535b56eda3bf0ecf255a17190b438186b60b999c78e1ee4854d2`
- Dictionary JSON-LD SHA-256: `a461612f8d2df805f4dd9f311e7a96702eee108cab04d752a6cbb7564b8b6993`
- v21.2 training split SHA-256: `7cb97c1c1d1afd05d0b7ddd04f64180182bf9de261bf5473d07b28bdb43967f0`

The exact row-construction method and counts are frozen in `probe_manifest.json` and `PREREGISTRATION.md`.

## Inference

Every model receives the same ordered 340 rows in one process invocation. The fixed decoder is:

- `num_beams=1`
- `no_repeat_ngram_size=4`
- `repetition_penalty=1.10`
- `length_penalty=1.0`
- `max_new_tokens=208`
- source language `eng_Latn`
- target language `gvn_Latn`

The evaluator is `training/translation/evaluate_nllb_lora.py`. The scorer is
`training/translation/analyze_kuku_version_probe.py`. No prediction may change row inclusion, accepted references,
model membership, decoder settings, or primary endpoints. All inference stdout, elapsed time, peak RSS, prediction
JSON, analysis JSON/Markdown, and checksums are retained in this directory.

## Endpoints and interpretation

The primary lexicon endpoint is normalized exact match against **any** canonical headword sharing the English gloss.
The elder primary descriptive endpoint is corpus chrF++; exact match, mean sentence chrF++, empties, and source copy
are also reported. Pairwise row bootstrap intervals use 10,000 resamples with the fixed seed in the scorer output.

This is a post-training regression audit. The dictionary is not an untouched sample and the elder set was observed
in earlier analyses. Results do not estimate speaker acceptability, dialect coverage, training-seed variance, or
unrestricted translation quality.