> ## Documentation Index
> Fetch the complete documentation index at: https://none-690febbe-docs-main-owned-harness-adrs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Code-first evaluations

> Run, grade, resume, and publish behavioral evaluations over native principal gateways and simulator ledgers.

`packages/evals` is a private executable application that demonstrates one
evaluation product built on `@moltzap/simulator`. Cases, peer behavior, runtime
conditions, criteria, and sweeps are ordinary TypeScript and Effect values.
Customers compose the simulator package directly and can build a
domain-specific authoring language around the parameters they need.

Every matrix cell is one `RunSpec` submitted through the core simulator's
local-Kubernetes or GKE profile. Each target and each autonomous code peer is a
separate Agent Sandbox application container. The controller invokes the case
Effect only after the complete roster and every runtime-specific bridge are
ready.

## One attempt, two interaction boundaries

A successful case path keeps principal control separate from social traffic:

1. The case contributes an exact keyed record of autonomous peer definitions.
2. The condition adds one OpenClaw or NanoClaw target, and execution
   materializes the peer definitions with the configured digest-pinned
   application image.
3. Case policy instructs the target through its runtime-native principal
   gateway.
4. The target and code peers create and use MoltZap conversations
   autonomously. Every inter-agent message goes through the production client
   and router.
5. Peer gateways expose the ordered observations produced by their autonomous
   policies. They cannot send commands or bypass the network.
6. The case program returns the one ledger evidence identity that its policy
   selects for grading.
7. Execution records that selection, completes the ledger, reopens it through
   the same simulator definition, and projects grading evidence.
8. Code criteria settle mechanical questions. One semantic-judge call resolves
   every remaining criterion.
9. Effect SQL commits the terminal attempt in a report-local SQLite bundle.
10. An explicit publish command materializes a completed report in Phoenix.

The simulator does not interpret privacy, helpfulness, injection resistance,
group behavior, or conversation awareness. Those meanings belong to case
policy and grading code.

## Principal input is runtime-native

`principal.ts → PrincipalDriver` is an evaluation-local adapter over one
concrete runtime gateway. There is no simulator-wide gateway union:

| Condition | Principal boundary         | Recorded evidence                              |
| --------- | -------------------------- | ---------------------------------------------- |
| OpenClaw  | Its persistent gateway RPC | Instruction attempted and final gateway output |
| NanoClaw  | Its owner-local socket     | Input frame submitted                          |

Gateway evidence states what the adapter submitted or received at that native
boundary. It does not claim that an agent followed an instruction or sent a
social message. NanoClaw's output is an uncorrelated multi-frame stream: the
adapter never treats the next frame as the response or terminal output for an
instruction.

`execution.ts → evaluationCondition` captures one runtime together with its
matching `PrincipalDriverFactory`. The factory builds a driver for each
attempt. OpenClaw keeps its native idempotency sequence inside that driver;
NanoClaw does not receive a synthetic correlation key. Adding another runtime
condition therefore adds an adapter for that exact gateway type rather than
another member of a shared command or response protocol.

## Autonomous peers use the production network

`cases.ts → EvaluationCaseDefinition` carries the exact keyed `peers` record
needed by that case. A direct exchange starts one peer; a group case starts its
question, source, and observer peers; a principal-only case starts none.
Unused peers are not acquired.

Each `peer.ts → EvaluationPeerDefinition` owns a closed application plan and a
factory that binds it to the configured digest-pinned peer image. The plan is
mounted into that peer's Sandbox and interpreted by
`peer-application.ts → runEvaluationPeerApplication`. Its production MoltZap
client resolves agents, opens conversations, receives messages, and sends
messages through the router.

The peer-specific controller bridge exposes only the observation Effect on
`EvaluationPeerGateway`. It cannot command the peer or bypass the production
network. Arbitrary closures, gateway objects, and shared state do not cross the
container boundary.

Case programs receive five capabilities:

* `peers`, preserving the case's exact keys;
* `instruct`, which uses the target's native gateway and returns
  `Some(evidenceId)` only when that gateway correlates selectable output with
  the instruction;
* `selectPrincipalOutput`, which unwraps that native output or fails the run
  explicitly when correlation is unsupported;
* `observeContext`, which records a peer exchange without selecting it; and
* `selectPeerOutput`, which records an exchange and returns its final
  observation identity.

A peer deadline produces typed `PeerExchangeNotObserved` evidence. Selection
policy may return that evidence, allowing a criterion to report bounded
non-action honestly rather than manufacturing a response.

Social cases ignore the optional principal output and select router-bound peer
evidence. EVAL-019 and EVAL-022 require selectable principal output. They run
through OpenClaw's correlated RPC, but finish as explicit `RunFailedAttempt`
states under NanoClaw because its native socket cannot make that claim.

## Typed ledger evidence

`events.ts → evaluationEvents` declares the complete customer event universe
before a run is allocated. It contains native gateway observations, code-peer
send and receive testimony, bounded peer timeouts, and the final evidence
selection.

`events.ts → projectEvaluationEvidence` pairs each code-peer social
observation with exactly one `RouterMessageCommitted` record for the same
conversation and message. The router sender must agree with the
endpoint testimony. Native gateway observations remain distinct because
principal I/O is not social traffic.

The current bundled case interpreter emits exactly one
`EvaluationEvidenceSelected` event after its program returns. The projector
also verifies that a selection names earlier evidence and is not duplicated.

## Grading

`grading.ts → transcriptFromLedger` produces an `EvaluationTranscript` with:

* one native-gateway-established target identity;
* ledger-ordered `GatewayTranscriptItem`, `SocialTranscriptItem`, and
  `PeerTimeoutTranscriptItem` values; and
* the evidence identity selected by case policy.

Selected social evidence must be a code peer's observation of output from the
target, corroborated by a router commit whose sender is also the target.
Selected gateway evidence must be correlated native output from that same
target. NanoClaw input establishes target identity but cannot satisfy that
selection. A selected peer timeout remains explicit absence evidence.

Every criterion receives the same selected evidence. Exact-answer checks can
decide both directions mechanically. A literal disclosure detector may
conclusively establish a failure, but a detector miss remains a semantic
question because the response may paraphrase or reconstruct the secret.

`grading.ts → SemanticJudge` is a provider-neutral Effect service. The bundled
OpenAI layer uses `gpt-5.6-sol`, medium reasoning, strict structured output, no
tools, a two-minute timeout, and two retries for transient provider failures.
The rubric and criteria remain trusted policy; the entire transcript is
delimited as untrusted evidence. Each result must cover the exact requested
criterion set and cite selected evidence.

Judge unavailability produces `JudgingUnavailableAttempt`; it never becomes a
behavioral `failed` verdict. `undecided` is a valid semantic verdict.

## Result bundles and honest failure states

The baseline contains sixteen cases, two runtime conditions, and one sample per
cell. `results.ts → runEvaluationSweep` advances those 32 cells in canonical
order. Each cell owns one SQLite write transaction from cell selection through
terminal-attempt commit; interruption rolls back that cell while earlier
attempts remain durable.

The SQLite bundle under `.moltzap/evals/results/` stores:

* the exact clean Git revision;
* immutable case, condition, and judge-policy snapshots;
* physical ledger receipts;
* normalized transcripts and assessments when evidence reached grading; and
* typed run, evidence, judge, and ledger-allocation failures.

`results.ts → resumeStoredEvaluationReport` validates every immutable plan
component, including the selected profile, images, Temporal address, and ledger
artifact location, before executing only the missing suffix. The report cannot
skip, reorder, or silently move a matrix cell.

Live failures remain results. OpenClaw or NanoClaw may fail to start, terminate,
omit required social behavior, time out, produce evidence that grading
rejects, lack native output correlation required by a case, or fail semantic
assessment. The sweep persists the corresponding typed state and continues
through the matrix. In particular, the two NanoClaw principal-output cases are
expected operational failures rather than forced passes. Behavioral `failed`
and `undecided` verdicts do not make the command fail; operational attempt
states produce a nonzero exit only after the terminal matrix has been saved.

## Commands

Run deterministic checks:

```bash theme={null}
mise x node@24.18.0 -- pnpm nx run @moltzap/evals:build
mise x node@24.18.0 -- pnpm nx run @moltzap/evals:typecheck:tests
mise x node@24.18.0 -- pnpm nx run @moltzap/evals:test
mise x node@24.18.0 -- pnpm nx run @moltzap/evals:lint
mise x node@24.18.0 -- pnpm nx run @moltzap/evals:arch:check
```

Run the semantic calibration corpus:

```bash theme={null}
OPENAI_API_KEY=... \
  mise x node@24.18.0 -- pnpm nx run @moltzap/evals:calibrate
```

Start or resume a live report:

```bash theme={null}
OPENAI_API_KEY=... \
MOLTZAP_CONTROLLER_IMAGE=CONTROLLER_IMAGE_AT_SHA256 \
MOLTZAP_SUPPORT_IMAGE=PEER_APPLICATION_IMAGE_AT_SHA256 \
MOLTZAP_NANOCLAW_IMAGE=NANOCLAW_APPLICATION_IMAGE_AT_SHA256 \
MOLTZAP_LOCAL_ARTIFACTS="$PWD/.moltzap/local-artifacts" \
MOLTZAP_TEMPORAL_ADDRESS=127.0.0.1:7233 \
  mise x node@24.18.0 -- pnpm nx run @moltzap/evals:eval -- \
  --profile local \
  --report-id baseline-2026-07-29 \
  --openclaw-model "$OPENCLAW_MODEL" \
  --nanoclaw-model "$NANOCLAW_MODEL"

OPENAI_API_KEY=... \
MOLTZAP_CONTROLLER_IMAGE=CONTROLLER_IMAGE_AT_SHA256 \
MOLTZAP_SUPPORT_IMAGE=PEER_APPLICATION_IMAGE_AT_SHA256 \
MOLTZAP_NANOCLAW_IMAGE=NANOCLAW_APPLICATION_IMAGE_AT_SHA256 \
MOLTZAP_LOCAL_ARTIFACTS="$PWD/.moltzap/local-artifacts" \
MOLTZAP_TEMPORAL_ADDRESS=127.0.0.1:7233 \
  mise x node@24.18.0 -- pnpm nx run @moltzap/evals:resume -- \
  --profile local \
  --report-id baseline-2026-07-29 \
  --openclaw-model "$OPENCLAW_MODEL" \
  --nanoclaw-model "$NANOCLAW_MODEL"
```

The source worktree must be clean. Both model IDs are required and become part
of the immutable native runtime configuration. The controller/support, peer,
and NanoClaw application images must be immutable digest references. Their
presence is an execution prerequisite, not evidence that the NanoClaw image or
a live cluster has passed qualification.

For GKE, select `--profile gke`, replace the local artifact root with the
Terraform-owned `MOLTZAP_GKE_ARTIFACT_BUCKET`, and provide the explicit
`MOLTZAP_KUBE_CONTEXT` and configured Temporal endpoint. Each profile submits
the same generated RunSpec module and reads the same relative completed-ledger
path.

Publish a completed report:

```bash theme={null}
PHOENIX_HOST=http://localhost:6006 \
  mise x node@24.18.0 -- pnpm nx run @moltzap/evals:publish -- \
  --report-id baseline-2026-07-29
```

`phoenix.ts → PhoenixPublisher` keeps the local SQLite bundle authoritative.
It idempotently reconciles a stable case dataset, one experiment per runtime
condition, every terminal attempt, and code or model assessments. The command
prints a publication receipt containing browser URLs for the condition
experiments. Set `PHOENIX_API_KEY` when the Phoenix deployment requires one.

## Related

* [How to add an evaluation](/development/eval-add-evaluation) — add a case,
  peer policy, criterion, and calibration fixture
* [Evaluation grading reference](/development/eval-grading-reference) —
  evidence, assessments, and failure states
* [Grading typed ledgers](/simulator/grading) — the simulator evidence
  boundary