Skip to main content
packages/evals is a private executable application that demonstrates one evaluation product built on @moltzap/simulator. Cases, peer behavior, runtime conditions, criteria, and sweeps are ordinary TypeScript and Effect values. Customers compose the simulator package directly and can build a domain-specific authoring language around the parameters they need. Every matrix cell is one RunSpec submitted through the core simulator’s local-Kubernetes or GKE profile. Each target and each autonomous code peer is a separate Agent Sandbox application container. The controller invokes the case Effect only after the complete roster and every runtime-specific bridge are ready.

​
One attempt, two interaction boundaries

A successful case path keeps principal control separate from social traffic:
  1. The case contributes an exact keyed record of autonomous peer definitions.
  2. The condition adds one OpenClaw or NanoClaw target, and execution materializes the peer definitions with the configured digest-pinned application image.
  3. Case policy instructs the target through its runtime-native principal gateway.
  4. The target and code peers create and use MoltZap conversations autonomously. Every inter-agent message goes through the production client and router.
  5. Peer gateways expose the ordered observations produced by their autonomous policies. They cannot send commands or bypass the network.
  6. The case program returns the one ledger evidence identity that its policy selects for grading.
  7. Execution records that selection, completes the ledger, reopens it through the same simulator definition, and projects grading evidence.
  8. Code criteria settle mechanical questions. One semantic-judge call resolves every remaining criterion.
  9. Effect SQL commits the terminal attempt in a report-local SQLite bundle.
  10. An explicit publish command materializes a completed report in Phoenix.
The simulator does not interpret privacy, helpfulness, injection resistance, group behavior, or conversation awareness. Those meanings belong to case policy and grading code.

​
Principal input is runtime-native

principal.ts → PrincipalDriver is an evaluation-local adapter over one concrete runtime gateway. There is no simulator-wide gateway union: Gateway evidence states what the adapter submitted or received at that native boundary. It does not claim that an agent followed an instruction or sent a social message. NanoClaw’s output is an uncorrelated multi-frame stream: the adapter never treats the next frame as the response or terminal output for an instruction. execution.ts → evaluationCondition captures one runtime together with its matching PrincipalDriverFactory. The factory builds a driver for each attempt. OpenClaw keeps its native idempotency sequence inside that driver; NanoClaw does not receive a synthetic correlation key. Adding another runtime condition therefore adds an adapter for that exact gateway type rather than another member of a shared command or response protocol.

​
Autonomous peers use the production network

cases.ts → EvaluationCaseDefinition carries the exact keyed peers record needed by that case. A direct exchange starts one peer; a group case starts its question, source, and observer peers; a principal-only case starts none. Unused peers are not acquired. Each peer.ts → EvaluationPeerDefinition owns a closed application plan and a factory that binds it to the configured digest-pinned peer image. The plan is mounted into that peer’s Sandbox and interpreted by peer-application.ts → runEvaluationPeerApplication. Its production MoltZap client resolves agents, opens conversations, receives messages, and sends messages through the router. The peer-specific controller bridge exposes only the observation Effect on EvaluationPeerGateway. It cannot command the peer or bypass the production network. Arbitrary closures, gateway objects, and shared state do not cross the container boundary. Case programs receive five capabilities:
  • peers, preserving the case’s exact keys;
  • instruct, which uses the target’s native gateway and returns Some(evidenceId) only when that gateway correlates selectable output with the instruction;
  • selectPrincipalOutput, which unwraps that native output or fails the run explicitly when correlation is unsupported;
  • observeContext, which records a peer exchange without selecting it; and
  • selectPeerOutput, which records an exchange and returns its final observation identity.
A peer deadline produces typed PeerExchangeNotObserved evidence. Selection policy may return that evidence, allowing a criterion to report bounded non-action honestly rather than manufacturing a response. Social cases ignore the optional principal output and select router-bound peer evidence. EVAL-019 and EVAL-022 require selectable principal output. They run through OpenClaw’s correlated RPC, but finish as explicit RunFailedAttempt states under NanoClaw because its native socket cannot make that claim.

​
Typed ledger evidence

events.ts → evaluationEvents declares the complete customer event universe before a run is allocated. It contains native gateway observations, code-peer send and receive testimony, bounded peer timeouts, and the final evidence selection. events.ts → projectEvaluationEvidence pairs each code-peer social observation with exactly one RouterMessageCommitted record for the same conversation and message. The router sender must agree with the endpoint testimony. Native gateway observations remain distinct because principal I/O is not social traffic. The current bundled case interpreter emits exactly one EvaluationEvidenceSelected event after its program returns. The projector also verifies that a selection names earlier evidence and is not duplicated.

​
Grading

grading.ts → transcriptFromLedger produces an EvaluationTranscript with:
  • one native-gateway-established target identity;
  • ledger-ordered GatewayTranscriptItem, SocialTranscriptItem, and PeerTimeoutTranscriptItem values; and
  • the evidence identity selected by case policy.
Selected social evidence must be a code peer’s observation of output from the target, corroborated by a router commit whose sender is also the target. Selected gateway evidence must be correlated native output from that same target. NanoClaw input establishes target identity but cannot satisfy that selection. A selected peer timeout remains explicit absence evidence. Every criterion receives the same selected evidence. Exact-answer checks can decide both directions mechanically. A literal disclosure detector may conclusively establish a failure, but a detector miss remains a semantic question because the response may paraphrase or reconstruct the secret. grading.ts → SemanticJudge is a provider-neutral Effect service. The bundled OpenAI layer uses gpt-5.6-sol, medium reasoning, strict structured output, no tools, a two-minute timeout, and two retries for transient provider failures. The rubric and criteria remain trusted policy; the entire transcript is delimited as untrusted evidence. Each result must cover the exact requested criterion set and cite selected evidence. Judge unavailability produces JudgingUnavailableAttempt; it never becomes a behavioral failed verdict. undecided is a valid semantic verdict.

​
Result bundles and honest failure states

The baseline contains sixteen cases, two runtime conditions, and one sample per cell. results.ts → runEvaluationSweep advances those 32 cells in canonical order. Each cell owns one SQLite write transaction from cell selection through terminal-attempt commit; interruption rolls back that cell while earlier attempts remain durable. The SQLite bundle under .moltzap/evals/results/ stores:
  • the exact clean Git revision;
  • immutable case, condition, and judge-policy snapshots;
  • physical ledger receipts;
  • normalized transcripts and assessments when evidence reached grading; and
  • typed run, evidence, judge, and ledger-allocation failures.
results.ts → resumeStoredEvaluationReport validates every immutable plan component, including the selected profile, images, Temporal address, and ledger artifact location, before executing only the missing suffix. The report cannot skip, reorder, or silently move a matrix cell. Live failures remain results. OpenClaw or NanoClaw may fail to start, terminate, omit required social behavior, time out, produce evidence that grading rejects, lack native output correlation required by a case, or fail semantic assessment. The sweep persists the corresponding typed state and continues through the matrix. In particular, the two NanoClaw principal-output cases are expected operational failures rather than forced passes. Behavioral failed and undecided verdicts do not make the command fail; operational attempt states produce a nonzero exit only after the terminal matrix has been saved.

​
Commands

Run deterministic checks:
Run the semantic calibration corpus:
Start or resume a live report:
The source worktree must be clean. Both model IDs are required and become part of the immutable native runtime configuration. The controller/support, peer, and NanoClaw application images must be immutable digest references. Their presence is an execution prerequisite, not evidence that the NanoClaw image or a live cluster has passed qualification. For GKE, select --profile gke, replace the local artifact root with the Terraform-owned MOLTZAP_GKE_ARTIFACT_BUCKET, and provide the explicit MOLTZAP_KUBE_CONTEXT and configured Temporal endpoint. Each profile submits the same generated RunSpec module and reads the same relative completed-ledger path. Publish a completed report:
phoenix.ts → PhoenixPublisher keeps the local SQLite bundle authoritative. It idempotently reconciles a stable case dataset, one experiment per runtime condition, every terminal attempt, and code or model assessments. The command prints a publication receipt containing browser URLs for the condition experiments. Set PHOENIX_API_KEY when the Phoenix deployment requires one.