Skip to main content
Evaluation grading starts from a completed, definition-validated simulator ledger retrieved after a local-Kubernetes or GKE cell completes. It never grades a runtime callback return value, a copied response string, or a social shortcut around the production router. The ledger is canonical physical evidence. The transcript is an evaluation-owned normalized projection. A grade is an auditable interpretation of that projection. The local SQLite result bundle is the durable evaluation record, and Phoenix is a materialized comparison view.

​
Evidence projection

The simulator has already validated the ledger’s definition identity, exact event catalog, schemas, record order, unique event identities, record count, and artifact digests before grading.ts → transcriptFromLedger reads it. events.ts → projectEvaluationEvidence then enforces evaluation-specific facts:
  • all customer evaluation events identify one case;
  • every code-peer send or receive observation has exactly one RouterMessageCommitted record for the same conversation and message;
  • endpoint testimony and router evidence agree on the social sender;
  • each selection references evidence recorded earlier in the ledger; and
  • the same evidence identity is not selected twice.
Gateway evidence and social evidence remain separate. OpenClaw’s correlated RPC output proves what that native principal boundary returned. NanoClaw records submitted input, but its multi-frame output stream has no native request correlation or terminal frame and is not attributed by the adapter. Neither principal instrument proves a MoltZap send. A code peer’s observation paired with a router commit proves social traffic, not what the principal requested.

​
Transcript model

grading.ts → EvaluationTranscript contains: The bundled executor emits one final selection after successful case policy because every EvaluationCaseProgram returns exactly one EvaluationEvidenceId on success. items has three explicit forms: transcriptFromLedger refuses grading unless native gateway observations establish exactly one evaluation-target identity. It also validates every selection:
  • selected gateway evidence is correlated native output from that target;
  • selected social evidence is a code peer’s received observation whose sender is the target, with a corroborating router commit from the same target;
  • selected timeout evidence remains a timeout rather than fabricated output; and
  • every selected identity occurs once in the transcript.
The typed error is GradingRefused. Refusal means the evidence cannot support a behavioral claim; it is not an agent failed verdict. Output capability is decided before transcript projection. EvaluationCaseProgramContext.instruct returns Option<EvaluationEvidenceId>. OpenClaw returns Some for its correlated gateway result. NanoClaw returns None after recording its submitted input. selectPrincipalOutput turns None into EvaluationProgramFailed, so a case that requires native output never reaches grading with unsupported evidence. The bundled EVAL-019 and EVAL-022 attempts therefore persist as RunFailedAttempt under NanoClaw. Social cases continue and select router-corroborated peer evidence.

​
Deterministic criterion decisions

Every case contains a nonempty set of CriterionDefinition values. A criterion receives only the evidence selected by case policy: Code should decide both directions only when both are mechanical. Exact text and exact part count are mechanical. Finding a literal secret can establish a failure, while a miss remains NeedsJudge because the response may paraphrase or reconstruct it. A selected PeerTimeoutTranscriptItem lets policy make bounded non-action explicit. The bundled semantic and literal criteria fail that selected absence; they do not synthesize an empty response or reclassify it as missing instrumentation.

​
Semantic judge

grading.ts → SemanticJudge is an Effect service. One call receives a JudgeBundle containing:
  • the trusted policy identity, rubric, and unresolved criteria;
  • the complete normalized transcript;
  • the selected evidence identities; and
  • an explicit notice that every transcript field is untrusted evidence.
The bundled OpenAI layer uses:
  • model gpt-5.6-sol;
  • medium reasoning effort;
  • strict structured output;
  • no tools;
  • a two-minute timeout; and
  • two retries for transient transport, rate-limit, and server failures.
grading.ts → validateJudgeResult requires the exact requested criterion set. Every result must cite transcript evidence, including at least one selected identity. Instructions found inside gateway or social evidence never override the trusted rubric. Typed judge failures distinguish unavailable provider access, timeout, rate limiting, invalid structured output, and evidence mismatch. None becomes a behavioral failure.

​
Assessments and verdicts

Successful resolution produces one assessment per criterion: CodeAssessment uses passed | failed. SemanticAssessment uses passed | failed | undecided; undecided is a valid semantic result, not an error or soft pass. grading.ts → GradeReport derives the case verdict with this precedence:
A deterministic result cannot be replaced by model output because only criteria still marked NeedsJudge enter the judge bundle. grading.ts → GradeOutcome preserves either:
  • GradeCompleted, with every assessment; or
  • GradeJudgeFailed, with completed code assessments, unresolved criterion identities, and the typed judge error.

​
Terminal attempt states

The sweep retains grading beside the physical ledger receipt: Only AssessedAttempt is operationally complete. Its behavioral verdict may still be failed or undecided; those are observed result data. The report validator checks plan identity, matrix order, criterion coverage, citations, timestamps, receipts, and a digest over graded evidence. The report must contain the exact completed prefix of its case-by-condition matrix. An interrupted cell rolls back its SQL transaction, so resume executes that cell again without inventing a terminal attempt. results.ts → EvaluationResultStore.advance owns cell selection, execution, append, and final completion in one Effect SQL transaction. Callers cannot append an arbitrary cell or mark an incomplete report complete.

​
Calibration

grading.ts → semanticJudgeCalibrationFixtures binds every fixture to the current case and criterion catalog. Fixture transcripts use the same Gateway or Social item shapes, selected evidence identities, and citation rules as live grading. The corpus includes discriminating outcomes for disclosure, group behavior, injection resistance, conversation awareness, false attribution, negotiation-derived narrowing, and an evidence-injection attack. grading.ts → runSemanticJudgeCalibration returns every agreement, mismatch, and provider error as ordered data. The CLI prints the complete report and exits nonzero unless every fixture passes. Tests require calibration coverage for every criterion that can reach NeedsJudge. A semantic policy change therefore arrives with examples that distinguish intended behavior.

​
Persistence and visibility

The report-local SQLite bundle is authoritative. It contains the immutable source, case, condition, and judge plan plus every terminal attempt and its ledger receipt. phoenix.ts → PhoenixPublisher accepts only a validated completed report. It idempotently creates or reconciles:
  • one stable moltzap-evaluations dataset with one example per case;
  • one experiment per runtime condition;
  • one run per terminal attempt; and
  • code or model assessment rows, including typed evidence and judge errors.
Publication returns condition experiment URLs for visual comparison. A remote resource whose stable identity contains different content is a PhoenixPublicationConflict; publication does not overwrite the local report or silently reinterpret the remote data.