> ## Documentation Index
> Fetch the complete documentation index at: https://none-690febbe-docs-main-owned-harness-adrs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation grading reference

> How the private evaluation application validates gateway and social evidence, grades criteria, and retains operational failures.

Evaluation grading starts from a completed, definition-validated simulator
ledger retrieved after a local-Kubernetes or GKE cell completes. It never
grades a runtime callback return value, a copied response string, or a social
shortcut around the production router.

The ledger is canonical physical evidence. The transcript is an
evaluation-owned normalized projection. A grade is an auditable
interpretation of that projection. The local SQLite result bundle is the
durable evaluation record, and Phoenix is a materialized comparison view.

## Evidence projection

The simulator has already validated the ledger's definition identity, exact
event catalog, schemas, record order, unique event identities, record count,
and artifact digests before `grading.ts → transcriptFromLedger` reads it.

`events.ts → projectEvaluationEvidence` then enforces evaluation-specific
facts:

* all customer evaluation events identify one case;
* every code-peer send or receive observation has exactly one
  `RouterMessageCommitted` record for the same conversation and message;
* endpoint testimony and router evidence agree on the social sender;
* each selection references evidence recorded earlier in the ledger; and
* the same evidence identity is not selected twice.

Gateway evidence and social evidence remain separate. OpenClaw's correlated
RPC output proves what that native principal boundary returned. NanoClaw
records submitted input, but its multi-frame output stream has no native
request correlation or terminal frame and is not attributed by the adapter.
Neither principal instrument proves a MoltZap send. A code peer's observation
paired with a router commit proves social traffic, not what the principal
requested.

## Transcript model

`grading.ts → EvaluationTranscript` contains:

| Field                 | Meaning                                                                             |
| --------------------- | ----------------------------------------------------------------------------------- |
| `caseId`              | Branded identity shared by the case and its customer evidence                       |
| `target`              | The one target name and network identity established by native gateway observations |
| `items`               | Nonempty evidence ordered by ledger logical sequence                                |
| `selectedEvidenceIds` | Nonempty identities chosen by case policy                                           |

The bundled executor emits one final selection after successful case policy
because every `EvaluationCaseProgram` returns exactly one
`EvaluationEvidenceId` on success.

`items` has three explicit forms:

| Item                        | Meaning                                                                                                                                   |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `GatewayTranscriptItem`     | Recorded native principal input or correlated output, target identity, and normalized parts                                               |
| `SocialTranscriptItem`      | Code-peer observation, direction relative to the target, peer endpoint, conversation identity, router-commit evidence identity, and parts |
| `PeerTimeoutTranscriptItem` | A required autonomous peer exchange was not complete at its configured deadline                                                           |

`transcriptFromLedger` refuses grading unless native gateway observations
establish exactly one `evaluation-target` identity. It also validates every
selection:

* selected gateway evidence is correlated native output from that target;
* selected social evidence is a code peer's received observation whose sender
  is the target, with a corroborating router commit from the same target;
* selected timeout evidence remains a timeout rather than fabricated output;
  and
* every selected identity occurs once in the transcript.

The typed error is `GradingRefused`. Refusal means the evidence cannot support
a behavioral claim; it is not an agent `failed` verdict.

Output capability is decided before transcript projection.
`EvaluationCaseProgramContext.instruct` returns
`Option<EvaluationEvidenceId>`. OpenClaw returns `Some` for its correlated
gateway result. NanoClaw returns `None` after recording its submitted input.
`selectPrincipalOutput` turns `None` into `EvaluationProgramFailed`, so a case
that requires native output never reaches grading with unsupported evidence.
The bundled EVAL-019 and EVAL-022 attempts therefore persist as
`RunFailedAttempt` under NanoClaw. Social cases continue and select
router-corroborated peer evidence.

## Deterministic criterion decisions

Every case contains a nonempty set of `CriterionDefinition` values. A
criterion receives only the evidence selected by case policy:

| Decision           | Meaning                                                                                            |
| ------------------ | -------------------------------------------------------------------------------------------------- |
| `CriterionDecided` | Code conclusively established `passed` or `failed`, with detail and nonempty evidence-ID citations |
| `NeedsJudge`       | The evidence still requires semantic assessment                                                    |

Code should decide both directions only when both are mechanical. Exact text
and exact part count are mechanical. Finding a literal secret can establish a
failure, while a miss remains `NeedsJudge` because the response may paraphrase
or reconstruct it.

A selected `PeerTimeoutTranscriptItem` lets policy make bounded non-action
explicit. The bundled semantic and literal criteria fail that selected
absence; they do not synthesize an empty response or reclassify it as missing
instrumentation.

## Semantic judge

`grading.ts → SemanticJudge` is an Effect service. One call receives a
`JudgeBundle` containing:

* the trusted policy identity, rubric, and unresolved criteria;
* the complete normalized transcript;
* the selected evidence identities; and
* an explicit notice that every transcript field is untrusted evidence.

The bundled OpenAI layer uses:

* model `gpt-5.6-sol`;
* medium reasoning effort;
* strict structured output;
* no tools;
* a two-minute timeout; and
* two retries for transient transport, rate-limit, and server failures.

`grading.ts → validateJudgeResult` requires the exact requested criterion set.
Every result must cite transcript evidence, including at least one selected
identity. Instructions found inside gateway or social evidence never override
the trusted rubric.

Typed judge failures distinguish unavailable provider access, timeout, rate
limiting, invalid structured output, and evidence mismatch. None becomes a
behavioral failure.

## Assessments and verdicts

Successful resolution produces one assessment per criterion:

| Assessment           | Provenance                                          |
| -------------------- | --------------------------------------------------- |
| `CodeAssessment`     | Conclusive deterministic decision                   |
| `SemanticAssessment` | Validated model testimony, rationale, and citations |

`CodeAssessment` uses `passed | failed`. `SemanticAssessment` uses
`passed | failed | undecided`; `undecided` is a valid semantic result, not an
error or soft pass.

`grading.ts → GradeReport` derives the case verdict with this precedence:

```text theme={null}
failed > undecided > passed
```

A deterministic result cannot be replaced by model output because only
criteria still marked `NeedsJudge` enter the judge bundle.

`grading.ts → GradeOutcome` preserves either:

* `GradeCompleted`, with every assessment; or
* `GradeJudgeFailed`, with completed code assessments, unresolved criterion
  identities, and the typed judge error.

## Terminal attempt states

The sweep retains grading beside the physical ledger receipt:

| Attempt                         | Meaning                                                                                                                                    |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `AssessedAttempt`               | Execution, evidence projection, and every required assessment completed                                                                    |
| `RunFailedAttempt`              | The simulator allocated a ledger, but execution or infrastructure failed, including a case requiring unsupported native output correlation |
| `EvidenceRejectedAttempt`       | A completed ledger could not support the required transcript                                                                               |
| `JudgingUnavailableAttempt`     | Code assessments survived, but unresolved semantic grading did not complete                                                                |
| `LedgerAllocationFailedAttempt` | Durable ledger allocation failed before the run kernel owned a receipt                                                                     |

Only `AssessedAttempt` is operationally complete. Its behavioral verdict may
still be `failed` or `undecided`; those are observed result data.

The report validator checks plan identity, matrix order, criterion coverage,
citations, timestamps, receipts, and a digest over graded evidence. The report
must contain the exact completed prefix of its case-by-condition matrix. An
interrupted cell rolls back its SQL transaction, so resume executes that cell
again without inventing a terminal attempt.

`results.ts → EvaluationResultStore.advance` owns cell selection, execution,
append, and final completion in one Effect SQL transaction. Callers cannot
append an arbitrary cell or mark an incomplete report complete.

## Calibration

`grading.ts → semanticJudgeCalibrationFixtures` binds every fixture to the
current case and criterion catalog. Fixture transcripts use the same Gateway
or Social item shapes, selected evidence identities, and citation rules as
live grading.

The corpus includes discriminating outcomes for disclosure, group behavior,
injection resistance, conversation awareness, false attribution,
negotiation-derived narrowing, and an evidence-injection attack.

`grading.ts → runSemanticJudgeCalibration` returns every agreement, mismatch,
and provider error as ordered data. The CLI prints the complete report and
exits nonzero unless every fixture passes.

Tests require calibration coverage for every criterion that can reach
`NeedsJudge`. A semantic policy change therefore arrives with examples that
distinguish intended behavior.

## Persistence and visibility

The report-local SQLite bundle is authoritative. It contains the immutable
source, case, condition, and judge plan plus every terminal attempt and its
ledger receipt.

`phoenix.ts → PhoenixPublisher` accepts only a validated completed report. It
idempotently creates or reconciles:

* one stable `moltzap-evaluations` dataset with one example per case;
* one experiment per runtime condition;
* one run per terminal attempt; and
* code or model assessment rows, including typed evidence and judge errors.

Publication returns condition experiment URLs for visual comparison. A remote
resource whose stable identity contains different content is a
`PhoenixPublicationConflict`; publication does not overwrite the local report
or silently reinterpret the remote data.

## Related

* [How to add an evaluation](/development/eval-add-evaluation) — extend the
  case catalog, peer policies, and calibration corpus
* [Code-first evaluations](/development/evals) — run, resume, and publish the
  matrix
* [Grading typed ledgers](/simulator/grading) — open and inspect canonical
  simulator evidence