aeval

aeval

Deterministic, evidence-sealed evaluation for agent trajectories.

确定性、证据可密封的 agent 轨迹评测框架。

v0.4.0 · Apache-2.0 · Python ≥ 3.12 · pip install aeval-harbor

---

aeval delegates execution to [Harbor](https://pypi.org/project/harbor/) (harbor[e2b]==0.23.0) and concentrates on a single question: can the score be trusted?

Every collected artifact is re-hashed with SHA-256 and bound to a runtime lock. If the evidence gate does not pass, the trial is rejected outright — fail loud, never a silent zero. A sealed bundle can be independently recomputed. Grading is split into an outcome layer and a trajectory layer, where threshold collapsing and veto have the final say instead of a weighted average. Reliability is measured with pass^k, not "it passed once."

aeval 把执行委托给 [Harbor](https://pypi.org/project/harbor/)（harbor[e2b]==0.23.0），自己专注一件事：让"分数"可信。每份采集产物逐一 SHA-256 校验并绑定运行时锁；证据门不通过就整条拒绝（fail loud，不给静默的 0 分）。密封 bundle 可独立复算。判分分 outcome 与轨迹两层，阈值折叠、违规 veto 终裁——不做简单加权平均；可靠性用 pass^k 度量，而不是"跑一次过了"。

The end-to-end chain: suite → aeval run → host control plugin → Harbor sandbox → evidence gate → grader → veto → sealed store → report. An interactive version with guided views and a dark theme is in [docs/diagrams/eval-chain.html](https://github.com/xyrrrrr-r/aeval/blob/main/docs/diagrams/eval-chain.html).

端到端主链：套件 → aeval run → 宿主控制插件 → Harbor 沙箱 → 证据门 → grader → veto → 密封证据库 → 报告。交互版（引导视图 / 明暗主题）见 [docs/diagrams/eval-chain.html](https://github.com/xyrrrrr-r/aeval/blob/main/docs/diagrams/eval-chain.html)。

---

Why aeval / 为什么是 aeval

• Grading is deterministic code, not a model's opinion. A grader is loaded through a versioned reference (graders/outcome.py@v7), must prove its own identity (GRADER_ID non-empty, entry point is a coroutine, declared layer matches the reference), and runs over sealed evidence only. A grader whose identity contradicts its declaration is rejected as a lying grader. No model call and no network client sits anywhere in the grading loop — the shipped graders are pure functions over sealed evidence.

判分是确定性代码，不是模型拍脑袋。 grader 按版本化引用加载（graders/outcome.py@v7），必须自证身份（GRADER_ID 非空、入口是协程、声明层与引用一致），且只对密封证据运行。身份与声明不符的 grader 会被当作"说谎的 grader"拒绝。判分环节没有模型调用、没有网络客户端——自带的 grader 都是对密封证据的纯函数。

• Six prerequisite facts are real gates, not annotations. A suite declares in verdict.requirements which facts must hold for its trials to be judgeable at all. Any declared fact that should hold but does not makes the trial cannot_judge, and it leaves the valid denominator. Not judged is neither a pass nor a failure.

六项前置事实是真正的门，不是注释。 套件在 verdict.requirements 里声明哪些事实必须成立；任何一项该成立却没成立，该试次即 cannot_judge 并退出有效分母。判不了，既不当作通过，也不当作做错。

• A tamper-evident evidence chain. The collection manifest is bound to the runtime lock and every artifact carries its SHA-256. aeval recompute re-derives a sealed bundle independently. aeval rejudge is a protected entry point that refuses rather than quietly regrading — in this revision it refuses unconditionally, since regrading over a source bundle whose manifest cannot be verified is exactly the silent rewrite it exists to prevent.

防篡改证据链。 采集清单绑定运行时锁，逐产物记录 SHA-256；aeval recompute 对密封 bundle 独立复算。aeval rejudge 是受保护的重判入口：它选择拒绝，而不是悄悄重判——本版本中它一律拒绝，因为在无法核验源 bundle 清单的情况下重判，正是它要防的那种静默改写。

• pass^k, because one green run proves little. The report gives pass@k and pass^k side by side. pass^k is the unbiased hypergeometric estimator C(passes, k) / C(total, k) — the probability that all k attempts pass. An exclusion rate above 5% flags the whole run as untrustworthy, in the report, rather than being hidden.

pass^k，因为"跑一次过了"说明不了什么。 报告同时给出 pass@k 与 pass^k；pass^k 取超几何无偏估计 C(passes, k) / C(total, k)，即 k 次尝试次次通过的概率。排除率超过 5% 会把整个 run 标记为不可单独采信——写在报告里，不是藏起来。

• Reproducible by construction. Virtual clock (clock: virtual_offset), environment baseline probes that must pass before the run counts, and declarative observables instead of ad-hoc scraping.

可复现是构造出来的。 虚拟时钟（clock: virtual_offset）、跑前必须通过的环境基线探针、声明式 observables——而不是临时抓取。

• Agent-neutral. aeval new-agent scaffolds an adapter, aeval conformance verifies it. DSH, deepagents and OpenAI-ACP adapters ship in-tree, plus a reference conformance adapter used to prove onboarding costs one adapter and no framework surgery.

agent 中立。 aeval new-agent 生成适配器骨架，aeval conformance 一致性验证。仓库自带 DSH、deepagents、OpenAI-ACP 三个适配器，另有一个参考一致性适配器，用来证明"接入一个新 agent 只需一个适配器"、不必改框架。

• The evaluation framework is itself evaluated. aeval selftest injects faults: every failure class that must be caught has to be caught.

评测框架自己先被评测。 aeval selftest 做故障注入：每一类该拦下的失败都必须被拦下。

---

The six prerequisite facts / 六项前置事实

Each bit is produced stage by stage while a trial runs. It is never defaulted to True, never bulk-copied from an unverified source, and marked exactly once — a completed stage cannot become un-completed. judge_finished can only be set after grading actually ran, and an infra_error stop always clears it: a trial that died on infrastructure must not be recorded as having been judged.

六位事实在试次执行过程中按阶段逐位产生：绝不默认 True、绝不从未经验证的来源批量拷贝、每位只标记一次（已完成的阶段不能退回未完成）。judge_finished 只能在判分真正跑完后置位；infra_error 停机总是清掉它——死于基础设施的试次，不能被记成"判过分"。

| Fact / 事实 | What it asserts / 断言内容 |

|---|---|

| input_complete | The task input arrived whole and unmodified. / 任务输入完整、未被改动。 |

| agent_finished | The agent reached a terminal state. / agent 到达终态。 |

| integration_valid | The integration layer produced valid records. / 集成层产出有效记录。 |

| render_valid | The render/assembly step produced a valid artifact. / 渲染/装配步骤产出有效产物。 |

| judge_finished | Grading actually completed (set only by mark_judging_finished). / 判分确实跑完（只由 mark_judging_finished 置位）。 |

| artifact_schema_ok | Artifacts validated against their declared schema. / 产物通过其声明 schema 的校验。 |

---

Quickstart: 30 seconds, fully offline / 快速开始：30 秒，完全离线

No Docker, no