Benchmarks
Every number on this page is dated and marked historical; none is a current accuracy claim for today’s service. Dataset is public and harness is MIT-licensed for the AVeriTeC accuracy numbers below. The /v1 latency numbers are archived wall-clock measurements from before the Tanilo rename and are not re-runnable today — see the note on that section. The harness and dataset are public, so the method can be re-run.
If you only read three sentences: An earlier Tanilo pipeline was evaluated on the public AVeriTeC dataset on 28 May 2026. These historical results do not measure the current service. Running the harness against today's service produces a new evaluation, not a reproduction of that pipeline.
Accuracy — AVeriTeC 2024 Historical — 28 May 2026
Public academic fact-checking benchmark from AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web — Schlichtkrull, Guo and Vlachos, NeurIPS 2023. 500-claim dev set with 4-label space: Supported / Refuted / Not Enough Evidence / Conflicting Evidence. Tanilo's /evaluate pipeline was run end-to-end against the live API, free during the beta run. Calibration / held-out split is 250/250 by deterministic dataset index.
Tanilo's historical score on a public benchmark.
Internal baseline, not a leaderboard result: retrieval recall is not reported and parametric-knowledge contamination is not controlled for on Tanilo's run. Measured 28 May 2026 on the pipeline as it was then — the backend was replaced in September 2026, and this number has not been re-run since.
| System | Accuracy on dev |
|---|
Tanilo /evaluate | 57.6% |
AVeriTeC is multi-evidence fact-checking with a 4-label space.
Headline
| Split | N | Overall accuracy |
|---|
| Full dev set | 498 | 57.6% |
| Calibration half | 250 | 57.6% |
| Held-out half | 248 | 57.7% |
Reported on the AVeriTeC 2024 dev set, with the verdict mapping selected by inspection on the calibration half. Retrieval recall is not reported and parametric-knowledge contamination is not controlled for. Treat as an internal baseline, not a leaderboard result.
Held-out scores were similar to calibration scores. That is consistent with, but does not establish, a mapping that generalises. Selected via inspection on the calibration half; held-out is untouched during selection.
Not re-run since 28 May 2026. The backend that produced this run was replaced in September 2026. This number describes the pipeline as it existed then, not the pipeline live today.
Per-category accuracy (held-out half)
| Claim category | Accuracy |
|---|
| Supported | 70.6% |
| Refuted | 61.6% |
| Not Enough Evidence | 27.3% |
| Conflicting Evidence / Cherrypicking | 13.6% |
Observed error direction in this historical run: Of the 25 misclassified claims labelled Conflicting Evidence / Cherrypicking, 19 received Refuted verdicts. This describes the direction of classification errors in this run; it does not establish calibrated confidence, a safer failure mode, or the behavior of the current service.
Run the harness against today’s service
This does not reproduce the 28 May 2026 result. The commands below send the dataset to today’s /evaluate, whose models, prompts and gate rules have changed since that run, so they produce a new evaluation of the current service. The 28 May run’s own results file in the harness repo is the record of the historical number. Each claim sent makes model calls on Tanilo’s side (free during the beta, rate-limited).
git clone https://github.com/TKCollective/tanilo-eval-harness
cd tanilo-eval-harness
curl -sL https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json -o dev.json
python3 scripts/run_dev_eval.py
python3 scripts/score.py results/2026-05-28-dev/results.jsonl
At the time of the 28 May run, it took about 25 minutes at 3 concurrent workers. Submission registered at TKCollective/tanilo-benchmark. MIT-licensed. Open submissions — see submission-format.md.
Footnote on inference path
The harness used POST /evaluate, which was free and unmetered at the time of the run. If paid access ships later, whether the inference path stays identical hasn't been tested.
Shipped versus proposed. The 57.6% number above measures the model-based
/evaluate pipeline on AVeriTeC’s full claim mix. Separately, rule-based checks are
shipped on
POST /v1/verify-facts: the caller chooses that endpoint and a check type explicitly, no model is used, and it never falls back to a model (
deterministic mode). What is only
proposed is automatic routing: having
/evaluate itself send structurally checkable claims down a rule-based path and disclose a
resolution_path on each receipt. That routing is not implemented and has no date; no current receipt carries a
resolution_path field.
Why we publish this honestly: the value of a benchmark is that anyone can re-run it and check. If we picked numbers that flatter and hid the rest, the benchmark would be marketing. The point is the opposite — falsifiability is the moat. Run it yourself.
Latency — /v1 endpoints Historical
Raw curl wall-clock from a single sandbox client, measured 2026-06-25T22:14:00Z against agentoracle.co — tanilo.io did not exist yet at that date. tanilo.io itself is a static site; the API answers at api.tanilo.io. These are archived historical numbers, not something you can re-run against tanilo.io today.
POST /v1/sign — single Ed25519 signature
Takes one canonical_bytes_b64u, returns one JWS signature entry. The primitive every composed envelope leg uses.
| Statistic | ms |
|---|
| min | 120.8 |
| p50 | 136.2 |
| p90 | 156.2 |
| p99 | 400.8 |
| max | 400.8 |
| mean | 146.1 |
Withdrawn. This endpoint signed bytes supplied by the caller under the production key with no authentication (the forgery-oracle defect, see the incident record). Since 28 August it refuses: it returns 503 and signs nothing. The figures below are historical.
n=50 sequential calls. Host: https://agentoracle.co (Vercel edge), measured 2026-06-25 — before tanilo.io existed. Dominated by network round-trip and TLS, not crypto — the Ed25519 sign itself is sub-millisecond.
POST /v1/compose — full 2-signer composed envelope
Closed 28 Aug 2026. This agentoracle.co endpoint stopped issuing on 28 August 2026 (a8ce2a18d) and returns 503. It is a different route from the API gateway's /v1/compose, which was withdrawn on 24 September 2026.
Benchmark result withdrawn 2026-08-28. These timings measured a path whose AgentOracle leg returned a hard-coded verdict without evaluating the claim, so they are not comparable to any figure that includes verification work. The endpoint no longer issues while that path is in wire-up. Republished when the verdict path is connected and re-measured. The hard-coded verdict came from the AgentOracle stub, not AgentTrust. AgentTrust's leg signs its own bytes under its own key; its signature is not evidence that AgentOracle evaluated the claim.
Single HTTP call. Orchestrates AT /v1/compose → v_gate_skill, AO computes v_gate, builds canonical bytes once, AT /v1/sign for AT's signature, AO signs locally, assembles JWS general serialization. Returns full 2-signer envelope verifiable against both published JWKS.
| Statistic | ms |
|---|
| min | 380.2 |
| p50 | 401.8 |
| p90 | 417.6 |
| p99 | 745.0 |
| mean | 417.3 |
n=20 sequential calls. Includes one round-trip to AT /v1/compose + one to AT /v1/sign over the public internet from the AO origin region to the AT origin region. Co-location would cut this further; published as-is. Host was agentoracle.co, before tanilo.io existed.
POST /v1/sign/batch — N canonical bytes, N signatures
Withdrawn. This endpoint signed bytes supplied by the caller under the production key with no authentication (the forgery-oracle defect, see the incident record). Since 28 August it refuses: it returns 503 and signs nothing. The figures below are historical.
Bulk signing primitive for high-frequency agent loops. Max 100 items per request. Returns one signature entry per input; all entries share the same kid (issuer is constant). It was offered for anchoring many envelopes in one network round-trip; it no longer signs.
Latency is dominated by HTTP round-trip, not crypto. Batched cost-per-item drops linearly with N for any N ≥ 2.
What this is and isn't: these are archived wall-clock numbers from a single sandbox client, measured 2026-06-25 against agentoracle.co before the rename. They include network and TLS overhead. They are not the verification accuracy (see AVeriTeC above), they are not isolated crypto benchmarks (Ed25519 sign is <1ms on any modern CPU), and they are not the latency you would see from a co-located client. tanilo.io itself is a static site; the API answers at api.tanilo.io. These commands are historical reference only — not something you can re-run right now.
Conformance — composed envelope Historical — 24 June 2026 · 11/11 vectors
Parallel Node + Python verifiers, both byte-identical on Phase 1 (4 accept + 3 reject) and Phase 2 (3 accept + 1 reject). Independently verified end-to-end on 2026-06-24 by @giskard09 (IETF action-ref maintainer at argentum-core).
| Vector | What it covers | Result |
|---|
| comp-001…004 | 2-signer accept (both-act, AT-halt, AO-halt, trail-resolved) | PASS |
| comp-r01…r03 | 2-signer reject (tampered sig, null mycelium_trail_id, AND_PRESENT mismatch) | PASS |
| comp-005…007 | 3-signer accept (with Presidio screen_ref, comp-006 = PII_BLOCKED halts a payment both gates approved) | PASS |
| comp-r-screen | screen_ref recompute mismatch | PASS |
Full suite + verifiers at TKCollective/tanilo-receipt-spec. Spec anchor: draft-krausz-verification-state (IETF Internet-Draft; -02 at the time, current revision -03).
Reproducibility is the moat. Numbers without a re-run script are claims, not benchmarks. Methodology and harness MIT-licensed at TKCollective/tanilo-benchmark and TKCollective/tanilo-eval-harness.
