Benchmarks

Tanilo's score on a public benchmark, against published paper baselines.

Internal baseline, not a leaderboard result: retrieval recall is not reported and parametric-knowledge contamination is not controlled for on Tanilo's run. The paper baselines below were not re-run by us — they are the numbers reported in Schlichtkrull et al. Measured 28 May 2026 on the pipeline as it was then — the backend was replaced in September 2026, and this number has not been re-run since.

AVeriTeC is multi-evidence fact-checking with a 4-label space. Published academic systems land near 30% in the original paper; Tanilo's own pipeline, run under different conditions and without controlling for retrieval recall or contamination, scores higher on the same dev set. That gap is a reason to treat the comparison cautiously, not as a head-to-head result.

Headline

Reported on the AVeriTeC 2024 dev set, with the verdict mapping selected by inspection on the calibration half. Retrieval recall is not reported and parametric-knowledge contamination is not controlled for. Treat as an internal baseline, not a leaderboard result.

Held-out matches calibration → verdict mapping is not overfit. Selected via inspection on the calibration half; held-out is untouched during selection.

Not re-run since 28 May 2026. The backend that produced this run was replaced in September 2026. This number describes the pipeline as it existed then, not the pipeline live today.

Per-category accuracy (held-out half)

Why the lower numbers on Conflicting / NEE are a safety property, not a model failure: when evidence genuinely conflicts, Tanilo leans skeptical — most misclassified Conflicting claims received Refuted verdicts, not falsely-confident Supported ones. That is the right failure mode for regulated content. A verifier that fails toward caution is safer than a verifier that fails toward confident-wrong. We publish the absolute numbers as-is so a buyer can audit the failure shape themselves.

POST /v1/sign — single Ed25519 signature

Takes one canonical_bytes_b64u, returns one JWS signature entry. The primitive every composed envelope leg uses.

Withdrawn. /v1/sign is the endpoint later found to sign caller-supplied bytes under the production issuer key with no authentication (the forgery-oracle defect, see the incident record). It now requires an authorization header and returns 503 until its verdict path is completed — same disclosure-asymmetry family as /v1/compose below.

n=50 sequential calls. Host: https://agentoracle.co (Vercel edge), measured 2026-06-25 — before tanilo.io existed. Dominated by network round-trip and TLS, not crypto — the Ed25519 sign itself is sub-millisecond.

# Historical only — endpoint is closed and auth-required today
curl -w "\n%{time_total}s\n" -X POST https://agentoracle.co/v1/sign \
  -H "Content-Type: application/json" \
  -d '{"canonical_bytes_b64u":"eyJ0ZXN0IjoidHJ1ZSJ9"}'

POST /v1/compose — full 2-signer composed envelope

Withdrawn 2026-08-28. These timings measured a path whose Tanilo leg returned a hard-coded verdict without evaluating the claim, so they are not comparable to any figure that includes verification work. The endpoint no longer issues while that path is in wire-up. Republished when the verdict path is connected and re-measured.

Single HTTP call. Orchestrates AT /v1/compose → v_gate_skill, AO computes v_gate, builds canonical bytes once, AT /v1/sign for AT's signature, AO signs locally, assembles JWS general serialization. Returns full 2-signer envelope verifiable against both published JWKS.

n=20 sequential calls. Includes one round-trip to AT /v1/compose + one to AT /v1/sign over the public internet from the AO origin region to the AT origin region. Co-location would cut this further; published as-is. Host was agentoracle.co, before tanilo.io existed.

# Historical only — endpoint does not issue today
curl -w "\n%{time_total}s\n" -X POST https://agentoracle.co/v1/compose \
  -H "Content-Type: application/json" \
  -d '{"claim_hash":"sha256-yourclaim","mcp_content":{"tool":"web.search"}}'

POST /v1/sign/batch — N canonical bytes, N signatures

Withdrawn. Like /v1/sign above, this is the forgery-oracle endpoint — closed, now requires an authorization header, and returns 503 until its verdict path is completed.

Bulk signing primitive for high-frequency agent loops. Max 100 items per request. Returns one signature entry per input; all entries share the same kid (issuer is constant). Useful when an agent needs to anchor many envelopes in one network round-trip.

# Historical only — endpoint is closed and auth-required today
curl -X POST https://agentoracle.co/v1/sign/batch \
  -H "Content-Type: application/json" \
  -d '{"canonical_bytes_b64u":["...","...","..."]}'

Latency is dominated by HTTP round-trip, not crypto. Batched cost-per-item drops linearly with N for any N ≥ 2.