ACSI
Open-source certification for LLM model swaps
Added to this résumé on July 16, 2026
Overview
ACSI (AI Certification System Infrastructure) decides whether one model can safely replace another. It replays real LLM traffic through both the original and the candidate model and returns a pass or block verdict, backed by an ed25519-signed certificate.
The reference benchmark certified an Opus 4.1 to Sonnet 5 swap on real GitHub issues from Kubernetes, VS Code, React, and Rust. The public BLOCK certificate at acsi.dev includes a downloadable cert.json with working signature verification.
Demo
How it works
The output contract
Every response must satisfy a JSON schema with required keys summary (max 400 chars), category, severity, affected_area (max 60), action_items (max 5), needs_more_info, and no additional properties.
The largest failure clusters were summaries over 400 characters, affected_area over 60, too many action items, and code fences leaking into output. The noise floor first measures how much the original model varies on identical inputs, and a difference only counts as a regression if it clears that floor.
