Contract Testing for AI Tool Calling

Treat tool schemas as product APIs

Tool calling looks soft because the model produces natural language, but the side effects are hard: payments, tickets, emails, and database writes. If the tool contract is only documented in a prompt paragraph, every model upgrade becomes a silent breaking change.

Publish tools as versioned schemas with required fields, enums, idempotency keys, and explicit deny lists. Contract tests assert that both the model adapter and the tool executor agree on that shape—before a user session pays the cost of disagreement.

Test the boundary, not the poetry

Golden fixtures should include valid calls, near-miss malformed calls, and adversarial payloads that try to smuggle extra fields or confuse nested objects. Replay recorded production failures into the suite so the next provider bump cannot reintroduce the same bug class.

Separate model-output contracts from tool-executor contracts. The first verifies that parsing and validation reject unsafe shapes; the second verifies that the executor still behaves correctly when given a valid, versioned payload. Mixing them hides whether the model or the backend regressed.

• Pin tool schema versions and fail CI on undeclared breaking changes
• Assert required fields, type coercion rules, and unknown-field rejection
• Include multi-tool sequences and partial-failure recovery paths
• Shadow-run new model routes against frozen contract fixtures

Make drift visible in release gates

Provider SDKs, JSON mode quirks, and tool-choice policies drift over time. Add a scheduled contract suite that hits staging with the same fixtures used in PR CI, and alert when pass rates drop even if product metrics still look fine.

Track contract pass rate, top failing tools, and argument-shape histograms per model route. A green dashboard with a red contract suite is how you ship a regression that only appears under real tool pressure.

Recover like a product, not a parser

When a contract fails, prefer fail-closed for irreversible tools and fail-soft for read-only assistance. Surface structured repair hints to the model—missing field names, allowed enums—instead of opaque 500s that encourage retry loops.

Contract testing is how AI products keep tool calling boring: schemas evolve deliberately, and models are guests of the API—not its authors.

• Pact — Contract testing
• OpenAI — Function calling
• JSON Schema — Specification