Cost Attribution for Shared LLM Gateways

Untagged traffic is unowned spend

Shared gateways earn their keep by centralizing auth, model routing, retries, and safety filters. They fail when usage lands in one provider invoice with no join key back to product surfaces. Without request-level tags—tenant id, workspace, feature flag, caller service, model tier—you cannot answer whose prompt burned the budget or whether a regression doubled completion tokens.

Treat attribution as a gateway contract, not a BI afterthought. Reject or quarantine calls that omit required metadata in non-prod; in prod, default tags must still be unambiguous enough for chargeback and incident response.

Meter tokens, price dollars, reconcile both

Providers bill on tokens, cached tokens, tool calls, and sometimes image or audio units. Your gateway should emit normalized usage events: input/output/cached tokens, model id, latency class, and whether the response came from a semantic cache. Convert to dollars with a versioned price table so historical reports stay auditable when list prices change.

Agents that fan out to multiple model calls need a correlation id that rolls child costs into a parent session. Otherwise product dashboards under-count agent workflows and over-count leaf microservices.

• Require tenant, product, and caller tags on every authenticated request
• Emit usage events with model, token splits, cache hits, and correlation ids
• Version price tables so dollar reports survive provider rate changes
• Expose soft quotas and hard caps per tenant with clear 429 semantics

Chargeback that teams can actually act on

Finance-friendly monthly rollups are necessary but insufficient. Engineering needs daily spend by feature and model so they can cut temperature, shrink context, or switch tiers. Product needs per-tenant burn rates for pricing and fair-use enforcement. Publish both views from the same usage stream.

Credit semantic-cache hits and prompt-cache discounts explicitly. If you hide savings, teams stop investing in cache keys; if you over-credit, finance disputes the model. Document who pays for shared platform overhead versus tenant-driven traffic.

Close the loop with budgets and alerts

Attribution without enforcement is a report. Wire budgets to gateway policy: warn at 70%, throttle non-critical routes at 90%, and page owners on runaway fan-out. Pair cost alerts with quality metrics so teams do not silently degrade answers to hit a number.

A shared LLM gateway is an internal product. Cost attribution is part of its SLA—same as availability and latency.

• OpenAI — Usage and costs
• Anthropic — Usage and rate limits
• Portkey — LLM gateway observability