Cost-Aware Architecture for AI Products

Unit economics belong in the design doc

An AI feature that delights ten users and bankrupts the company at ten thousand users is not a finished design. Estimate tokens per request, expected concurrency, cache hit rate, evaluation overhead, and the willingness of customers to pay for the outcome. These numbers should influence model choice and interaction design before launch.

Cost awareness is not the same as cheapness. Some workflows deserve an expensive model because the alternative is human labor or lost revenue. The engineering task is to spend deliberately where quality creates leverage and refuse spend where it does not.

Route work by difficulty

Not every request needs the strongest available model. Classify tasks by risk and ambiguity. Deterministic extraction, classification, and formatting can often use smaller models or classical software. Open-ended synthesis, planning, and high-stakes advice may justify a stronger model with tighter guardrails.

Routing should be explicit and measurable. Track quality, latency, and cost by route. A cascade that escalates only when confidence is low preserves experience while keeping the average request affordable.

• Cache stable retrieval and repeated prompts
• Prefer structured outputs that reduce retries
• Budget evaluation runs like production traffic
• Expose cost alarms before invoices arrive

Product shape changes the bill

Streaming long essays is expensive. Asking for concise structured recommendations is cheaper and often more useful. Interface decisions—when to call a model, how much history to send, whether to regenerate—are cost controls as much as UX choices.

Batch offline work, precompute frequent answers, and avoid sending the entire account history when a small relevant slice will do. The cheapest token is the one the system never sends.

Make spend a health signal

Track cost per successful outcome, not only cost per request. A cheap endpoint that users retry five times is not cheap. Connect finance metrics to product analytics so teams can see whether spend is buying retention, conversion, or support deflection.

Sustainable AI products treat model spend as an architectural parameter. When the budget is visible, teams invent better systems instead of hoping traffic stays small.