Designing AI-Powered Web Systems That Scale

Most AI features fail at the seams, not the model. The demo works; the product wobbles. The difference is rarely the prompt — it is everything around the prompt: retries, budgets, fallbacks, observability, and the honest admission that a model call is a dependency with moods.

The operating idea

Treat every model call the way you would treat a third-party payment API. It has latency you do not control, failure modes you cannot enumerate, costs that compound quietly, and output you must validate before trusting. Nobody would let a payment provider write directly into their database. Somehow, models get that pass all the time.

Concretely, that means a contract at the boundary:

type ModelCall = {
  budgetMs: number; // hard latency ceiling
  fallback: "cache" | "degrade" | "fail-visible";
  validate: (raw: string) => Result<Parsed>;
};

Three properties follow. The system stays responsive because a slow model hits a ceiling instead of dragging the request with it. It stays honest because degraded answers are labeled, not smuggled. And it stays debuggable because every call has a trace: what went in, what came out, what it cost, what validated.

Where scale actually bites

The first thousand users stress your GPU bill. The next ten thousand stress your architecture. Queues appear where synchronous calls used to be. Caches appear in front of anything deterministic. Evaluation harnesses appear because “it feels worse this week” is not a bug report you can act on.

The pattern that survives: keep the model at the edge of the system, behind a boundary that speaks your domain types — never inside the core, speaking prose. Boring architecture around an ambitious component. That ordering is the whole trick.