Integrating LLMs in Production Web Applications

An LLM in production is a dependency with moods. Treat it like one: contracts at the boundary, fallbacks behind it, and a budget it cannot exceed. This is the unglamorous half of AI engineering, and it is the half users actually feel.

Latency is a product decision

A p50 of two seconds is not an infrastructure metric — it is your UX. The decisions that matter happen before a model is called: can this be streamed? Can it be precomputed? Can a smaller model handle the common case with the big one behind a “better answer” affordance? Streaming turns dead air into perceived progress; caching turns repeated questions into free ones. Neither requires a smarter model. Both beat one.

Cost compounds in silence

Token spend scales with success, which is exactly when nobody is watching the meter. Put the counter in the code, not the dashboard:

const result = await llm.call(prompt, {
  maxTokens: budget.remaining(),
  onUsage: (u) => metrics.record(feature, u),
});

Per-feature attribution from day one. The alternative is a monthly invoice and an archaeology project.

Failure is a mode, not an exception

Models time out, refuse, hallucinate, and drift after provider updates. Each needs a designed response: timeouts get fallbacks, refusals get honest UI, hallucinations get validation against source data, drift gets an evaluation suite that runs on every provider version bump — because “the same model” quietly is not.

None of this is exciting. All of it is the difference between a feature that demos well and a feature that survives contact with January’s traffic.