A chatbot demo takes an hour. A feature that behaves under real traffic, with real users and a real budget, takes considerably longer. These are the lessons that cost me the most time when integrating LLMs into backend services.

Treat the model as an unreliable dependency

An LLM API is a remote service with variable latency, occasional failures and nondeterministic output. Design around it the way you would design around a flaky third-party payment gateway:

• Set explicit timeouts. Median latency is not the number that hurts you; p99 is.
• Retry only idempotent calls, with exponential backoff and jitter.
• Provide a fallback: a cheaper model, a cached answer, or a graceful "unavailable" state.
• Keep the call out of the request path whenever the user does not need the answer immediately.

Get structured output, then validate it anyway

Free text is hard to build on. Ask for JSON that conforms to a schema, and validate it on your side regardless of what the provider promises.

import { z } from "zod";

const Ticket = z.object({
  category: z.enum(["billing", "bug", "feature", "other"]),
  priority: z.number().int().min(1).max(5),
  summary: z.string().max(200),
});

async function classify(text: string) {
  for (let attempt = 0; attempt < 3; attempt++) {
    const raw = await callModel(text);
    const parsed = Ticket.safeParse(safeJson(raw));
    if (parsed.success) return parsed.data;
  }
  throw new Error("Model failed to produce a valid ticket");
}

The validation step turns a nondeterministic component into one with a defined contract. Everything downstream can trust the type.

Retrieval before fine-tuning

When a feature needs company-specific knowledge, retrieval-augmented generation is almost always the right first step. It is cheaper, easier to update and easier to debug than fine-tuning.

A workable pipeline looks like this:

• Split documents into chunks of a few hundred tokens with some overlap
• Embed each chunk and store the vectors alongside the source reference
• At query time, embed the question, fetch the top results and include them in the prompt
• Ask the model to answer only from the supplied context, and to say so when the answer is missing

Most RAG quality problems are retrieval problems. If the right passage is not in the prompt, no model will save you, so measure retrieval separately.

Evaluate like you test

"It looked good in the playground" does not scale. Build a small evaluation set early, even twenty to fifty real examples with expected outcomes, and run it on every prompt or model change.

• Use exact-match checks wherever the output is structured
• Use a rubric-based model grader for free text, and spot-check it by hand
• Track results over time so regressions are visible

This is the LLM equivalent of a regression suite, and it is what lets you upgrade models without fear.

Control cost and latency

Spend grows quietly. A few habits keep it predictable:

• Log token usage per feature, not just per account
• Cache responses for identical or near-identical inputs
• Route simple tasks to a smaller model and reserve the large one for hard cases
• Trim the prompt. Long system prompts are paid for on every call

Security is not optional

Anything the user can type, and anything you retrieve from documents, can contain instructions aimed at the model. Never let model output directly trigger a privileged action.

• Keep secrets and authorization decisions outside the prompt
• Validate and constrain tool calls on the server
• Treat retrieved content as untrusted data

The takeaway

The model is one component in a system. The reliability of that system comes from the same things it always has: contracts, validation, observability and sensible fallbacks.