AI Agents in Production: Reliability, Observability, and Cost Control

The Demo-to-Production Gap

Building an AI agent that works in a demo is relatively easy. Making it work reliably for thousands of users — with predictable costs, full observability, and graceful failure handling — is the real engineering challenge. This gap catches many teams off guard.

Reliability Patterns

Retry with Exponential Backoff

import tenacity

@tenacity.retry(
    wait=tenacity.wait_exponential(min=1, max=60),
    stop=tenacity.stop_after_attempt(5),
    retry=tenacity.retry_if_exception_type(openai.RateLimitError)
)
def call_llm(messages):
    return client.chat.completions.create(model="gpt-4o", messages=messages)

Human-in-the-Loop Checkpoints

For high-stakes actions (sending emails, processing payments, deleting data), add a confirmation step before execution. LangGraph's interrupt_before makes this straightforward to implement.

Observability with LangSmith

import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"]    = "ls__..."
# All LangChain/LangGraph calls now automatically traced

Set up tracing before you go to production. Debugging an agent without traces is like debugging production without logs.

Cost Control

Model routing: Use gpt-4o-mini for classification and simple tasks, gpt-4o for complex reasoning.
Prompt caching: OpenAI's caching reduces costs by 50% for system prompts over 1024 tokens.
Hard limits: Always set max iterations. A runaway agent is a very expensive one.
Output compression: Tell the agent to be concise in intermediate steps.