Why AI Agents Need Guardrails: Preventing Cart Hallucination in LangGraph

This is for anyone building AI agents that talk about carts, checkouts, or other money-adjacent state — whether you're on the infra team, the product team, or wiring LLMs into commerce flows.

The bug was real: our AI shopping agent was adding products to the cart that the user never asked for. Here is what caused it, how we fixed it, and the general principle behind constraining agent actions.

Early in ShopAgent's development, we noticed something wrong. A user would type "add the Nike shoes to my cart", and the agent would confirm adding not just the Nike Pegasus but also a second product that had been mentioned earlier in the conversation. Sometimes it would add items with slightly different prices than what the search returned. Once, it reported adding a product that did not exist in the catalogue at all.

The LLM was hallucinating cart operations. It was generating responses that described adding products as if it had direct control over the cart — and we had designed the system to trust those responses.

This post explains the root cause, the fixes we applied, and the broader pattern of making agent actions deterministic when the stakes are real.

The original design gave the LLM both the ability to classify intent and the authority to describe the cart state in its response. This meant:

1. The LLM would classify "add Nike shoes" as add_to_cart — correct
2. The add_to_cart node would update the cart — correct
3. The respond node would ask the LLM to describe the cart — and the LLM would invent details

When asked to summarise the cart, the LLM draws on the entire conversation history. It sees earlier mentions of products, prices from previous searches, and vague references to "the shoes" or "the cheaper option". Without a hard constraint on what the cart actually contains, it fills the gaps with plausible-sounding but incorrect information.

The fix required two things: removing the LLM's ability to trigger cart actions through conversation, and forcing it to use only the actual cart state when describing the cart.

The distinction is between authority and description:

• Who can CHANGE the cart: Only the UI. Button clicks send structured prefixes ([CART_ADD], [CART_UPDATE]) that bypass the LLM entirely. The agent has zero write access.
• Who can DESCRIBE the cart: The LLM — but only using ground-truth state injected at response time. It cannot reference prices, quantities, or products from conversation memory.

Every fix below enforces one side of this boundary.

The first guardrail is in the system prompt:

SYSTEM_PROMPT = """...
CART GUARDRAIL: You CANNOT directly add, remove, or change items in
the cart. Cart modifications are handled exclusively through the UI
buttons on product cards and the cart panel. If a user asks you to
add, remove, or change cart items via chat, politely direct them to:
- Click "Add to Cart" on a product card to add items
- Use the +/− buttons in the cart panel to change quantities
- Click ✕ in the cart panel to remove items
Never claim you have added or removed a cart item.
"""

This tells the LLM explicitly that it does not control the cart. When a user types "add the Nike to my cart" in the chat, the LLM does not attempt to execute this — it directs the user to the product card button instead.

System prompts are not foolproof. A sufficiently adversarial prompt can override them. The architectural change below is the real enforcement mechanism.

The classify_intent node's intent list does not include add_to_cart or update_cart:

_INTENT_LITERAL = Literal[
    "search_products",
    "compare_products",
    "initiate_checkout",
    "provide_shipping",
    "complete_checkout",
    "get_order_history",
    "respond",     # <-- chat requests to add/remove items route here
]

Even if the user types "add the Nike to my cart" in the chat, the LLM cannot classify this as add_to_cart — that intent does not exist in the output schema. It routes to respond instead, where the system prompt instructs the LLM to redirect the user to the UI button.

The only way to trigger add_to_cart or update_cart is through the structured prefixes below. The LLM never chooses these routes.

When a user clicks "Add to Cart" on a product card, the frontend sends:

[CART_ADD] product_id=shoe-001 quantity=1

When they click + in the cart panel:

[CART_UPDATE] product_id=shoe-001 quantity=2

The classify_intent node checks for these prefixes before any LLM call:

if content.startswith("[CART_ADD]"):
    body = content[len("[CART_ADD]"):].strip()
    fields = {}
    for part in body.split():
        if "=" in part:
            k, v = part.split("=", 1)
            fields[k.strip()] = v.strip()
    return {
        "next_action": "add_to_cart",
        "_pending_product_id": fields.get("product_id", ""),
        "_pending_quantity": int(fields.get("quantity", 1)),
    }

No LLM call. No product name lookup. No ambiguity. The product ID and quantity come directly from the UI event, not from a language model. The add_to_cart node reads _pending_product_id from state and looks up the product in either the candidates list or the MCP catalogue.

This eliminates the hallucination completely. The agent can only add products that exist, with quantities that the user explicitly selected, triggered by a button click — not by the LLM inferring what it thinks the user wants.

The LLM still needs to describe the cart in its response — "I've added the Nike Pegasus ($89.99) to your cart." This is where the second class of hallucination was occurring: the LLM would reference prices from memory instead of from the actual state.

The fix is to inject the authoritative cart state as context before the LLM call:

cart_items = [CartItem(**c) for c in state.get("cart", [])]

if cart_items:
    cart_lines = "\n".join(
        f"- **{item.name}** x{item.quantity} — "
        f"**${item.price:.2f}** each — "
        f"line total **${item.price * item.quantity:.2f}**"
        for item in cart_items
    )
    cart_subtotal = sum(i.price * i.quantity for i in cart_items)
    context_parts.append(
        f"CURRENT CART (ground truth):\n{cart_lines}\n"
        f"- Cart subtotal: **${cart_subtotal:.2f}**\n"
        f"The cart contains EXACTLY {len(cart_items)} item(s).\n"
        f"CRITICAL: Ignore any product names or prices from earlier "
        f"messages. Use ONLY the cart data above."
    )

The LLM receives the exact cart contents from state — the same CartItem objects that were written by the add_to_cart node, which got the product data from MCP. There is no possibility of a price mismatch or a phantom product appearing in the response.

The same pattern applies to product search results:

context_parts.append(
    f"PRODUCT SEARCH RESULTS (ground truth):\n{product_lines}\n"
    f"CRITICAL: When referring to products, use ONLY the exact "
    f"names and prices listed above. Do NOT invent product names "
    f"or prices that are not in this list."
)

The word "CRITICAL" in the context is not for aesthetics — it reliably causes the model to treat the constraint as higher priority than its own knowledge.

The same pattern applies to shipping addresses. A user might type their address in the chat: "my address is 123 Main St, Amsterdam". If the agent extracts this and submits it to the checkout flow, there is no confirmation that the user intended this to be their shipping address for the current order — they might have been mentioning it in a different context.

The checkout form sends a structured prefix when submitted:

[FORM] Shipping address provided. Full Name: Jane Smith, Street: 123 Main St, ...

The routing function enforces this:

if action == "provide_shipping":
    for m in reversed(messages):
        if getattr(m, "type", "") == "human":
            latest_human_msg = str(m.content or "")
            break
    if latest_human_msg.startswith("[FORM]"):
        return "collect_shipping"
    # User typed address in chat — redirect to form
    return "respond"

Shipping addresses only enter the checkout flow from the confirmed form submission. Addresses mentioned in conversation are ignored.

There is a subtle fourth guardrail. The respond node sees the full conversation history — including raw [CART_ADD] product_id=shoe-001 quantity=1 messages from button clicks. If these reach the LLM as-is, it can misinterpret them: it might try to "add" the product again in its response, or confuse the structured syntax with user intent.

Before the LLM call, the respond node replaces every structured-prefix message with clean, human-readable text:

if content.startswith("[CART_ADD]"):
    pid = ""
    for part in content[len("[CART_ADD]"):].strip().split():
        if part.startswith("product_id="):
            pid = part.split("=", 1)[1]
    name = _product_names.get(pid, "a product")
    return HumanMessage(
        content=f"I added {name} to my cart.",
    )

The LLM sees "I added Nike Pegasus to my cart." instead of "[CART_ADD] product_id=shoe-001 quantity=1". The same cleaning applies to [CART_UPDATE], [FORM], and [ORDER_HISTORY] prefixes. This prevents the LLM from leaking implementation details into its response or treating structured commands as conversational context.

If we were adopting these incrementally, this is the order we would prioritise:

1. State immutability assertions (start here) — After the add_to_cart node runs, an assertion that the cart state matches the incoming [CART_ADD] parameters would catch any discrepancy between what was requested and what was written. This is a lightweight sanity check — a few lines of code, no new infrastructure — and it catches reducer bugs immediately in development.

2. Cart state hash — Computing a hash of the cart state before and after each node and including it in the node return value would make it immediately obvious if any node is unexpectedly modifying cart state as a side effect. This requires slightly more wiring but catches an entirely different class of bug: unintended state mutation by nodes that should be read-only.

Cart hallucination is a specific instance of a general problem: AI agents that are trusted to describe state they do not actually control. The fix is not a better prompt — it is removing the agent's authority over the action and replacing it with deterministic, structured operations.

Four layers enforce this in ShopAgent:

1. The system prompt says the agent does not control the cart
2. The intent list makes it impossible for the LLM to route to cart actions
3. Structured prefixes bypass the LLM entirely for all cart and form operations
4. Conversation hygiene replaces raw structured prefixes with clean text before the LLM sees them

Ground-truth context in the response node ensures the LLM's descriptions match the actual state, not its own memory of the conversation.

This article covers the guardrail layer. For the routing architecture underneath it — how classify_intent works, how multi-intent extraction is handled, and why the OpenAI SDK is used directly instead of LangChain — see Intent Classification as a LangGraph Router. For what sits on top — how AP2 mandates cryptographically chain the user's confirmed cart to the actual payment charge — see What Is AP2 and Why Should Your AI Agent Never Authorise a Payment Without It?.

The ShopAgent demo is live at https://shop-agent.agilecreativeminds.nl. See the demo showcase or follow the demo walkthrough. Built by Agile Creative Minds.