Search Found the Product. The AI Rejected It. Here’s How We Traced It.

Our Shop AI Agent returned a sensible-looking answer but left out a relevant product. A forest print existed in the catalog, native search could find it, and the assistant sometimes recommended it. Yet a fresh request could make it disappear.

The breakthrough was establishing which stage received the product, which stage rejected it, and what evidence would be needed before calling the repair successful.

The request was simple: “Find a nature print with plants or flowers.” Two relevant catalog items were a Flowers Print and a Foggy Forest Print. In the reproduced failure, the assistant displayed the flowers and omitted the forest.

One successful attempt had already shown both. A second fresh request exposed the omission. That difference mattered: repeating a query until it succeeds is not evidence that the underlying problem has gone away.

The final answer alone could not tell us whether the problem was search, missing product information, model judgement, an evidence-validation step, or card rendering. All of those stages could produce the same visible symptom.

We found an actual limitation in the connector data. Drupal supplied a short description excerpt; the relevance decision did not receive the proposed full description, categories, product attributes and enrichment hints.

That was worth investigating. A decisive product fact can sit beyond an excerpt limit or in an attribute rather than the body text. But identifying a useful improvement did not establish that it caused this particular omission.

We had to separate two questions: Could richer context improve future decisions? And where did this product disappear in the request we had reproduced? Treating the first answer as proof of the second would have sent us back into guesswork.

We used LangSmith tracing to follow the application through retrieval and relevance, then compared the recorded result with the browser. Because identical requests could behave differently, we needed a record of an actual failing application request. Tracing provided that record; we did not need to introduce a separate Studio workflow for this diagnosis.

Our working map was:

Customer request
  → native shop search
  → candidate products
  → AI relevance assessment
  → validated selection
  → response and visible product cards

At each boundary, the question was concrete: did the forest product arrive here, and did it leave? Product IDs made that comparison less ambiguous than titles or conversational summaries.

The final staging instrumentation captured the relevance input, selected product IDs and the nested model call. Its application trace ID connected that decision to the application request. A trace was useful only when those records actually arrived in LangSmith and matched the browser result.

The reproduced request showed both products reaching relevance: Flowers was product 16, and Foggy Forest was 28. The initial model judgement rejected the forest while retaining the flowers.

That narrowed the diagnosis substantially. The forest had not vanished from the native search results. In that captured failure, an evidence-repair step or a missing browser card was not responsible for discarding an otherwise accepted selection. The loss happened in the initial relevance judgement.

This distinction changed the next experiment. We needed to test whether the model could apply the customer’s purpose to the product’s depicted subject, rather than treating an art print’s physical product type as the whole question.

We also retained the failed attempt. A later successful answer could not replace the evidence of what the earlier request had actually done.

We refined the relevance rules and evidence checks, then compared the smaller and full GPT‑5.4 profiles against the same fixed evaluation cases. The smaller profile still failed the forest decision under the revised rules. The full profile passed the bounded comparison used for this release.

The resulting staging configuration assigned the full model specifically to relevance and its evidence repair:

OPENAI_CATALOG_RELEVANCE_MODEL=gpt-5.4-2026-03-05
OPENAI_CATALOG_RELEVANCE_REASONING_EFFORT=medium

Other shopping-model assignments stayed unchanged. This was a configured task-specific choice, not an agent automatically deciding to upgrade itself.

Changing the model was only part of the repair. We also had to preserve source-grounding checks and explicit customer constraints. Retaining a forest print is not a reason to accept wireless headphones when the customer explicitly asks for wired headphones with no Bluetooth.

A successful browser response did not guarantee a usable trace. During staging rollout, we found serialization failures in our tracing integration: timestamps were formatted incorrectly, and a scrubber tried to handle a schema class as though it were a model instance.

Those failures prevented parts of the telemetry from being ingested. We corrected the integration and tested its export behaviour, then read the real root and child runs back from the EU LangSmith projects. We did not treat a tracing-enabled setting, or a successful application response, as proof that the telemetry had arrived.

This is an easy trap in AI debugging: adding observability can create confidence before it creates evidence. We checked the trace output itself, including the recorded model and reasoning configuration, alongside the application logs.

After deploying the relevance and trace-export repairs to staging, we completed eight browser checks through the native shop widgets. Each product request was matched to its relevance root and child traces in LangSmith.

We checked complete replies, matching labels, loaded product images and unchanged empty carts. We also checked that the selected IDs in LangSmith agreed with the products visible in the browser. All eight traced decisions used the configured GPT‑5.4 snapshot with medium reasoning.

After the repair, the forest print appeared alongside the flowers in all three nature-print browser checks. The additional headphone checks passed across Drupal and WordPress, including a request that explicitly excluded wireless products.

The broader engineering lesson is more durable. A correct search result, a valid source quote, a reasonable model judgement and a correctly rendered card are separate things. Verify the boundary between them before deciding which one to improve.

If an AI assistant appears to ignore available information, start by following one real request through those hand-offs. A useful trace lets you explain exactly where the answer changed.