Shopify CLI-equipped Hermes agent bought the wrong thing 28.8% of the time

Shopify’s UCP CLI lets agents browse stores, find products, and check out. The problem it doesn't solve for is enforcing intent: when the user delegates a task, no one is verifying that the agent's purchase actually satisfied the user's intent.

This is a huge problem. In a test we ran internally, using random intents, our Shopify CLI-equipped Hermes agent bought the wrong thing 28.8% of the time.

Agentic commerce is still in its infancy, and intent enforcement is one of the problems that have to be solved before it can gain adoption. I will personally never hand my agent my credit card unless I get explicit, chargeback-like guarantees that it will only buy things that satisfy my requirements, and I say that as a founder in the space of agentic payments.

Methodology: what makes a purchase wrong?

We ran 100 purchase intents through two agents, generated to span the full range of difficulty from single-constraint lookups to multi-constraint combinations that may not exist. All intents were single-product. Both agents were using Shopify's UCP CLI for discovery, and one of the agents additionally used delta Mandate for intent enforcement. Full results and raw data are available on request.

Each intent carried explicit constraints: size, color, material, price, and so on.

The intents ranged from easy ("Bamboo cutting board, under $30") to hard ("Cotton bandana, red, solid pattern, unisex") to potentially impossible ("Leather journal cover, solid pattern, not black, under $60, with elastic closure, leather book cover material").

Both agents were instructed to pass if they couldn't find a suitable item rather than picking something that doesn't fit.

A purchase was wrong if the agent bought a product that violated one or more constraints: wrong size, wrong color, wrong material, wrong price, ...

28.8% of the control agent's purchases failed that test. On hard intents (5+ constraints), the error rate hit 71.4%.

For example: asked for cotton joggers, the agent bought fleece joggers and called them cotton. Asked for "dishwasher safe" stainless steel, it passed products where the catalog data simply doesn't mention dishwasher safety. Every failure was the agent asserting a constraint was met without evidence to back it.

Enforcement has to be solved and it's not a model problem

The Shopify CLI works well. Both agents were able to navigate stores and find products. With UCP (the emerging agent-commerce protocol from Google and Shopify that makes storefronts legible to agents), it is essentially possible to have agents execute purchases today.

The problem is in enforcing intent. Better models will reduce the error rate, but the problem is structural.

Payments are (boringly) all about liability, and when we insert an agent in the flow it raises a huge question of who should pay for the agent's mistakes.

And the answer is very likely none of the current players:

  • Users are already used to chargebacks when the merchant makes an error
  • Merchants have taken a clear stance: your bot, your responsibility. They are not going to underwrite the risk of an agent they don't control, and "I never asked my agent to buy this" is an unfalsifiable refund claim that no merchant will absorb.
  • Card networks are in a similar position. The agent orchestrator sits between them and the intent provider, disintermediating their relationship with the transaction. They can't underwrite liability for decisions they can't see or control.

How delta Mandate works

delta Mandate inserts a verification step between discovery and payment. The full flow of a delegated payment now looks as follows:

delta extracts the product's actual attributes from the source and checks each one against the user's constraints. If the catalog says "fleece" but the user asked for cotton, the purchase is blocked before any money moves.

The result

Whereas the control agent had a 28.8% error rate (purchased an item it shouldn't have), the agent that was additionally enabled with the delta Mandate MCP did so 0 times.

And the Mandate agent wasn't just being conservative: there was no intent the control agent satisfied that Mandate couldn't. It found more valid products AND got every one of them right.

The control agent, by contrast, passed on intents 32.4% of the times the Mandate agent found a correct product.

The verification step runs server-side and adds no noticeable latency to the end user.

The Mandate Guarantee

If the checks passed when they shouldn't have (the user wanted white, the merchant site said black, and the evidence layer and policy engine let it through), we reimburse the user. We call this the Mandate Guarantee.

Credit card chargebacks protect the user from merchant risk. The Mandate Guarantee protects the user from agent risk. The difference is that we do it before the money moves, not after.

Since we pay for every false positive out of pocket, we naturally care a lot about keeping the error rate low. The benchmark is not just a marketing exercise; it's a direct measure of how much money we lose when our system gets it wrong.

Try it out

If you're building agents that make payments, this is the layer that makes it safe to hand them a card, a bank account, or a crypto wallet.

delta Mandate is available in live preview at delta.network. delta Mandate is available in live preview at delta.network. You can download the MCP directly and try it with your own agent, or through our public demo at sonofanton.shop.

Note: Our initial launch is limited to Shopify through UCP, and this live preview does not come with the Mandate Guarantee.