AI in retail customer support: what the published evidence supports

Retail and e-commerce support has a recognizable shape: high volume, seasonal peaks, and a long list of repetitive questions about orders, deliveries, returns and product details, mixed with a smaller number of upsets that need a person. That shape is one reason customer support attracts so much attention as a use of generative AI. It is also why the gap between a good demo and a safe deployment is easy to underestimate.

This post sticks to what has been published. The evidence is better on some questions than others, and we flag where it runs out.

Evidence one: AI that helps the human agent

The most rigorous evidence is a field study of 5,179 customer support agents using a generative AI conversational assistant, rolled out in stages. Productivity, measured as issues resolved per hour, rose about 14% on average. For novice and low-skilled workers the gain was 34%, while experienced, highly skilled workers saw minimal impact. The authors also report improved customer sentiment and higher employee retention, and suggest the tool spreads the practices of more capable workers ^[1].

The paper was first issued in April 2023, revised in November 2023, and later published in the Quarterly Journal of Economics ^[1]. The abstract does not name the company or industry ^[1], so it cannot be read as a retail result.

For a retailer, the pattern is what matters. Retail support teams often rely on seasonal hires and rapid onboarding. A tool that brings newer agents closer to the performance of experienced ones addresses exactly that problem, and it keeps a person accountable for every reply.

Evidence two: AI that talks to the customer

Klarna, the payments and shopping company, published results one month after launching an AI assistant ^[2]:

• 2.3 million conversations handled, about two-thirds of its customer service chats

• Work equivalent to roughly 700 full-time agents

• Resolution time under 2 minutes, down from 11 minutes

• Repeat inquiries down 25%

• Coverage in 23 markets, around the clock, in more than 35 languages ^[2]

Treat this as a vendor claim from the first month, not an independent audit. The release also reports an estimated $40 million profit improvement for 2024, which is a projection ^[2]. It does not describe how customer satisfaction was measured, how many chats were escalated to people, or how "resolved" is defined, and those are the questions a retailer should ask first.

What a retailer should take from this

Start with assistance, then automate narrow, well-defined tasks. Anthropic's guidance on building agents recommends finding "the simplest solution possible, and only increasing complexity when needed." It separates workflows, which run through predefined code paths and suit well-defined tasks where consistency matters, from agents, which direct their own steps; agentic systems in general often trade latency and cost for better task performance ^[3]. Order status lookups and return-eligibility checks are well-defined. A customer disputing a charge after a damaged delivery is not.

Build an escalation path before launch. Agents can compound errors, which is why Anthropic advises extensive testing in sandboxed environments with guardrails ^[3]. In practice, that means the system needs clear rules for when it hands a conversation to a person, and the person needs the full conversation history.

Measure on real conversations from the start. Anthropic's research team began with about 20 queries reflecting real usage and found small test sets are enough when early effects are large ^[4]. A retailer can do the same: pull a sample of past tickets across order status, returns, product questions and complaints, and score the system's answers against what your policies actually say.

Watch the cost of complexity. Anthropic reports that agents typically use about four times more tokens than chat, and multi-agent systems about fifteen times more ^[4]. A single-purpose support bot is far cheaper to run than a multi-agent design, so match the architecture to the question you are answering.

What the evidence does not tell us

Three gaps are worth stating plainly. There is no controlled, published study here of AI answering retail customers directly in place of people. The one large study is not retail-specific. And the headline automation figures come from a company reporting on its own product in the first month. Any vendor promising a specific percentage of tickets resolved should be asked to show how that was measured, on what mix of inquiries, and over what period.

Practical implication: Pilot AI first as an assistant to your support team and as a bot for a few narrow, policy-driven request types with a clear route to a human, and judge it on your own ticket history rather than on another company's headline numbers.