An agent can carry out an instruction competently and still leave you with an outcome you wouldn’t have chosen. Anthropic’s Project Swap offers a small, useful example of how that happens.
In the company’s experiment, 201 employees across six offices brought books to trade. Each had a short conversation with Claude about reading preferences before an agent negotiated on their behalf. Participants also ranked ten books themselves so the researchers could compare what people wanted with what their agents inferred.
Anthropic reports that the inferred rankings agreed with participants on 61 percent of book pairs. Its analysis attributed 85 percent of the gap between the market’s outcome and the computed optimum to imperfect preference information. Within this experiment, the agents generally negotiated more successfully than they represented the people who sent them.
What an instruction leaves out
I see considerable promise in agents taking on meaningful work. But if I ask one to select a vendor, the assignment needs more than a price target. An existing relationship might matter because that vendor has been dependable when a deadline slipped. A slightly higher quote might include support we’d otherwise have to provide ourselves. Those reasons are easy to leave out when the person giving the instruction already knows them.
The practical lesson I draw from Project Swap is to make that context visible before expanding an agent’s authority. Explain a previous decision and why an apparently attractive alternative was rejected. Distinguish a preference, which can be traded against another benefit, from a requirement that must be met.
An agent also needs a way to handle a situation the original instruction didn’t cover. For a low-stakes choice, it may be reasonable to let it proceed and review the result. A commitment affecting another person or an important relationship may need to come back for a decision.
Check understanding through examples
I’d want to see how an agent handles a few representative choices before trusting it with recurring work. Ask it to explain which option it would choose and what information could change that decision. Correct a misunderstanding while the cost is small, and make sure the correction remains available on the next assignment.
That record needs review over time. A growing organization may be able to accept a cost it once couldn’t afford, while a family may need to change plans because someone requires more care. An old preference should not quietly become a permanent instruction.
Project Swap was a controlled book market involving Anthropic employees. It doesn’t establish how agents will perform in consequential commercial negotiations. Its model comparisons also belong to that experimental setting; they aren’t a general ranking of AI companies.
As we give agents longer assignments, I’d make understanding the person part of the acceptance check. Before the agent commits us to a decision, we should be able to inspect its reasons and recognize our actual priorities in them.