We Seem to Agree

When I ask my team to look into a problem, I’m always interested in what each of them sees that the others might miss. Having AI help them work through it has been an interesting addition, though I’ve begun to wonder what happens when they’re all getting similar advice. The more I think about it, the more I’d want to understand how they reached their conclusions before taking their agreement at face value.

Researchers Anil Doshi and Oliver Hauser examined how AI assistance affected creative writing. They asked almost 300 people to write a short story. Some worked without assistance, while others could request a starting idea from AI. A third group could request up to five ideas. Readers then evaluated the finished stories without initially knowing which writers had received help.

Access to AI improved ratings for novelty and usefulness, particularly among people who’d scored lower on an earlier creativity exercise. When the researchers compared the stories with one another, however, the assisted work was more similar across writers. Individual stories had improved while the collection had become less varied.

There are limits to how far I’d carry this finding into a business setting. These writers couldn’t customize their prompts or discuss the suggestions with the model, so their experience was more restricted than using an assistant at work. The experiment doesn’t establish that business recommendations would converge in the same way, but it gives me reason to wonder how much independent judgment their agreement would represent.

Consider a company looking at a new market. Several people assess the opportunity, each using AI to help with the analysis. They contribute their experience, check information, and develop recommendations they’re comfortable defending. The assistance could allow them to examine the opportunity in considerably more detail than they would have managed on their own.

If the model encourages them to focus on similar customers or assumptions about demand, though, those suggestions may influence what they investigate next. Each person could spend substantial time developing a case for much the same approach. By the time they compare their work, they have several persuasive recommendations and broad agreement about how to proceed.

I’d find that encouraging, but I’d want to know what supported the agreement. One person might have spoken with prospective customers, while another examined the experience of companies already serving the market. If those investigations support the same recommendation, there’s a reasonable basis for additional confidence. If everyone has elaborated on similar suggestions from AI, their agreement tells us less about how thoroughly the opportunity has been explored.

Businesses already encourage this sort of overlap. Give people the same strategy presentation and a sense of what management wants to hear, and their recommendations may have quite a bit in common. AI could add another influence, even when people work separately and believe they’re contributing independent perspectives.

That makes me think more carefully about the questions I’d ask people to investigate. Someone looking into why customers haven’t bought comparable products may learn something different from a colleague studying how an unconventional competitor succeeded. AI could help both, and the team would have more to discuss than the relative merits of several similar proposals.

They might still conclude that the original approach is the best one. I’d be more comfortable with that decision knowing they had examined evidence that could have led them elsewhere. The number of people supporting a recommendation would tell me something, but I’d put more weight on what they’d learned before deciding to support it.

Algorithm & Blues publishes on Sundays. If this was forwarded to you, you can subscribe at joefuqua.blog.

Research reference

Doshi, A. R., and Hauser, O. P. Generative artificial intelligence enhances creativity but reduces the diversity of novel content. Author manuscript.