Do consumers prefer AI or human to recommend products?

Many platforms employ a mix of algorithms and humans for recommending products. For example, Spotify features algorithm and editor / celebrity playlists. Amazon employs both algorithmic product recommendations and “staff picks.” Apple News features experienced editors in conjunction with algorithms when recommending products. This phenomenon begs the question, why do these big platforms, run by smart people, use a mix of AI and humans? The intuitive answer is that the two types of product recommenders must serve complementary purposes.

In new research conducted with colleagues at Temple and U. of Science and Technology China, we run a field experiment on over 50k users of a mobile book platform to test how consumers interpret human and AI curations differently by randomizing in-app messages that nudge consumers to explore either AI or human curation feeds.

After running the experiment for one week, we notice that the 2 messages performed similarly well in lifting purchases compared to the control message.

However, when we dug deeper, we found that the lift was achieved in very different ways. The algorithm message primarily boosted sales in genres previously purchased by the customer while the human message increased sales primarily in genres new to the customer. In essence, algorithm messages led to a depth selling effect while human messages led to a breadth selling effect. Importantly, the purchases were primarily driven by titles located outside of the curated feeds with explicit algorithm or human editor labels, suggesting that customers were not simply driven to curation labels that matched the message, but rather interpreted the curation messages differently. Algorithms are interpreted as offering a comparative advantage at personalization while human editors offer a comparative advantage at recommending novel products across all genres.

If AI = depth selling and human = breadth selling, how can we optimally combine the AI and human messaging nudges in an optimal messaging strategy? If we borrow the exploration then exploitation framework when making decisions in an unknown space, we would probably want to perform breadth selling then depth selling. The idea is simple, cultivate customers preferences by broadening their horizons using a human curation messages and then leverage those new preferences to harvest the sales of similar products using AI curation messages. The red square on the right of the schematic below represents the missed opportunity when reversing the message sequence.

To test this hypothesis, we run the experiment for a second week where those who received the treatment messages get randomly assigned one of the two treatments again. We end up with 4 total sequences over 2 weeks (HA, AH, AA, HH). Our intuition appears correct for the 2 weeks’ total sales. While all treatment sequences improve upon the control message, the HA sequence dominates all other sequences, including the reversed AH sequence.

So which curation source is more effective? The answer is that it depends on the objective. For maintaining the status quo preferences, emphasizing algorithmic curations work better. For growing consumer preferences, emphasizing human curation works better. Better yet, optimally combine the both in sequential messaging campaigns the get the best of both worlds.