‘Levers being pulled that are unseen’: Measurement errors inside Amazon’s OSP program setting publishers on edge

amazon vacuum

In late August, staffers at several publishers that participate in Amazon’s Onsite Publishing program, or OSP, logged into their dashboards and found what looked like a nice surprise. 

Revenues had leapt up, in many cases by double digits and for many by more than 20%. For a program that has turned into a seven- or even eight-figure source of revenue for some publishers, the jumps were more than welcome, especially for those who had seen their OSP revenue flatten or even decline earlier that year. 

Those same participants got a much less pleasant surprise a few days later, when those gains all vanished from their dashboards, with no explanation or warning. 

After days of complaints and inquiry, publishers received an email saying Amazon had discovered a reporting issue that “inaccurately updated performance metrics” for June, July and August. 

That hiccup proved not to be the last. Earlier this month, some participants received an email saying that OSP had under-reported commissions due to a processing error that occurred in September. 

These hiccups, coming so close to the start of the shopping season, have put OSP participants on edge. They are also, sources said, emblematic of a program that has grown increasingly frustrating for its participants over the past year, thanks to abrupt changes in content requirements, the downgrading or removal of publisher content from search results, as well as a persistent lack of communication that many participants say is becoming untenable. 

While some participating publishers have continued to see revenue from OSP trend upward overall, many have seen it decline precipitously. Sources at two publishers participating in OSP said their revenues had declined so much that they were considering abandoning the program altogether. A third said they worried that they would have to abandon it next year.   

“The future of OSP’s pretty murky for us,” said a source at a fourth publisher. “Amazon provides enough reporting on the affiliate side [with Amazon Associates] where you can hire staff, forecast performance, you can follow best practices. With OSP, it feels like there are levers being pulled that are unseen by the content creators.” 

Reached for comment, Amazon sent a statement in lieu of responding to a set of questions Digiday sent. 

“At Amazon, we are constantly experimenting with our programs to both create the best shopping experience for customers and to help publishers find success on Amazon,” the statement read. “Onsite Publishing is still in an invite-only phase and we will continue to test and learn with how we are sharing this content with customers —testing new placements across Amazon, making it even easier for customers to discover great content and introduce publishers to a new audience. Our goals for the program are aligned with the goals of our publishers, and we’re committed to helping them succeed.” 

A loss of standing

While all affiliate programs are at least somewhat dependent on the retailers offering the commissions, OSP requires the publishers participating to invest a lot of faith in Amazon. OSP allows publishers to distribute product guides through Amazon’s site and app, then earn commissions if a person who reads those guides buys one of the featured products. 

Though publishers can house all their guides on profile pages inside Amazon’s properties, the overwhelming majority of the guides’ views come from Amazon search queries; for most of the past two years, a person who entered a search query such as “best air fryer,” for example, or “best pillows,” would likely find excerpts from a guide written by a participating OSP publisher among the first few search results. 

Like any digital media company, Amazon regularly tinkered with what it put in its search results. Those tests sometimes had catastrophic results for OSP publishers, who said Amazon rarely warned them that tests were coming, though sources contacted for this story said it was perfectly reasonable for them to conduct them. “They wouldn’t be doing their job if they weren’t trying to optimize,” a sixth source said. 

But this spring, many publishers noticed Amazon started to lean into one experiment that they found especially troubling. Instead of showing publishers’ guides in many search results, Amazon was instead showing users products that Amazon itself recommended, or products that were highly rated by Amazon customers. 

Those recommendation widgets, which featured many of the same design elements as publishers’ guides, felt like an ominous step toward an outcome many participants had feared since the program started: That Amazon, after using publishers’ content to learn how content might compel a user to purchase something, would simply cut publishers out of the equation. 

“They’ve basically replaced most of the OSP [content] I could see with ‘Highly Rated’ [results],” said a source at a fifth participating publisher, whose revenues from the program have fallen by close to 90% from their peak.

Opinions differ on just how much share of search results OSP content has lost. But beyond the loss of revenue, the change has set off a wave of suspicion among several participants that Amazon could be using the data created by OSP conversions to inform which items they are recommending to consumers. Amazon has, in several different instances, taken data generated by sellers and other partners and used it to spin up features or products that directly compete with those offered by those sellers and partners. 

“There are not industrial-strength MSAs [master service agreements] governing the use of [OSP] data,” the third source said. 

A change in standards

But Amazon’s support of its own recommendations was especially galling to some publishers in light of a separate change it began making in the spring, when it told publishers they would have to make changes to their guides. 

In its earliest incarnation, most of the content inside OSP came from well-known, heritage media brands, much of it lightly edited versions of content they’d already published on their respective sites. But within a few months, a new crop of publishers had come aboard, many of them much more obscure, some of questionable provenance and making recommendations based on much less rigorous evaluations. 

“They did not go out to try and achieve an environment that rewards great content; they went out to drive sales,” said a sixth source. “On the one hand, they expected the quality to go down [when they let smaller publishers in]. I don’t think they anticipated how quickly people would push the boundaries.”

Earlier this year, OSP participants got a big dose of guidance. In late spring, Amazon began telling participants that it was changing the guidelines and list of attributes that each guide needed to include. 

To several sources, the guidance was welcome, but the timeframe was not: Many were told they would have 30 days to get their guides, which in some cases amounted to hundreds of pieces of content, into compliance. 

“This is hundreds of articles. This is stuff that’s [been] previously approved,” the second executive fumed. 

Many made the changes, not just to avoid a significant drop in revenue. but because some of the changes would might lead to better performance. For some, that didn’t bear fruit.

“We never saw material improvement from those changes,” the third source said. “There was a bitter taste from that.”


More in Media

NewFronts Briefing: Samsung, Condé Nast, Roku focus presentations on new ad formats and category-specific inventory

Day two of IAB’s NewFronts featured presentations from Samsung, Condé Nast and Roku, highlighting new partnerships, ad formats and inventory, as well as new AI capabilities.

The Athletic to raise ad prices as it paces to hit 3 million newsletter subscribers

The New York Times’ sports site The Athletic is about to hit 3 million total newsletter subscribers. It plans to raise ad prices as as a result of this nearly 20% year over year increase.

NewFronts Briefing: Google, Vizio and news publishers pitch marketers with new ad offerings and range of content categories

Day one of the 2024 IAB NewFronts featured presentations from Google and Vizio, as well as a spotlight on news publishers.