Why publishers’ engagement metrics are all over the map

Another firm, Newswhip, counts shares from Facebook posts, comments on links, use of share buttons on site, copy-and-paste shares. It looks at native content, but doesn’t include them in its publicly available monthly rankings.

Another key difference: Crowdtangle looks at engagements on all posts in a given timeframe, regardless of when the post was published; for its monthly rankings, Newswhip just looks at engagements on content published in the given month only, rather than counting engagements on posts that get surfaced over and over.

As a result, publisher rankings can vary widely. The Huffington Post, for instance, ranked No. 1 in engagements by Newswhip, whereas Crowdtangle had it in third place behind Fox News and NowThis.

Then there are individual publishers that are creating homegrown definitions of engagement. Purch, a network of tech sites, for its part has come up with its own reader score that gauges how deeply a person has gone with an article, based on how far they scrolled, whether they clicked on it and if they took an action afterwards such as sharing the article or signing up for a newsletter. The system was initially meant to be used by the editorial staff, with an eye toward using it as part of the ad sales process.

The varying measurements matter because publishers are increasingly using them to show they can pull in audiences that are getting more of their content through social side doors. As of last summer, Facebook accounted for nearly 40 percent of publishers’ social referral traffic, surpassing Google’s 38 percent, according to Parsely’s network of some 400 media sites.

Moreover, publishers and networks provide numbers that are inconsistent with independent reporting and different reporting tools that return significant differences in engagement results, said Benjamin Arnold, business director at We Are Social. Using different time frames, whole number versus averages and dividing by followers can account for major discrepancies in results, he said. “This is a serious issue in our industry,” he said.

Complicating matters, Facebook has upwards of 200 metrics that can be exported by publishers to use to cook up an engagement score, said Maikel O’Hanlon, vp of social media strategy at Horizon Media.

“We see organizations that base it on likes, comments and shares, and others say they want to include every definable user action like a click to expand a photo, a link clickout to an article, all that counts as engagement,” he said. “The advent of video has been a whole new slew of metric dissonance because they say we’re going to count video separately and other people say it should count as engagement. The volume of social metrics puts that cherry-picking on steroids. It’s easy to make any publishing entity look like the cat’s meow.”

Further, publishers aren’t always open about where their numbers come from, whether they’re straight from Facebook or customized by a third party, or how much of their social traffic is organic or paid for, which has a connotation of lower quality, said Noah Mallin, head of social for MEC, North America.

“If we’re going to partner with a publisher, we need to figure out what the value of their engagement is, and there are a lot of ways of measuring that,” he said. “If you’re weighting, you might put more weight on a click versus a share. Obviously they’re going to want to look at ones that tell a better story for them. If that lines up with what we want, OK. But if a publisher would come to us and say, ‘We’re overweighting likes,’ I wouldn’t look at that as a positive.”

In time, it’s possible the engagement metric will take care of itself. Agencies have their eye on other metrics as they see evidence that engagement on publishers’ links on Facebook is declining, perhaps as Facebook gives priority to video and natively uploaded content in users’ feeds.

Mallin, for one, is increasingly looking at completed video views and time spent as a stand-in for engagement. “The rate of content being viewed in and of itself is significant; that’s a better proxy.”