How the Guardian is using article surveys to inform editorial decisions

“The data teaches us we shouldn’t make assumptions about reader levels of knowledge,” said Chris Moran, the Guardian’s strategic projects editor. “Editors are now treating a story that they might know inside and out much more objectively.”

Another example: Readers of articles covering the ongoing civil unrest in Venezuela were repeatedly asking why the country is in crisis. Now, articles covering the news there contain a box of text detailing the economic and social background leading to the unrest. Readers can vote on whether the snippet was useful. Perhaps unsurprisingly because the boxes contain information readers requested, they are getting 97 percent positive response rate.

The Guardian started the project in March, adding the tool to one article a day that was published from the foreign desk. According to the publisher, each question has thousands of reader responses, amounting to 9 percent of people on average who see the question. In some cases, this rises to 20 percent, but three weeks ago, this dropped to 3 percent. “We thought something had gone wrong with the tool,” said Moran, “but the data was telling us the article had done a good job in answering the question.”

The data is fed into Ophan, the Guardian’s analytics tool, and incorporated into the right text format for an article later that day. According to the publisher, 80 percent of all the questions asked since the project began are from regular readers, which the Guardian defines as readers that have been in contact with the Guardian at least seven times in total, with no more of a gap of seven days in between each visit.

This reader response data is examined in isolation, so the publisher can’t see whether these people are already Guardian members, for instance, but Moran isn’t concerned with this. “The regular bucket is a proxy for people who go on to form a deeper relationship with us,” he said.

Since April, it has expanded Reader Questions to three more desks: the sports desk; the business desk; and the environment, tech, and science desk. Ideally, each day each desk would add the tool to one article. Within four months, the Guardian hopes to roll it out across all the news desks in the organization, approximately seven. It’s also looking to roll it out to the newsrooms in the U.S. and Australia, too.

“The three months working with the foreign desk has really fleshed out this process,” said Nathan Good, the Guardian’s product manager, adding this was a collaborative effort across multiple departments. “We know the types of questions to ask, what the data looks like and how people can access it; now, we’re working on the different vehicles for answering the questions.”

The Guardian decides the questions to ask and the articles they appear on during its morning editorial meetings. As well as editorial intuition, the Guardian mines the comments section to see whether there are themes in what people are querying. There have been a few kinks: One question, which was by far the top one that people wanted more information on, was Donald Trump’s Syria strategy. But no one, including the Guardian, had the answer.

“The key journalistic ambition of the organization over the next year is to explain ourselves to our readers as best we can,” said Moran. “We’ve started off with small specific news cases, but we’re building something more expandable. Ultimately, these won’t just be 100-word snippets, but the basis of articles.”

Images courtesy of the Guardian