NPR's visuals team is using an analytics bot to measure if people care about stories

Credit: By vlad09 on Shutterstock

How many analytics platforms is your newsroom using? The answer to that question goes back to, or should go back to, what the organisation is trying to measure on the web and how it interprets what every engagement or audience development editor is trying to find a definition for: a story's impact.

Back in November, NPR received a $35,000 grant from the Knight Foundation to develop an analytics bot that would help the visuals team take better action informed by what they measured about their work, but also rethink their goals and definition of success.

"We'd been playing around with alternative metrics for longer than one and a half years and this idea came out of our questioning of what our mission is, why have a visuals team at a radio organisation?", said Brian Boyer, editor of NPR's visuals team.

"We decided that a page view certainly does not tell us that, and the things that we were already measuring as an organisation didn't answer that question. So we set out to try to find other metrics, other things we could measure that would give us a better idea of whether a story mattered to someone.

"Everyone talks about impact in news and at NPR, impact is less about new laws being passed and more about creating empathy and getting people to care about what's being talked about in the news," he added.

Carebot has been tested by the NPR visuals team for the last two and a half months and the current version has been improved and expanded based on feedback from the newsroom, specifically on stories containing graphics and visual elements.

Initially, the bot also looked at some of the metrics the team was already measuring to try and get some relevant meaning out of them, such as dividing the number of Facebook shares of a story by the number of page views.

When a story is published by someone on the visuals team, Carebot starts tracking it and sends regular notifications on how it's performing through Slack. The frequency of the updates changes as the story gets older, so the reporter would receive information on how long people are reading a story for every four hours the day it has been published, and then twice a day for the following two days.

Image courtesy of Livia Labate/NPR

Livia Labate, the former Knight-Mozilla fellow who is developing the bot with NPR, told Journalism.co.uk that Slack was chosen as the delivery medium based on feedback received from the team, who did not want yet another analytics dashboard to dig through.

That being said, anyone who wants to look deeper into traditional metrics can go into Google Analytics and put the time spent on the story in context, as Carebot continues to track the article in the background after the three days without sending notifications to prevent it from overwhelming the journalists.

Image courtesy of Livia Labate/NPR

"One of the things we saw by testing it live is that people started asking Carebot questions," Labate said, "such as how a story posted a few days earlier was doing and there was an expectation that Carebot would go and retrieve that, which is why we built that feature in."

A notification from Carebot would say the average number of seconds people were spending looking at a graphic. This has recently been shifted to median numbers at the team's request, which are also depicted in an actual graphic as the notification comes through Slack.

"What we're doing next is actually looking at aggregation for the first time," said Labate.

"So far we've only been talking about one story or one graphic, so we will also start representing how that story did compared to all other stories in the same category, or all graphics stories," she explained.

"These different metrics are proxies that will help us define the notion of what we've called an engaged completion rate," Boyer added, "so the number of people who actually finish a story out of how many start reading it."

In the past, NPR has tried to gauge if readers cared about a story by manually adding a button at the end of an article or graphic asking them if they liked the article or whether news such as that mattered to them. "It turns out people totally clicked that button," said Boyer.

Because Carebot is currently only tracking stories from the visuals team, there was also a need to come up with some parameters to help define a story in order to make meaningful comparisons.

For example, would it make sense to compare an interactive quiz to a text-based article from the point of view of success? "Probably not," said Labate.

So she implemented a feature into Carebot that allows the tool to track stories' performance better, looking at whether or not they contain graphics, and how readers are meant to engage with them, whether that's by clicking or scrolling.

"We wanted to have the minimum amount of classification necessary and if we can reduce the current set we have, even better," she said.

Newsrooms are addicted to having that one tool they can download or pay for that will solve all their analytics problems. But they don't have analytics problems, they have organisational problemsBrian Boyer, NPR

"But for the graphics team, it might be valuable to know the difference in linger rate between graphics that require interaction versus graphics that are just there to inform people, which is a greater level of granularity in classifying a story and we will uncover the need for that as we go along."

Labate said one the next steps is trying to define what the engaged completion rate looks like for a story that requires users to interact by scrolling.

"We know what it looks like for a slideshow or a click-through story, but on scroll-based we're experimenting with identifying the percentage that you've scrolled down to up to the end of the story but before comments begin, and then if you continue scrolling and actually engage with the comments, that will be counted beyond 100 per cent."

After the development and experimental phase of the project concludes at the end of April, Boyer said he is hoping to keep expanding Carebot not just in his team, but to try it out across different NPR desks.

"What we've been measuring at NPR will certainly be of interest to other people, but what we're building is really based on what impact means for us and how we have found it useful to communicate within our organisation.

"Newsrooms are addicted to having that one tool they can download or pay for that will solve all their analytics problems. But they don't have analytics problems, they have organisational problems and difficulties in defining what success means for them, and that's the question they should ask themselves," he said.

Free daily newsletter

If you like our news and feature articles, you can sign up to receive our free daily (Mon-Fri) email newsletter (mobile friendly).