How Nature and TBIJ measure the impact of their journalism

Credit: Photo by Diana Polekhina on Unsplash

Editorial teams are swamped with metrics these days, each one telling us something about how readers interact with our work. An industry-wide shift has seen us turn from reach and page-views to engagement, building habit and in many cases growing reader revenue.

But what about what happens after people read the article and, if we are lucky, comment, share or pay to support our journalism? What happens out in the real world?

More newsrooms are adding roles or responsibilities focused on impact-tracking, and Journalism.co.uk spoke to three people in this space to find out what other newsrooms can learn from their work: Grace Murray, an impact producer at The Bureau of Investigative Journalism (TBIJ); Anna Nowogrodzki, a science journalist who recently completed a one-year term as impact editor for Nature; and Lindsay Green-Barber, founder of Impact Architects, which supports media organisations in measuring impact, including through its Impact Tracker.

Defining impact

When beginning to track the impact of an organisation's work, the first step is agreeing on what definitions make sense in the context of that publication.

Impact Architects splits impact into four broad categories: structural, community, and individual change, as well as media amplification, and these can be further broken down (structural change could be an institution changing how it works, or a law change, for example).

Some publications or teams develop their own definitions or categorisation, based on what makes sense for their own context.

Nature uses 12 impact categories which reflect its audience of science professionals, including "educational impact" and "influenced the direction of scientific research". At TBIJ, different teams put the focus on different forms of impact, with the Enablers team for example - which investigates how networks of lawyers and advisors are enabling oligarchs - prioritising policy impact in Westminster.

Nature's Impact Tracker dashboard

While both Murray and Nowogrodzki said that policy impacts were often very compelling, they both also appreciated learning of individual impacts - exactly the kind of outcomes that may go unnoticed without dedicated tracking.

For example, a TBIJ story on the grant application process for making housing more accessible for people with disabilities had the result that a local council sped up a process with one of the people featured - the team plan to follow up with them in a few months to confirm the outcome. And after an article about hate speech in Ethiopia, the fact-checking organisation interviewed for the article said the coverage had helped to confer legitimacy to them and boosted their morale and profile.

At Nature, alongside reporting that catalysed further research, law changes, and the expulsion of known sexual harassers from the US National Academy of Science, the impact example that made the greatest impression on Nowogrodzki was the reaction to a photo profile of a Uruguayan chemist.

María Fernanda Cerdá spoke about her experience in a male-dominated field, and following the story, she received funding for her research, and received messages from interested collaborators as well as an outpouring of support from women working in science in Uruguay and elsewhere in South America.

So how do you balance the more classic examples of impact with those which are harder to quantify?

Green-Barber, Murray and Nowogrodzki all agreed that impact 'scores' are not helpful - there is no formula such as 'ten media impacts are equivalent to one policy impact', and different types of impact will carry more weight with different people and on different stories. Some types of impact, for example interviewees giving positive feedback about a publication's editorial process, are valuable internally but not relevant to readers.

Why measure impact?

The obvious motivation for tracking the impact of journalism is to check if our reporting is indeed creating change in society. It gives reporters a different way of tracking articles' 'performance' - one that may not be as easy to put into numbers as page views, but which closely reflects the goals of many mission-driven publications.

"Having a clear way to track and measure impact that goes beyond ad metrics can demonstrate the value of this important and nuanced work, potentially creating more resources for this type of reporting in the newsroom," says Green-Barber, founder of Impact Architects and an expert in media strategy,

In fact, Nowogrodzki said her team had found several articles that had relatively low numbers of page views but had generated important policy impacts. This helps show reporters the value of their work and can inform editorial decisions - whether the outcome is simply continuing to commission these stories, or to adjust how they are told so that reporting which is known to be impactful also reaches a wider audience.

"I think it is important for journalists to take a moment and think about why we do what we do. Especially journalists who are minorities or women, we get a lot of negative feedback, so I think it's important that we get positive feedback as well," adds Nowogrodzki.

There is also a very practical reason to find out and inform people about the changes your journalism prompted: showing concrete examples of impact can be useful in applying for grants and updating donors. Global Press, ProPublica, and The Marshall Project are examples of publishers which address impact in their stakeholder reports.

These examples could also go some way to rebuilding trust in news, by demonstrating the value of good reporting in a clearer way than any marketing campaign could do.

As part of her year-long role at Nature, Nowogrodzki produced 'impact stories' illustrating outcomes of their reporting to readers, with an audience survey showing that 80 per cent of readers found these stories valuable.

Finding and tracking examples of impact

"We have found that the best source of compelling impacts are our own reporters and editors," adds Nowogrodzki. "We can’t build a robot that scrapes the web and automatically collects impact. It would give you way too much unhelpful information."

Nowogrodzki also carries out her own research, including searching for citations of articles in policy documents - here Nature has an advantage as all articles have a DOI (digital object identifier). And occasionally the publication has directly asked readers if a Nature article has led to a change for them, as part of surveys and in newsletter callouts.

Likewise, Murray regularly contacts TBIJ reporters to collect impact examples, and she says reporters increasingly come to her proactively, having seen how these are used to inform reporting processes. The organisation has recently begun building follow-up interviews with sources into their workflow, a few months after the initial reporting, and this is something Murray wants to continue.

"We want to find out the impact either way - it's great to find out when something happens but it's equally as valid and important to know that something didn't happen," she adds.

Nature's Impact Tracker - which recently won an INMA Award for Best Practice - was a multi-year collaboration between Nature and Altmetric (which built the tracker). It was first conceived by Anna Jay and Helen Pearson and funded by the Google News Initiative.

Another widely used tool is Impact Architects' tracker (originally developed by the Centre for Investigative Journalism) which TBIJ is about to begin using.

Impact Architect's Tracker

Technology brings advantages such as clear data visualisation, accessibility and the ability to spot patterns easily, but Nowogrodzki says that any newsroom could begin tracking impact by using simpler technology, such as a Google Form. The crucial step is finding a way to integrate impact reporting into daily work, and making sure journalists can see that carrying out this work generates useful insights or resources for them.

"If you just tell people to enter impacts, well, everyone is very busy and nobody has time. It’s not meant to be busywork, it’s meant to be actually valuable," she explains. Equally important is having people to champion the role of impact, and she says that Jay and Pearson's efforts before the newsroom set up the full-time impact editor job were a big part of the project's success.

Using impact

Like any kind of metric, impact data is only useful if it is actually used.

"In best case scenarios, impact tracking is distributed across the newsroom and the insights gained feed back into editorial processes around story selection, production, distribution, and audience engagement," explains Green-Barber.

Nowogrodzki's first task as impact editor was to make the database user-friendly, and now it can easily be searched by author, date, editorial section, or by impact type such as 'policy' or 'education'.

She notes that it notes that it is still "early days" for the impact tracker at Nature and editors are developing ways to put insights into action, but the positive response to the impact stories from readers shows that there is a value to sharing these outcomes directly with the audience.

It has also confirmed certain editorial decisions. Increased coverage of diversity and inclusion topics, originally motivated by the organisation's values, has also been seen to generate compelling impacts, which helps make the case to continue this reporting.

At TBIJ, impact is something the team think about at every stage of the reporting process.

Each story has an impact strategy alongside the reporting strategy, which means the newsroom plans how much resources to devote to 'impact actions' such as contacting NGOs, briefing politicians, or meeting with community groups, and also decides what those actions could be.

"If we want our story to get in front of a particular audience, then we think at an early stage about how we want to tell that story," says Murray. "Some of this is just basic tenets of good journalism, but having the impact process is about making sure it actually happens."

In a recent investigation in the global health team which focuses on tobacco advertising in Peru, it was important to work with a local Peruvian partner (TBIJ uses a partnership model where they publish with partners rather than to a direct audience, so choosing the right partner is one way of maximising impact).

For some stories, this includes a systems mapping exercise where reporters will talk to a group of stakeholders (for example, subject experts, people from a directly affected community or community group), and map the context of the issue - relevant groups, laws, the history of the issue, and other important background.

"It's good to know where the pressure points are when you go into a story, even if we would never say what should happen - it's always more about the 'could'," she explains.

TBIJ does not work towards any specific impact and draw a distinction between their mission-driven work and advocacy.

"We want to see our stories bring about positive change without being prescriptive about what that change is. We provide the diagnostics but it’s up to other people to provide the cures," concludes Murray.

Join the Journalism.co.uk news channel on Telegram to receive news and updates straight to your phone every week

Free daily newsletter

If you like our news and feature articles, you can sign up to receive our free daily (Mon-Fri) email newsletter (mobile friendly).