Defining new metrics for journalism

Greg Linch led a ses­sion in the morn­ing at BCNI Philadelphia about how we can define new met­rics for jour­nal­ism. It was prompted by the sim­i­lar post he wrote ear­lier this year.

To help we need to look at the met­rics other indus­tries are using to define suc­cess in their work. Greg’s idea stems from Alexis Madrigal’s notion that you have to define your own met­rics. You can find influ­ence from oth­ers but don’t just adopt their method.

We have meth­ods for mea­sur­ing how many peo­ple see our work and how long they spend on a site. But, we don’t have solid met­rics for mea­sur­ing the impact of jour­nal­ism. As Greg said, “What does a pageview mean?” The met­rics we cur­rently have mean things. They tell us how many, how long, and other aspects of our work’s impact. But they are largely one-dimensional and don’t tell the whole story.

Current approaches

Altmetrics.org was men­tioned as one dif­fer­ent approach. As their site says:

Altmetrics expand our view of what impact looks like, but also of what’s mak­ing the impact.

Sabermetrics was the focus of the Moneyball movie and is a sim­i­lar approach in sports to redefin­ing the met­rics of an indus­try. Things like VORP and OPS bring new light to the per­for­mance of ath­letes who may appear suc­cess­ful by tra­di­tional metrics.

Propensity score match­ing is yet another approach. Tries to answer the ques­tions of, “What’s the pos­si­bil­ity of X to hap­pen based on Y?”

Greg also metioned the way the Cleveland Orchestra mea­sures per­for­mance. They strive toward supe­rior per­for­mance and mea­sure it by stand­ing ova­tions, ticket demand, and more.

Someone in the audi­ence men­tioned the way Gawker rotates staff. Nieman recently wrote a bit about this. For a dif­fer­ent way of rotat­ing staff check out how The Economist does it.

Ideas

Someone men­tioned that one way to approach this is ask­ing read­ers the ques­tion “Do you under­stand this issue bet­ter after read­ing this article?”

I brought up the way KISSmetrics approaches site stats from a user-specific per­spec­tive. For jour­nal­ism that could be look­ing at how many peo­ple read the first story your pub­lish on an issue and all of the fol­lowups. You could also mea­sure how many jump from one report you’ve pub­lished to another one on a dif­fer­ent topic.

Greg men­tioned the net pro­moter score that many orga­ni­za­tions use to mea­sure the loy­alty and rep­u­ta­tion of a com­pany or a prod­uct. One per­son in the room works at an orga­ni­za­tion which uses this on a per-brand basis. They sep­a­rate the con­tent cre­ation from the con­tent dis­play. Each cov­er­age team is given a net pro­moter score as well as each prod­uct or brand their con­tent is flow­ing through.

The con­cern over not being able to mea­sure impact in com­mu­ni­ties that are not online was also raised. Ultimately the way to fix this is to get out in the com­mu­ni­ties and talk with peo­ple about whether they under­stand issues and how they’d like to learn more.

Albert Sun said that part of why met­rics are so con­fus­ing is because they are tra­di­tion­ally geared toward what adver­tis­ers want. The pre­dom­i­nant approach is, “All met­rics are good but we need to make money off of them.” The busi­ness side is fail­ing, though, and what we need to do is cre­ate a new, bet­ter model. What we need is a way to mea­sure the dif­fer­ence and change our jour­nal­ism causes in the world.

If any­one has more to add feel free to leave your own notes as a comment.