Tag: knowledge systems

Archiving Twitter With WordPress

Yesterday I had a spare couple hours and decided to follow Doug Bowman’s example and set up a self-hosted archive of my Twitter stream with WordPress. You can see the finished product of that here.

There was some interest expressed on Twitter of others wanting to do something similar so I thought I’d help out by making what I did available for download. You can grab a copy of the theme and required plugins which will provide pretty close to a turn key solution for getting this running.

I highly suggest following Bowman’s tutorial for downloading and importing the initial archive of previous tweets. Once you have that done and the plugins and theme are installed there’s a couple things you’ll want to do:

  • Replace the profile_image.jpg file in the theme folder with your own profile image.
  • Head to your profile page within your WordPress installation. You’ll see two new fields, one for the url of your Twitter profile and the other for your Twitter username. These power the text in the header so just fill them both out and the header text will be linked to your profile.
  • The tagline below the username in the header is pulled from your blog’s tagline so fill that out in General tab underneath Settings.
  • Setup Twitter Tools to create a new blog post every time you tweet. You can find more information about doing that at the WordPress plugin directory.
  • Run two queries using the Search Regex plugin (for more info on these queries read the original source). This will link up all the @usernames and #hashtags from your tweets.
    • For @usernames enter /(^|s)@(w+)/ into the Search Pattern field and then enter 1@<a href="http://twitter.com/2">2</a> into the Replace pattern field. Check the Regex box.
    • For #hashtags enter /(^|s)#(w+)/ into the Search pattern field and then enter 1#<a href="http://search.twitter.com/search?q=%232">2</a> into the Replace pattern field. Check the Regex box.
    • In both cases I suggest running a Replace before running Replace & Save. This will allow you to look everything over before making changes that will affect your database.

That’s it. After doing those steps you should have a searchable, self-hosted archive of everything you’ve posted on Twitter. If you run into questions or problems feel free to fire away in the comments.

Update 3/23: Emily Ingram pointed me to a plugin that will achieve the same auto-linking of @replies and #hashtags that the regex calls do. It’s a super simple solution and can be downloaded from the WordPress directory. Sounds like it works quite well.

Making sense from the noise

As the technology to publish on the web becomes more streamlined and the process from brainstorm to publication shortens the ability to aggregate and filter will be immensely important. In order to make sense out of all this information published and distributed in real-time online we need a solid set of tools to filter the important information from the noise.

In a world with widely available online publishing tools we need ways of seeing relationships between data. From a political standpoint the ability to filter for relevant data and relationships is invaluable. We finally have tools developed enough to allow us to aggregate and filter information to find these important relationships. More importantly, the tools that we now possess are in the control of individuals who can now aggregate and filter their own personalized information flows. Among the many data aggregation tools out there two are particularly useful in their application to political participation: Google Reader and Fever.

The modern tools that have developed around aggregating and filtering data are tremendously flexible and powerful. These technologies empower more refined aggregation that allows for a more informed public. Information is the key to well formed political decisions and tools like Google Reader and Fever give the American public the opportunity to partake in more informed political participation.

The technology of aggregation

The mainstay of aggregating data online has, since its invention in the late 1990s, been Really Simple Syndication. This technology, commonly known as RSS, allows for users to automate the subscription process to streams of information online.1 Any site that provides an RSS feed of content can be subscribed to by a user using an RSS client like Google Reader or Fever. After subscribing all content feeds automatically update after publication.

This technology is already a default standard on news sites, blogging platforms, photography sites, and social applications like Twitter. Furthermore, it is an open technology that can be implemented free of charge which means there are more client applications than could be covered. Due to this massive number of client applications this essay will focus upon the two pieces of software that hold the greatest application to political participation in the United States: Google Reader and Fever.

Fever, an RSS reader designed and developed by Shaun Inman, was released in the summer of 2009, It took a different approach to reading information through RSS. While the previous paradigm of RSS subscriptions had been to treat them like an email inbox, Fever approaches the model from a different standpoint. Instead of displaying a list of unread items, similar to an inbox, that only disappear when all are read Fever allows for users to focus on just what is important to them while not feeling like they will miss other important news.

By moving past the inbox mindset of RSS reading, Fever changes the way users discover and read feeds on the web. Instead of having to make a choice between subscribing to a plethora of feeds (thus overwhelming themselves) or subscribing to a select few feeds (potentially missing important news), Fever approaches the problem by encouraging users to make a distinction between essential and supplementary information sources while still subscribing to both. Essential feeds are marked as “Kindling” while supplementary feeds are put into a “Sparks” folder.

These two designations of content are, together, the source of Fever’s most politically important aspect: the Hot List. Shaun Inman describes the Hot List on Fever’s site by saying that,

Fever reads your feeds and picks out the most frequently talked about links from a customizable time period. Unlike traditional aggregators, Fever works better the more feeds you follow.

Fever analyzes the links of sources in both the Kindling and Sparks folders. Through this analysis it then presents the most popular stories as determined by the sources one follows. This Hot List can be narrowed down to a range of days or for the most recent week. Furthermore, not only does it show the most linked to items but it also shows the originating sources for those items. Thus, it allows a user to see the most popular items and how they relate to the information sources that he or she follows.

While Google Reader does not provide the same type of personalized Hot List as Fever, it nonetheless represents an important web application for data aggregation. While Fever presents some compelling features, Google Reader is the dominant market leader for RSS readers. In February of 2007 this market share measured 59% of the web-based RSS reader market.

With this dominant market share Google has integrated a significant toolset of social features into Google Reader. Most important are the personalization features that have been built into Google Reader. The two main aspects of this are the recommendation engine and the individual social tools.

The most politically compelling feature of Google Reader is its recommendation engine. Upon launching the new feature Beverly Yang, a Google employee, wrote,

to make it easier to find interesting feeds, we’re moving recommendations into the new Explore section and giving it a new name — “Recommended sources.” Like always, it uses your Reader Trends and Web History (if you’re opted into Web History) to generate a list of feeds we think you might like.

Simply put, Google Reader has the ability to track a user’s reading habits and use that as the basis for suggesting additional content either popular across the web, or particularly interesting to that user’s interests. This takes the burden of categorizing information sources off of the user. Anyone can start with the information sources that they know they want to read and partially rely upon Google Reader to find similar sources from around the web.

Another important aspect of Google Reader’s features is its ability to share and recommend items to people within your existing social circle. Google Reader allows users to share articles with other users and to comment on these articles when doing so. This allows for users to discover new articles and new sources of information by either leveraging the algorithm behind Google’s recommendation engine or through their social network of friends, coworkers, and contacts.

One final feature important to keep in mind about Google Reader is the way it tracks a user’s reading habits and displays this data in accessible charts. These reading trends allow any user to automatically see which information sources garner their attention most consistently. Ultimately the ability to track reading habits without having to rely upon what one remembers reading can allow for a user to analyze their own reading habits and perhaps restructure their information intake accordingly.

Overall, two key aspects of Fever and Google Reader are important to keep in mind for a discussion their political potential. Fever provides analysis of information sources already familiar to a person. This allows them to sort through massive amounts of information and leverage an algorithm to filter for importance. Google Reader, then, provides the ability to use a social network of contacts as well as an algorithm to find new information flows and news items. Most importantly from a political standpoint, all of these technologies are able to used and structured by an individual user. Individuals determine the structure of an information flow in both Fever and Google Reader.

Translating aggregation to politics

In a political system grounded in the involvement of the general populace the relative education and knowledge of the citizenry is crucial to the legitimacy of the political decisions made. Thomas Jefferson recognized this when writing “A Bill For The More General Diffusion of Knowledge.” In this he writes that,

even under the best forms, those entrusted with power have, in time, and by slow operations, perverted it into tyranny; and it is believed that the most effectual means of preventing this would be, to illuminate, as far as practicable, the minds of the people at large.

Jefferson saw the education of the citizenry as a foundational guard against the threat of tyranny. Having what he terms an illuminated citizenry is the best protection against undemocratic elements that would oppress a people. Having the ability to aggregate information sources together using tools like Google Reader and Fever then provides many opportunities for citizens to take the initiative and inform themselves of political matters. The test of how well these technologies expand avenues for political participation, however, relies heavily upon the ability of users to filter out the important information from the irrelevant noise.

In order for filtering through online applications like Fever and Google Reader to be more effective at informing the citizenry than traditional media we need a solid technological response to an online world in which the many can publish. In The Wealth of Networks Yochai Benkler refers to early critiques of the democratizing effects of information online by describing the Babel objection. In Benkler’s words,

According to the Babel objection, when everyone can speak, no one can be heard, and we devolve either to a cacophony or to the reemergence of money as the distinguishing factor between statements that are heard and those that wallow in obscurity.2

A cacophony in which all publish but none consume would certainly represent an issue for a political system grounded in common movements for change. In a democratic political structure where majorities, whether they be citizens or representatives, determine decisions the ability of people to converse with one another about topics of mutual concern is paramount. In order for this conversation to happen we need some element of information consumption; people need to have consumed information to have a basis for discussion. In short, we need consumers of information and sharers of information just as much as we need the publishers.

Despite the critical importance of ensuring the consumption of information, Benkler views the cacophony of an open publishing world as not inherently debilitating. He writes that,

The Babel objection may give us good reason to pause before we celebrate the networked information economy, but it does not provide us with reasons to celebrate the autonomy effects of the industrial information economy.3

[blip.tv ?posts_id=1283699&dest=-1]
While the concern over a world of pure projection is important it should not be so debilitating that it makes political life in the United States complacent. In a talk at the Web 2.0 Expo in New York in 2008 Clay Shirky points to those proclaiming the disastrous effects of a world in which everybody speaks as missing the more significant, underlying problem: the filter. In this talk he says that,

Thinking about information overload isn’t actually describing the problem and thinking about filter failure is.

In an online reality where the many possess the ability to publish the weight of keeping the public informed and, in Jefferson’s words, politically effectual is not a problem of too much information, rather it is a problem with a filter that is not refined enough. With Google Reader and Fever we have tools that become increasingly more effective at filtering out the noise in information streams and allowing citizens to more efficiently stay informed. By leveraging the recommendations detailed above, Fever’s link analysis or Google Reader’s socially-powered recommendations, it becomes easier for people to take in a wide array of information and filter it for the most politically important and relevant material.

A final aspect of aggregation technology that improves the potential avenues of political participation is the way that Google Reader, Fever, and any other RSS reader pools together various sources into a single, contained information flow. In an information economy where print signifies the most technological option available information comes to people in distinct packets. The raw form can come in primary source documents that present something closer to plain data while the aggregation happens through secondary source material where the author must carefully aggregate and curate the information referenced. Data aggregation through RSS places this power in the hands of the people. Any individual can subscribe to feeds of various news or data sources and have all of that information flow into a central repository. Lastly, when combined with the type of recommendations and social features from Google Reader or the Hot List feature of Fever this places the power of finding relationships between information sources in the control of an individual. Each individual who uses a tool of data aggregation can compare various sources to find similarities, differences, and contradictions. Ultimately it greatly increases individual autonomy by allowing each person to receive information straight from the source and serve as their own filter without having to trust in the filtering abilities of a third party.

Aggregated Participation

Aggregated raw information and information recommendations from a personal social circle and a refined algorithm expand the potential information to which each person has access. Individuals can now personalize an information flow and leverage their wide ranging social networks, as well as the skills of software developers, to find additional sources of data. All of this serves an important function for expanding access to data and information, both of which serve as key foundational elements to political participation in the United States. These technologies can provide something greater though as well. Not only can they aggregate information and increase the information flows used as a basis for participation but they can also aggregate and filter participation.

The acts of political participation occurring through the real-time web combine with the maturation of self-publishing tools and advanced aggregation technology to provide a powerful redefinition of public political participation. This definition hinges upon individual activity aggregated in the collective to find relationships between opinions that help spur political action.

The most important aspect of being able to aggregate expressed political opinions using these technologies is the accessibility of these tools to the individual. As Dave Winer, one of the founding developers behind RSS, says in the aforementioned BBC article,

RSS makes it possible for information to flow to you.

The content of this information can be anything but ultimately it all flows to a single individual who then can make judgements based upon it. This is not political information streaming through an interest group’s filter. It is not news being restricted to what passes the mainstream media filter. It is not political opinion coming down to citizens from a presumably trustworthy representative. Rather, it is information flowing directly to those who ultimately have to make political decisions: individuals.

The independence of individual judgement and information intake is important for participation to effectively be extended to a mass of individual citizens. By allowing individuals to aggregate their own information and come to their own judgements concerning it the political system can become reflective of what individuals actually desire from their government. Instead of relying upon a mass media outlet to aggregate and curate information users can now use something like Fever’s Hot List to independently view what the important stories of the day, week, or month are. Furthermore, with all of the social features packaged into Google Reader any user of that RSS reader could quickly and easily leverage his or her existing social network to inform their decisions.

Finally, all of this individual control over information consumption contributes to a political arena in which individuals are actually informed about the political actions they take. Information plays a central role in political actions. Whether that action is organizing a movement for political change, voting for a candidate, or simply discussing political issues over with family members information lies at the center of everything. The ways in which individuals gather and sort this information signals an inherently political act. That information informs political participation and discussion.

By placing the onus of informed participation upon the individual these tools allow for individuals to come to their own conclusions concerning information. A greater reliance upon a personally customized river of information means that individuals can come to rely upon other packaged versions of information less. By cutting out a middle step of interpretation individuals can learn to process ideas and knowledge themselves in such a way that it informs their participation in politics.

For a democratically-based political system like the United States the freedom and independence of individuals is vitally important. Modern tools of data aggregation like Fever and Google Reader provide the ability for people to take a faster flowing stream of data published by the many and turn it into their own, independent source for informed participation.

  1. This process is effectively explained in a 2005 article published by the BBC.
  2. Benkler, 10.
  3. Benkler, 171.

News as Software

Lately there have been a couple ideas bouncing around in my mind about news. To commit them to memory I wanted to write them down here. What follows is a rough outline of how I would structure a news organization’s online presence. These are by no means polished ideas but are first passes at a conception of journalism’s future.

The driving point behind all of this is an idea that I’m calling News as Software. What this means in my head is that news organizations need to start adopting some of the approaches that have been so successful for software developers on the web.

Use not Consumption

News organizations need to begin promoting the use of their product instead of its consumption. The traditional print product fit very well as a unit of consumption. Readers could sit down and consume the information. After reading whatever percentage of articles interested them there was not much left to do with the news. At least in my house the old product simply becomes paper to start fires with in the winter.

News as software requires a fundamentally different mindset. Software provides a sense of utility to users. It does something for them. More importantly it does that function over and over. Granted, this would be difficult to do but the first step is to break down the idea that news arrives in an organized package.

Experimentation and Play

Closely connected to this idea of news as software that people use time and again is the ability for users and developers to experiment and play with content. For the technically inclined among us Twitter is far more useful because of its API than it ever would have been as a limited website.

The API of a web service is what transforms something ordinary in something magical. The fact that I can use any number of client applications, or even other web apps, to read and post to Twitter is a testament to the ingenuity behind it all. By providing a platform from which users and developers can customize an experience Twitter has given us a service that we can customize to our liking.

Contrast this with any major news site. The New York Times simply does not allow for its users to play with the news. Sure, they’ve made initial attempts at doing so with things like Times People and the Times Skimmer but ultimately the content is staying within the New York Times packaged site.

Help your users

Every successful web app and desktop program I can think of has a thriving online community of users who help on another. Generally this is everything from fixing bugs in the software to promoting some of their favorite tips and tricks. Why doesn’t this hold true for mainstream news organization sites?

The New York Times has a “Help” page. You find it by scrolling all the way to the bottom and finding the link in about 11 px type. Perhaps this works for finding help with subscription related information or other things but it fails at stimulating the type of community that’s present in many web apps.

If I were a user looking to restructure the manner in which I consumed information from any mainstream news organization where would I go to find out about how others do it? Where could I engage with other readers about ideas for using the news? If there is any place that allows for this kind of thing (no, not Twitter, I want it on the news organization’s site) then tell me but I highly doubt it.

What do we do?

The current design of mainstream news sites is what I see as the biggest stumbling block toward this conception of news as software. Take a look at the New York Times, Washington Post, or The Guardian websites. Every one of those sites presents content in an already organized format.

Loading nytimes.com for example brings up a static grid of content. Once a user finishes looking over that content there is no sign of when new content will become available. There are very few ways for the user to customize or play with the content as well.

Furthermore, there is no method of tracking what content readers have already seen. Thus, were you to visit nytimes.com 2 hours from now there would be no way to tell what you had already seen. Some of the same stories will be exactly where you left them, others will have simply shifted around the page. To inspire people to start using a news site we’ll have to completely reorient the design of sites.

Currently, everything is prepackaged. Perhaps this makes things easier but somehow the success of services like Twitter, Google News, and others tells me that people like to have control over what information they see, where they see it, and how detailed it gets. Without inspiring creativity there is little reason for users to become attached to a news site. They all offer the same thing: prepackaged content organized by an editor with little connection to the millions of users. That’s not software, it’s shovel-ware. That’s not what people enjoy using and it’s certainly not what they are going to pay for.

Building off of a coral reef – a Whitman blog network

In the most recent episode of Dave Winer and Jay Rosen’s tremendous podcast titled “Rebooting the News” Dave mentions the metaphor of treating journalism like a coral reef. Basically the gist of the metaphor, as I understand it, is this: if you can put in the initial work to start covering many different aspects of a community then pretty soon that community will become involved in building and giving life to that local news site.

Daniel Bachhuber posted his thoughts on this a couple nights ago and expanded upon the concept in terms of data and information. This all got me thinking about how this concept could be applied to a small college paper like the Whitman Pioneer. As part of my work this summer on redesigning The Pioneer’s site I’m working to create a foundation from which the paper can build features in the coming years. Part of this involves creating a framework that can integrate and inspire community engagement. What better method of doing this than creating a “coral reef.”

Here’s what I’m thinking: Right now we have a few various blogs that are written by staff writers. Most of them are not updated all that frequently and those that we have essentially serve as beat blogs and expand the coverage of events and topics that The Pioneer already covers. What we don’t have that I think could prove to be incredibly interesting and engaging is a network of blogs written by other members of the Whitman community.

I know there’s other students, professors, etc. that have blogs but I have no idea what the urls are; consequently I have no easy way to find out what they’re writing about. What I’m envisioning is creating a Pioneer blog network where we do two main things:

  1. Aggregate the content from personal blogs – This could be anything from someone’s Tumblr to a WordPress or Blogger site. They would just go about posting content in the normal way and we would aggregate it on our site and provide links and some information so that The Pioneer’s readers would be able to get a better sense of what “regular” Whitman students are thinking.
  2. Provide a framework for additional blogs – Not all students are tech-savvy enough to have their own personal domain, or motivated enough to start their own blog on WordPress.com or Blogger. What I would propose here is that The Pioneer provides a blog for any student interested. These could be hosted on our domain and run custom installs of WordPress. I would put together a collection of themes (both self-designed and from the WordPress community) and they would be able to install any of those (or anything else they find) and all the plugins the want. Their blog could then be hosted at a sub-domain of The Pioneer’s site (something like http://blogs.whitmanpioneer.com/joeblogger).

What I see this as accomplishing is twofold.

For one, students (both prospective and enrolled) would have a place to turn to find the recent views of their classmates. We currently feature columns from students studying abroad but there are far more that keep blogs during their time in various countries. How cool would it be to have a place that aggregates all of this information about the Whitman experience?

Second, I see this as providing a forum through which both parents and the community in general can become more involved in the views expressed and discussed. There’s always a lot of talk about how Whitman students go to school in a bubble and I think this could help to combat that perception. It would provide a well-trafficked site through which they could express their opinions on issues spanning sports, politics, community service, crime, etc. It would also give a direct line for community members to interact with students. They would have a place to turn to find what Whitman students are saying and consequently would be able to easily add comments and their own perspectives.

Anyway, those are just my quick and very much rough draft thoughts on the matter. I’m sure the idea will grow the more I think about it in the coming days. I’ll also be working on posting a wireframe for the page in the coming weeks and we’ll see how feasible that is. What I’m wondering is what you think: Sound like a good idea? Think it’s awful and a waste of time? Do you know of any schools or news organizations that are attempting something similar?