Thinking about a data-driven college

In an effort to start tracking some of the ideas I have while reading I want to start making note of ideas and questions that come up here. This is the first of such posts and we’ll see what form they take in the future.

Tracking my book reading

Interesting article that examines some of the frustrations with current systems for tracking reading habits. Since I just finished writing an article for The Whitman Pioneer about open knowledge systems this got me thinking:

  • What if colleges started working together on building an open standard for tracking reading? I’m thinking of a system that would get me set up as a Freshman with a way to keep track of every article, journal essay, and book that I read while in school. Then, when I graduate I can either move the system to my server, or the college provides an export file to import into various other services. If I could go back four years and be presented with a choice between a school that had this system and one that didn’t I know which I would pick in a heartbeat.
  • Could we conceive of a service that would not only track reading but track conversations about books? What if I could record conversations with others about a book and upload them to a service, forever associating that conversation with that reading experience?
  • What good is it to track book titles and authors if I don’t also have a canonical, searchable copy of that book online?

The Data-Driven Life

Long feature piece from The New York Times about the various ways people are tracking data about their everyday lives. It turns out that seemingly mundane things can offer remarkable insight into how our minds and bodies work. Couple points about this:

  • All (unless I missed one) of the services mentioned are owned by single companies. Some, in the case of Nike+, by massive corporations. I think there’s a huge opportunity for someone to come up with an open source data tracking system that allows users to own their data. Follow up: what happens to all this wonderful, data-driven insight when these companies go out of business?
  • How can we tie this data-tracking to business interactions? What ways could I track data that would reveal the companies that most consistently affect my day in a positive way?
  • Academically, it’d be interesting to track attention during a semester-long course to see which subjects and discussions were most captivating.

Questions about the current state of knowledge management systems

Next week is the second iteration of BarCamp NewsInnovation Philadelphia. One of the ideas for a session is discussing the current state of knowledge management systems. Daniel Bachhuber describes this as:

how news organizations manage all of the data they’re privy to that is either stored in structured format or could be stored in a structured format if they had the tools to do so.

In preparation for the session there’s a thread going on Hacks/Hackers about what could be covered in such a session. Since I can’t make it out to Philly I wanted to outline my thoughts here.

I’m interested in three broad ideas about the way a knowledge management system could be effectively deployed in a news organization.

Cross-platform tracking of information

  • Can knowledge be tracked in standard formats so that a news organization’s KMS is valuable to non-news organizations as well?
  • What would it look like for various newsrooms to aggregate and integrate what is contained in their KMS? More importantly what would it look like if we had a system of standards-based KMSes from various fields that could be plugged into each other? What would the role of a news organization be here?

Role of a KMS in mobile

  • How can we present all of this data in a way that not only works for the desktop environment but is also discoverable enough for mobile users?
  • What forms could a KMS take that makes information even more relevant to mobile users?

Role of a KMS in ongoing coverage

  • What are the ways that this structured knowledge repository can be used to analyze and make adjustments to a news organization’s coverage?
  • Can tracking user interaction with the products of a KMS help us to create more (in both a quantitative and qualitative sense) journalism?
  • Do the views of a news organization’s topics of importance mesh with a community’s?

For those interested in this session I would also recommend this discussion with David Siegel about the semantic web and the notion of a pull economy of information. This post from “the human network” is also worth reading as a background for the discussion.

Those are just some brief thoughts for now. Wish I could be there in Philly but I look forward to tracking the conversations on Twitter.

Archiving Twitter With WordPress

Yesterday I had a spare couple hours and decided to follow Doug Bowman’s example and set up a self-hosted archive of my Twitter stream with WordPress. You can see the finished product of that here.

There was some interest expressed on Twitter of others wanting to do something similar so I thought I’d help out by making what I did available for download. You can grab a copy of the theme and required plugins which will provide pretty close to a turn key solution for getting this running.

I highly suggest following Bowman’s tutorial for downloading and importing the initial archive of previous tweets. Once you have that done and the plugins and theme are installed there’s a couple things you’ll want to do:

  • Replace the profile_image.jpg file in the theme folder with your own profile image.
  • Head to your profile page within your WordPress installation. You’ll see two new fields, one for the url of your Twitter profile and the other for your Twitter username. These power the text in the header so just fill them both out and the header text will be linked to your profile.
  • The tagline below the username in the header is pulled from your blog’s tagline so fill that out in General tab underneath Settings.
  • Setup Twitter Tools to create a new blog post every time you tweet. You can find more information about doing that at the WordPress plugin directory.
  • Run two queries using the Search Regex plugin (for more info on these queries read the original source). This will link up all the @usernames and #hashtags from your tweets.
    • For @usernames enter /(^|s)@(w+)/ into the Search Pattern field and then enter 1@<a href="">2</a> into the Replace pattern field. Check the Regex box.
    • For #hashtags enter /(^|s)#(w+)/ into the Search pattern field and then enter 1#<a href="">2</a> into the Replace pattern field. Check the Regex box.
    • In both cases I suggest running a Replace before running Replace & Save. This will allow you to look everything over before making changes that will affect your database.

That’s it. After doing those steps you should have a searchable, self-hosted archive of everything you’ve posted on Twitter. If you run into questions or problems feel free to fire away in the comments.

Update 3/23: Emily Ingram pointed me to a plugin that will achieve the same auto-linking of @replies and #hashtags that the regex calls do. It’s a super simple solution and can be downloaded from the WordPress directory. Sounds like it works quite well.

Participation Through Publication

As communication online continues to grow we must ensure that there are solid tools providing all with the ability to publish their voice. The ability to make one’s opinions known in a public forum is a requirement of a democratic political system. This right can be traced all the way back to Athenian democracy. Under this system all citizens came together in the Ekklesia to discuss and vote on issues of political importance.1 This can be seen in traditional spaces like town hall meetings, political rallies, and in newspaper editorial sections. The expansion of a desire to make one’s opinions known online signals the most recent manifestation of citizens’ desire to make their thoughts known in a public forum.

The current software available to people wanting to publish online allows for remarkably powerful publishing to occur. Numerous professional-level platforms are offered to any user for free. These tools allow for users to publish their thoughts through free and easy to use software in a public-by-default manner. Furthermore, a growing selection of tools allow for people to publish to a global audience from nearly anywhere. A stationary location with a full-featured computer is increasingly no longer a necessity to partake in online publishing. The ability to publish has been extended to anybody with a mobile phone.

The modern tools that have been developed for publishing online give more people a greater ability to make their voice heard from an expanding range of places. WordPress and Twitter take the ability to publish online and make it something that is accessible to a greater portion of the population. The political potential of the millions of people expressing their voice online can have a tremendous expansionary effect on participation in United States politics. Continue reading “Participation Through Publication”

Real-Time Politics

Communication has been able to happen nearly instantaneously over the web for years now. Technologies like push email have previously opened channels through which information can be transmitted in real-time. Today’s real-time web are different because of the public-by-default nature of messages. Communication through tools like Twitter allows for people to communicate in a matter of seconds and creates a public facing forum that allows any other user to add their voice to the discussion. The public nature of all this communication means that now any person can instantaneously communicate with any leader (be that politician, celebrity, or renowned professor) and engage in substantive discussion.1

How did we get here?

The year 2006 can be seen as an inflection point for what is now termed the real-time web. That year Twitter launched. Suddenly what we had grown accustomed to with email (waiting a few minutes for an update to arrive) seemed like an eternity when there was a service that provided for updates to stream in microseconds. The fact that Twitter limited these messages to 140 characters came to be overshadowed by the sheer rapidity of information transmission. The real-time web became less about reflecting with examined thoughts and more about spreading what was happening right now.

This trend toward short, instantaneous updates has continued with the launch of FriendFeed in 2008 and the open sourcing of its base web server technology (known as Tornado Web Server) in 2009.  A single company owns the technology behind Twitter but the server technology that powers FriendFeed differs. FriendFeed accomplishes the same rapidity of flow that Twitter popularized but does so with a web server that is open. This means that any developer can use the base layer of technology that FriendFeed open sourced and leverage it as a platform from which any forum for real-time communication could be built.2

These technologies provide a stream through which information can spread globally at an unprecedented rate. Messages can be sent, replied to, and echoed by millions of users within seconds. Most importantly this information is not limited in subject matter. The flow of information makes no distinction between a celebrity death and news of electoral protests in Iran. One service ends up being the focal point for news about the latest celebrity gossip as well as the locus for breaking political and economic events. Judgement is not made about the information that passes through Twitter’s channels, the channels simply exist to broadcast that information as quickly as possible to an audience that is now in the tens of millions.

This lack of distinction made between messages posted on Twitter arguably does add to the noise and presence of non-political information; however, this should not be seen as detracting from its political importance. Later, we will see how modern tools for aggregation are allowing for individuals to filter out the noise, but the mere presence of noise is a political benefit. If tools like Twitter were to restrict published information they would be making an explicit statement upon the political nature and source of information. In an open political society the judgement as to what constitutes noise must take place after publication and, thus, after everybody is able to let their voice be heard. Anything else restricts political dialogue that prevents certain people from participating.

The speed at which all types of information can be disseminated holds tremendous political potential within the United States. Our current political structure has served us well in an age when information traveled through a few select channels that were broadcast throughout the country as part of commercial media companies. As citizens we understood that we would have to wait for the nightly newscast or the morning’s paper to find out about the day’s important events. These media kept us informed in a world where news traveled in hours.

The instantaneous dissemination of information is a reality in 2010 and political participation needs to be reframed in order to take advantage of these tools. Ultimately the real-time web has created an ecosystem of communication that can be used to expand and redefine political participation. In an era that prizes the now, political participation must be reconceptualized as a continuous process.

These technologies are being leveraged to create a forum in which citizens can express their opinion at anytime. The political system of the past segmented participation to occur once every year, or even once every four years. Participation in a real-time political system allows for citizens to be involved every month or week, or possibly every day.

What is the real-time web?

In order to understand the political ramifications of all this technology we must first understand the real-time web. In August of 2009 ReadWriteWeb published a three-part series of articles explaining various aspects of the real-time web. In the first part Ken Fromm writes that the real-time web is,

a new form of communication [that] creates a new body of content [which] is public and has an explicitly social graph associated with it.

This characterization embodies the core of what these technologies accomplish. Twitter and the technology behind FriendFeed embody a combination of the elements outlined by Fromm. FriendFeed and Twitter have an inherently social element to them and both have allowed for a new form of communication that has effectively created a new body of content that did not previously exist. When these technologies are combined with the three elements of the real-time web that Fromm describes the potential arises to achieve a notion of political participation defined by constant citizen involvement.

Real-time politics

Three key areas of this technology hold the greatest impact in terms of political participation. A web that allows for instantaneous communication through the mediums detailed above redefines traditional notions of group formation and the political impact of direct citizen input. These two concepts will be explored at length below but in general the real-time web holds the potential to so drastically shift our conceptions of these actions that a radically different notion of political participation is needed.

The way in which groups form and eventually disband is an aspect of the modern American political system that fundamentally differs in a world where Twitter and FriendFeed exist. Politics in the United States has long been about gathering people together through shared opinions and concerns. In the early years of the nation this was primarily done through political parties. Thomas Jefferson wrote of the process of party formation and division in a letter to John Adams on June 27 of 1813,

Men have differed in opinion, and been divided into parties by these opinions, from the first origin of societies and in all governments where they have been permitted freely to think and to speak.3

While political parties characterized groups formed around shared opinions in the early years of the nation, more recently we can see this same effect in such organizations as, PETA, and the NRA. These interest groups arose out of situations in which political parties are no longer affective enough for citizens. Writing in the early twentieth century P.H. Odegard claimed that,

direct democracy falls down in the face of increasing numbers. The individual plain man, swallowed up in a sea of highly differentiated human beings, finds it necessary to organize with others of a like mind so that by concerted action they may bend the state to their will…It is this situation which has engendered the pressure group.4

These pressure groups, now better known as special interest groups, were the twentieth century’s solution to the problem of scale in a democracy as large as the United States. Throughout the last century not every citizen could realistically make his or her claims upon their government. As such they came to band together just like Odegard describes. The result was organizations like PETA and the NRA that mobilize people behind common interests for shared political action.

Not only do interest groups serve to mobilize citizens but they also play a large role in informing their political views. Phillip Agre writes in “Real-Time Politics: The Internet and the Political Process” that,

Political parties and legislatures, for example, do not simply transmit information; they actively process it, especially by synthesizing political opinions and interests into ideologically coherent platforms.5

The role of groups like, PETA, and the NRA as information centers makes older interest groups outmoded. With how information and communication flows on the real-time web these old institutions and structures no longer represent the most efficient outlets for information. In addition, as will be covered later, the reliance of citizens upon interest groups’ ability to process information is no longer a necessity.

The real-time web provides a toolset that alters the role that organizations like play in political mobilization. Furthermore, the technology behind the real-time web provides a partial solution to the problem of scale inherent in twentieth century efforts to involve a greater percentage of the populace in the decision-making process. Finding effective means toward disseminating political information for the goal of organizing political actions no longer hinges upon the abilities of interest groups. The real-time web allows for individuals to track flows of public information on their own and modern tools of data aggregation allow them take control of the processing of this information as well.

Defining participation through the real-time web

Group organization and action is another foundational aspect of politics that becomes transformed by communication through the real-time web. Clay Shirky writes in his recent book, Here Comes Everybody, that,

Group action gives human society its particular character, and anything that changes the way groups get things done will affect society as a whole.6

Shirky holds that group action represents a vital part of not just politics, but human society in general. The development that Shirky points to as creating change in group action is the same social graph that Fromm characterizes as an inherent part of the real-time web. Shirky claims that with tools based around social interaction,

We now have communications tools that are flexible enough to match our social capabilities…we are living in the middle of a remarkable increase in our ability to share, to cooperate with one another, and to take collective action, all outside the framework of traditional institutions and organizations.7

This increase in our ability to share and cooperate with one another forms the basis for a conception of political participation not constrained by the same problems as that of the twentieth century. When American citizens organized together over the last 100 years they largely did so under the auspices of special interest groups.

These interest groups were organizations that were governed by a board of directors or a similar group of full-time employees working in the best interest of the organization’s many members. This structure mirrors that of the political system at large where citizens communicate with their representatives through well defined channels.

Previous writers have remarked that the breakdown of these channels may hold negative ramifications for democracy. Writing in Radical Democracy and the Internet John Downey explains that,

The public sphere might be both more participative and deliberative [as a result of online communication] but there might not be a democratic bonus if the channels between the public sphere and representatives are severed.”8

What Downey fails to realize is that disruption of traditional channels does not necessitate complete destruction. The mass availability of the ability to communicate in real-time through any number of mediums means that anybody, from an individual that makes up the “public sphere” to a city mayor, can participate. The real-time web only destroys the connection between the public and their representatives if their representatives fail to adapt to a changing landscape of communication.

With information from millions of users being transmitted every minute only a small portion of that information needs to be political for its ramifications to be widespread in American politics. Twitter and the open-source technology behind FriendFeed allow for communication to happen in an inherently social medium. This medium is not limited in its applications. Communication can happen between any user with an account. There are no preferred channels. There are no appointment requirements. A citizen just needs a few short second to type their thoughts and click “Update” to communicate with their representative.

Conceptualizing the real-time citizen

We have long possessed tools that allow for citizens to communicate with representatives, but the real-time web changes the nature of this communication. While it can be argued that the ability to communicate through channels like Twitter merely iterates upon our decades long ability to write letters and emails to representatives this misses the central point about the real-time web: the instantaneous communication that occurs in public-by-default forums.

We finally have a software platform from which we can build a conception of political participation unconstrained by annual or quadrennial elections. This is participation for the real-time citizen.

A political process is an inherently iterative one. Bills are presented, refined, compromised, and eventually voted upon. Traditionally this has happened in the secluded halls of Washington and state capitals. The agents of iteration have been representatives that have been selected by the people but the real-time web provides an opportunity for individual citizens to become engaged in this process. Not only does it allow individuals to be involved in this process but it changes the very nature of participation. Participation becomes open to all and, more importantly, becomes something public to all.

Political participation must no longer confined to election cycles. Yes, election cycles need to play a role in our representative democracy, but we have technology that allows for something more engaging. Leveraging technologies of the real-time web politicians can present ideas to the public and receive immediate feedback. Furthermore, this garnering of feedback would be done with very little overhead. There would be no organizations that would have to mobilize, no buildings to rent, or speaking tours to arrange. The entire process could fit within a representatives current schedule and could take place from wherever a politician was at the moment.

Finally, political debates could use some recent conferences as a model and project a backchannel of discussion during sessions. This could bring a real-time stream of feedback into a legislative discussion. Particularly when combined with a live broadcast of the debate this method would allow for citizens to listen in on and speak up at important legislative events.

All of these potential avenues could be explored to accomplish a singular goal: reframe political participation as something that occurs in small pieces throughout the course of every day for every citizen. The technology has shown that there are millions of people who are willing to produce short pieces of information and convey brief opinions as a part of their everyday life. The only thing left is to incorporate this technology into our ideas of political participation.

  1. For an example of this type of communication see the following exchanges of messages on Twitter between Daniel Bachhuber, a 22 year-old entrepreneur, and Jay Rosen, a journalism professor at New York University. Jay posted a short message with a link to a longer article. Daniel posed a question in response to that post. Jay then proceeded to respond to Daniel’s question in two later posts. The entire conversation took place in less than 30 minutes.
  2. One recent example of this Quora, a real-time question and answer application that uses Tornado as its base.
  3. Jefferson, Thomas. The Life and Selected Writings of Thomas Jefferson. Ed. Adrienne Koch and William Peden. New York: The Modern Library, 2004. 574
  4. Jordan, Grant and William A. Maloney. Democracy and Interest Groups: Enhancing Participation? New York: Palgrave Macmillian, 2007. 1
  5. Agre asks for the copy of this essay that appeared in The Information Society journal to be cited but for reasons of accessibility I have cited the linked essay since it is freely available online.
  6. Shirky, Location 335-343.
  7. Shirky, Location 299-307.
  8. Downey, John. “Participation and/or Deliberation? The Internet as a Tool for Achieving Radical Democratic Aims.” Radical Democracy and the Internet. Ed. Lincoln Dahlberg and Eugenia Siapera. New York, Palgrave Macmillian, 2007. 111.

How we can participate

As technology and the tools for communicating online become more mature and accessible, some general trends emerge. First, with the rise of the real-time web and services like Twitter communication online happens faster. It is also coming from millions of individuals who can publish from nearly everywhere using the mobile capacities of WordPress, Twitter, and similar software. Finally, this increase in speed and quantity of communication fuels the development of sophisticated tools for aggregation and filtration of information flows. All of these tools are usable and deployable by individuals. What is possible with all this technology is a radical shift toward individual control and influence in political participation. Political participation need no longer be something mediated through interest groups and representatives. Instead, technology has allowed for the potential for individuals to play their part in the broader political arena.

We must keep in mind that as revolutionary as all of these technologies can seem they are no guarantee of expanded participation. Technology in and of itself does not determine politics. Application and adoption by citizens determines political impact. Yochai Benkler makes a similar point in the introduction to The Wealth of Networks when he writes that,

Neither deterministic nor wholly malleable, technology sets some parameters of individual and social action. It can make some actions, relationships, organizations, and institutions easier to pursue, and others harder.1

Communication platforms like the real-time web, Twitter, WordPress and consumption mechanisms like Google Reader and Fever drastically altered the parameters of potential individual and social action. Individual citizens have such significant opportunity at their fingertips that the boundaries of political participation have expanded significantly. None of this is assured, rather it is potential that we must put into political practice; however, some recent events can provide optimism.

The parameters that have been expanded through online communication are twofold. First, our ability for individual action and autonomy expands through these tools than previous methods of publication. Second, this individual autonomy allows for more independent group formation that maintains the identity of individual citizens.

Having the ability to publish to a potential global audience was something open to only a select few in a pre-internet age. To globally distribute information was something restricted to mass publishing houses and mainstream media publications. The democratization of publishing has changed this dynamic. If an individual wants a mass audience the potential exists to have one. The physical limitations of printing presses and capital resources to disseminate information have largely dissappeared. As Clay Shirky said at Web 2.0 Expo in 2008,

the internet introduced post-Gutenberg economics. The cost of publishing has fallen through the floor.

The availability of this type of publishing to individuals represents a remarkably political event. Through these tools individuals have tremendous power to publish their viewpoints and through software like Google Reader and Fever they can aggregate information from other individuals to find areas of common interest and shared concern. This ability to publish, aggregate, and organize presents a unique opportunity of group formation and mobilization online that is not the same under a traditional political system. Groups can come together as true collections of individuals who all have access to public-facing communication channels. Symbolic leaders are not needed to relay information and tell members what is important, this process can all be done by individuals.

Ultimately, though, as powerful as this technology is nothing will change by itself. In order for political participation to truly be revolutionized it will rely upon citizens taking advantage of the tools available to them and beginning to publish online and aggregate sources together into a personalized information flow. None of the potential matters if we, as political citizens, obstinately refuse to change our habits. If we continue to give precedence to organizations that do the aggregate, filter, and publish information for us then the potential of all these technologies disappears. However, if we decide to take individual ownership over the publication of our opinions and seek to construct personalized information streams, then the potential of these technologies will become fully realized in revolutionary political change. Through the political application of these technologies we have the ability to gain individual control over our information consumption and publication. We can organize rapidly as individual to undertake collective political action. Ultimately, we can transform political participation from a slow, occasional process that happens at the government’s convenience to something defined by small actions taken as part of a continual process that works toward iterative political change.

  1. Benkler, 17.

Building off of a coral reef – a Whitman blog network

In the most recent episode of Dave Winer and Jay Rosen’s tremendous podcast titled “Rebooting the News” Dave mentions the metaphor of treating journalism like a coral reef. Basically the gist of the metaphor, as I understand it, is this: if you can put in the initial work to start covering many different aspects of a community then pretty soon that community will become involved in building and giving life to that local news site.

Daniel Bachhuber posted his thoughts on this a couple nights ago and expanded upon the concept in terms of data and information. This all got me thinking about how this concept could be applied to a small college paper like the Whitman Pioneer. As part of my work this summer on redesigning The Pioneer’s site I’m working to create a foundation from which the paper can build features in the coming years. Part of this involves creating a framework that can integrate and inspire community engagement. What better method of doing this than creating a “coral reef.”

Here’s what I’m thinking: Right now we have a few various blogs that are written by staff writers. Most of them are not updated all that frequently and those that we have essentially serve as beat blogs and expand the coverage of events and topics that The Pioneer already covers. What we don’t have that I think could prove to be incredibly interesting and engaging is a network of blogs written by other members of the Whitman community.

I know there’s other students, professors, etc. that have blogs but I have no idea what the urls are; consequently I have no easy way to find out what they’re writing about. What I’m envisioning is creating a Pioneer blog network where we do two main things:

  1. Aggregate the content from personal blogs – This could be anything from someone’s Tumblr to a WordPress or Blogger site. They would just go about posting content in the normal way and we would aggregate it on our site and provide links and some information so that The Pioneer’s readers would be able to get a better sense of what “regular” Whitman students are thinking.
  2. Provide a framework for additional blogs – Not all students are tech-savvy enough to have their own personal domain, or motivated enough to start their own blog on or Blogger. What I would propose here is that The Pioneer provides a blog for any student interested. These could be hosted on our domain and run custom installs of WordPress. I would put together a collection of themes (both self-designed and from the WordPress community) and they would be able to install any of those (or anything else they find) and all the plugins the want. Their blog could then be hosted at a sub-domain of The Pioneer’s site (something like

What I see this as accomplishing is twofold.

For one, students (both prospective and enrolled) would have a place to turn to find the recent views of their classmates. We currently feature columns from students studying abroad but there are far more that keep blogs during their time in various countries. How cool would it be to have a place that aggregates all of this information about the Whitman experience?

Second, I see this as providing a forum through which both parents and the community in general can become more involved in the views expressed and discussed. There’s always a lot of talk about how Whitman students go to school in a bubble and I think this could help to combat that perception. It would provide a well-trafficked site through which they could express their opinions on issues spanning sports, politics, community service, crime, etc. It would also give a direct line for community members to interact with students. They would have a place to turn to find what Whitman students are saying and consequently would be able to easily add comments and their own perspectives.

Anyway, those are just my quick and very much rough draft thoughts on the matter. I’m sure the idea will grow the more I think about it in the coming days. I’ll also be working on posting a wireframe for the page in the coming weeks and we’ll see how feasible that is. What I’m wondering is what you think: Sound like a good idea? Think it’s awful and a waste of time? Do you know of any schools or news organizations that are attempting something similar?