Write the Docs: Ashleigh Rentz – The technical challenges of serving docs at Google’s scale

I’m at Write the Docs today in Portland and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Ashleigh started at Google in 2004 as a data center hardware technician. In 2010 she got involved with a team of tech writers working on API documentation. The story she told was of how Google’s CMS came to be.

Google now has so many developer products it fills a periodic table. Literally. They made one.

Scaling problems can show up so gradually, you barely notice them until you’re already in big trouble. This happened for Google with their CMS. What worked in 2005 was horribly broken by 2010.

In 2005 Google had just hired Chris DiBona as the head of Open Source at Google. He started by focusing on getting Google to contribute more to Open Source projects. They created code.google.com as a place for them to share code. When they launched this it was an introductory place to put some code. They started with documentation around their 10 APIs at the time. It’s build using EZT, or EaZy Templating. It’s a simple markup language you can use to define build objects in your documentation.

Google’s code site was optimized for small files, about 256K, and cached things in memory. This grew from Google’s issues scaling the hardware impacts of their consumption at the time. It was a time when a gigabyte of storage was still a lot.

In 2006 Google launched Project Hosting. In the days before Github this mean that they had a place to host and share open code projects.

By 2010 the builds for code.google.com started running in to serious issues. New docs weren’t going live and they were hitting consistent errors. Files were taking almost 45 minutes to build. This meant that a tech writer working on a document had to give themselves a 45 minute lead time. A new project document set to launch at 2pm had to be filed at 1pm. Any typo or issue in the doc submitted meant another 45 minute delay. All of that was compounded by the fact that each build would fail with a typo in any new doc. One doc with an issue caused problems with new docs across all services.

There were other failures, too. Outside of writer mistakes they hit issues with disk I/O. This caused them to push the build cron jobs back to once every 2 hours. The fun part of that was that to pull any technical documentation down from the web also took 2 hours. Picture how awesome that is when you accidentally publish something. This 2 hour turn around time just didn’t work for how Google wanted to publish technical content.

They faced a choice between a band-aid fix and pushing the reset button on their CMS. They decided to develop a CMS that was actually meant for developer documentation. A team of people worked on this new site and the new CMS. The product of this was developers.google.com.

Google’s new developer site as built differently. Gone were the days of having to do everything manually. Since Google now had App Engine they were able to leverage this as the platform from which they could build docs. Using Django nonrel so that they could work with the Django framework with the non-relational database structure of App Engine.

By moving the CMS away from EZT they avoided relying upon a site-wide build. Now they could build only what the writer asks for, when the writer asks for it. Syntax errors now returned in 60 seconds, not 60 minutes. And, your syntax errors don’t affect the system, just you. One downside to no site-wide builds is that when changes (for example, with pricing) happen outside the document tree Google has to manually rebuild the document to reflect the new pricing structure.

In late-2011 they started the process of migrating over to the new site. With 80,000 documents that’s a slow process. The problem is that it split their code documentation across 2 sites. It was a short-term issue that would eventually be fixed. The goal was to complete the move by May 2012 and all went smoothly.

The pace of support

A story

A while back my Google Apps email broke. I could still send and receive email through the web interface but all application-specific passwords were not functioning. I spent most of a Monday trying various steps. Nothing worked.

I remembered that Google offers phone support to its Apps for Business users. I only have one user account so this was $5 a month. Five dollars for dedicated phone and email support? I will take that deal every time.

I called Google and after a relatively painless phone tree I got a wonderfully helpful Googler on the line. He immediately spent 55 minutes on the phone with me. It totally blew me away. Phenomenal experience.

A problem

After 55 minutes, though, the Googler could not fix things. I had the luck of running in to an IMAP authentication error that was not supposed to happen. No worries, I told the Googler, some things take time and I will be patient. We ended the call and that evening he sent me a follow-up email with some steps to try. I gave them a shot (no luck) and replied the next morning. Then I waited. For 3 days.

I followed up via email to see what the status was. Shortly after that message my email was back up and running. Despite fixing my original issue, it left me with a bad experience. This showed me that sometimes the pacing of support is the most important aspect.

A lesson

First impressions matter. While that is no secret, I think last impressions are just as crucial. My first interaction with Google was brilliant. Fifty-five minutes of 1-to-1 help. That is unheard of and I was ecstatic.

Then, I waited. Three days. With no word on what was happening. Did someone forget? Did my email get lost? Do I have to contact them again? Will I have to start at square one with a new support engineer? As a company, those are not questions you want your customers asking. They cause thoughts of doubt. They make it harder to believe you care. When your customers wonder where you went it means you have paced your support poorly.

There are benefits to offering support across multiple channels. Phone support and live chat provide immediate gratification. Email allows for the delay in troubleshooting that can be necessary for more complex issues. The trick is in handling people who start in one medium and end in another. A support team must prepare to handle the inevitable follow-up emails from phone calls and live chats quickly. It is not because the questions are any more important. It is because your first contact with those users set certain expectations. Your job is to see things through.

A phone call sets certain expectations for the pacing of support. That call was my first experience with Google. It was great. The Googler was responsive, caring, and quick to help. All of that was lost after we moved to email. I went from feeling like I had that Googler’s full attention to feeling forgotten.

The medium of support is not important. Consistency and follow through are. The standard you set in phone support should follow every user through every interaction with your support team. First experiences set expectations. Proper pacing ensures they are met. Allowing for each mode of support to move at its own pace irrespective of where the user has been before creates a terribly disjointed and unsatisfying experience. Google may have fixed my issue but they left me far from happy.

Why Google cares if you use your real name

There’s a very simple business reason why Google cares if they have your real name. It means it’s possible to cross-relate your account with your buying behavior with their partners, who might be banks, retailers, supermarkets, hospitals, airlines. To connect with your use of cell phones that might be running their mobile operating system. To provide identity in a commerce-ready way. And to give them information about what you do on the Internet, without obfuscation of pseudonyms.

Simply put, a real name is worth more than a fake one.

Dave Winer – Why Google cares if you use your real name.

Google+ and importing identity

Thanks to an early invite from Raanan I played with Google+ over the weekend. I want to jot down one thought about how Google+ treats identity.

A significant barrier to entry for many social tools is finding the people you already know who are using the service. This is what gets you hooked on a social service.

After logging in and exploring the UI for a bit I went to start creating circles. Here are the options, besides search of course, Google gives for finding people.

Yahoo! and Hotmail are the only two external contact sources I can use. 1 Those of us who have taken the time and carefully created lists of people we follow elsewhere should be able to import those networks directly into Google+. I don’t particularly care about the people Google recommends for me to follow.

With all the data Google has they could be populating lists of people. Instead presenting new users with their email address book, Google recommendations, and Yahoo! and Hotmail imports Google+ could instead be pulling data from Twitter, Facebook, Google Reader, and much more. Break each service into a filtered list and voilà, easy and relevant user discovery for first-time users.

Our identities online are formed by much more than our address books. If Google+ is what comes next it would be nice if they acknowledged and worked with this.

What’s important as a first run experience on Google+ is whether the people I follow elsewhere are already there. The easier it is for people to bring in their existing networks of friends and followers the better Google+ will fare.

Notes:

  1. Seriously?! How miniscule is the Venn diagram of Google+ and Hotmail users?

Leo Laporte – Buzz Kill

Leo Laporte is back to writing on his blog:

I should have been posting it here all along. Had I been doing so I’d have something to show for it. A record of my life for the last few years at the very least. But I ignored my blog and ran off with the sexy, shiny microblogs. Well no more. I’m sorry for having neglected you Leoville.

Via Dave Winer.