Documenting Conferences

Last week I attended Write the Docs. As with pre­vi­ous events I took real-time notes of the talks. Over the last two years that resulted in 32,515 words of notes. It seems fit­ting to spend so much time doc­u­ment­ing a doc­u­men­ta­tion con­fer­ence.

The ephemer­al­ity of con­fer­ences frus­trates me. Events bring a knowl­edge­able group of peo­ple together. Dur­ing the con­fer­ence energy and ideas are pal­pa­ble but when atten­dees go home that all fades in to the ether. The occa­sional slide deck makes it online, but we lose so much. The con­text, ref­er­ences, and ver­biage of a speaker are lost. Scrawled notes on paper are rarely, if ever, shared with those who couldn’t make the conference.

Freely avail­able, web-based, text notes per­sist across time. Those not able to attend can still learn. Those who do attend can reflect and remem­ber. Ulti­mately, when the con­fer­ence ends the learn­ing con­tin­ues. When mul­ti­ple peo­ple pub­lish notes you learn what parts of a talk res­onate with whom. You are able to col­lec­tively doc­u­ment a shared experience.

In the future I would love to see more peo­ple actively doc­u­ment­ing con­fer­ences. Notes help us move our respec­tive com­mu­ni­ties for­ward. With notes speak­ers can build upon the talks of oth­ers rather than rein­vent­ing the wheel with each event. Notes help us spread knowl­edge beyond the few hun­dred peo­ple for­tu­nate enough to attend. With notes we assert con­trol over the ephemer­al­ity of con­fer­ence knowledge.

Write the Docs: Mark Tattersall — Documentation as a Product

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Mark wrapped up the con­fer­ence talk­ing about doc­u­men­ta­tion as a prod­uct. Mark is a prod­uct man­ager at Brain­tree. There the API is, in a sense, their prod­uct. When he started there they had no get­ting started guides, no tuto­ri­als. That was the first thing he worked on and the first things he shipped.

We tend not to talk about doc­u­men­ta­tion as a prod­uct in itself. We over­look it and lump it together with the prod­uct that it’s doc­u­ment­ing and rep­re­sent­ing. It’s fre­quently rel­e­gated to a line item on a project plan; “We need to write the docs so that we can get this shipped.”

Docs think­ing is some­what tac­ti­cal. You work through the tone and voice of your con­tent, who the audi­ence for it is, and the for­mat it will take. Mark wanted to take a step back from the tac­ti­cal, though, and look at doc­u­men­ta­tion more holis­ti­cally. We need to under­stand docs as some­thing inti­mately con­nected and inte­grated to the prod­uct. You think about the prob­lem first before jump­ing in to the ins and outs of cre­at­ing a solution.

Prod­uct think­ing is a lot about burn­ing through a series of ques­tions to get at what applies to your par­tic­u­lar need. Those ques­tions include:

What prob­lems are the docs solving?

The fun­da­men­tal ques­tion for any prod­uct. If you can’t under­stand the spe­cific prob­lem you must ques­tion what you’re doing in the first place. Your doc­u­men­ta­tion has to cater to both exist­ing and poten­tial users.

Doc­u­men­ta­tion is, in many ways, the ana­log to turn­ing over a phys­i­cal soft­ware box and review­ing the details of a prod­uct. Doc­u­men­ta­tion can be the first endur­ing inter­ac­tion that users have with your prod­uct. It acts as a ref­er­ence and an index all in one.

Who will be using your documentation?

You can get at this by cre­at­ing per­sonas of your users. You don’t have to do this but it can be a great exer­cise; it’s some­thing that they do a lot at Brain­tree. To do a good job of this you need to get out there and talk to your users. Talk to other groups within your com­pany, too: sup­port, account­ing, mar­ket­ing. Any­one who talks with your users is worth talk­ing to.

When think­ing about your users keep in mind that you don’t have a lin­ear rela­tion­ship with them. You don’t get to hand-hold them from A to B. They will choose their own path through your docs and dip in and out of the con­tent. Users may even find their way there from any num­ber of sources.

How should your doc­u­men­ta­tion com­mu­ni­cate with your users?

This is not a docs-only deci­sion. You need to put it in the con­text of the com­pany and the prod­uct set. Con­sider how your com­pany talks with cus­tomers and what the prod­uct empha­sizes. Find the voice and per­sona of your own prod­uct and make that shine through in your documentation.

What is the vision for the prod­uct you are documenting?

Where do docs fit in the grand scheme of things? You need to under­stand where the prod­uct and com­pany are going. You want to make sure that your docs can han­dle the changes that are com­ing down the prod­uct pipeline. By estab­lish­ing the core mes­sage of what you’re doc­u­ment­ing you make it eas­ier for users to ori­ent them­selves around that.

What does suc­cess look like?

Are your users achiev­ing their goals when they are using your docs? They need to be get­ting answers to their ques­tions and solu­tions to what they need. If you can, do a sur­vey. Find data and infor­ma­tion about the base­line for cus­tomers inter­ac­tions with your docs. That lets you see what direc­tion your efforts are mov­ing the docs.

You can also look at fun­nels, ana­lyt­ics, calls to action, and engage­ment. Talk to mar­ket­ing and other groups in the com­pany. There are likely shared things you can incor­po­rate in to your docs approach.

What do your com­peti­tors docs look like?

Learn from those around you and from dif­fer­ent fields. Get plugged in to where peo­ple are talk­ing about doc­u­men­ta­tion and learn how to make yours bet­ter. After that research do dif­fer­ent and do bet­ter. Don’t just mimic and mir­ror behav­iors else­where; iter­ate on them and make them your own.


So, with those ques­tions in place you can dive in to build­ing your docs. Before div­ing in con­sider your infor­ma­tion archi­tec­ture. Con­sider your con­tent, con­text, and the users you’re solv­ing for. Let your answers guide your archi­tec­ture; oth­er­wise your con­tent is at an imme­di­ate disadvantage.

Write the Docs: Patrick Arlt — Ditch your CMS with Git and Static Site Generators

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Patrick started the after­noon talks after the amaz­ing snack break. He posted his slides ahead of his talk, smart idea. Patrick works at Esri and pre­vi­ously worked at Geoloqi. There he fell in love with mak­ing prod­ucts for devel­op­ers. When Geoloqi was acquired by Esri he started work­ing on the ArcGIS site for developers.

This site aims to do some­thing great for Esri’s devel­oper com­mu­nity. They try to reach new mar­kets and pro­mote the broader plat­form. The first ver­sion of the site was a sin­gle, sim­ple page. It took a few weeks to get up to 6 or 7 pages. Over time they added more and more infor­ma­tion to help bring peo­ple on board.

Even­tu­ally they real­ized there wasn’t really doc­u­men­ta­tion on the site. It was a lot of exter­nal links and point­ers. What they had was thou­sands of pages of doc for dozens of projects with all of it sit­ting in an old CMS that no one liked. Entire teams had built their own CMS just so they didn’t have to use the painful CMS. This wasn’t what they wanted.

What they wanted was a way to develop doc­u­men­ta­tion like they develop their own prod­uct. They wanted to shift to not just writ­ing doc­u­men­ta­tion but build­ing doc­u­men­ta­tion. That led them to look at their exist­ing toolkit and see what could be applied to documentation.

The first thing they had was a sta­tic site gen­er­a­tor. This let them treat doc­u­ments like source code. There are tools to build large amounts of pages quickly. It’s like a CMS but all the files sit on your local com­puter. Need­less to say there are tons of choices. Sta­tic sites take care of the hard parts of build­ing a web­site so writ­ers can focus on writ­ing content.

Within this site gen­er­a­tor they use Mark­down for con­tent. It’s a very sim­ple lan­guage that, under the hood, gives them a lot of advan­tages for writ­ing documentation.

The last tool they really rely upon is the git / GitHub com­bi­na­tion. They use it to man­age all of their source code so it was a nat­ural exten­sion to use it for doc­u­men­ta­tion con­tent. The GitHub por­tion is used for track­ing issues. They use the issues fea­ture in GitHub to track bugs in doc­u­men­ta­tion just as you would with a code project.

These tools mean that teams have a lot more own­er­ship over their doc­u­men­ta­tion. Any­one who is a part of the orga­ni­za­tion can con­tribute to doc­u­men­ta­tion using the same tools they con­tribute to the prod­uct with.

To those con­cerns about the learn­ing curve with all these Patrick sug­gests plac­ing it in the con­text of bring new peo­ple up to speed with a com­pli­cated CMS sys­tem. In each case you’re need­ing to train peo­ple and main­tain inter­nal doc­u­men­ta­tion on best practices.

But what about all that doc­u­men­ta­tion in their old CMS? Well, they’re still using it. Patrick sees that as an okay com­bi­na­tion. They’re able to serve the sta­tic files along­side the old CMS content.

Esri is next look­ing at 3 main things:

  • More automa­tion. Things like mak­ing pre­view­ing site changes more easily.
  • More cus­tom tools.
  • More train­ing.

Write the Docs: Zach Corleissen — More Than a Reference: Better APIs through Empathy

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Zach is a tech­ni­cal writer at Moz where he focuses on API docs. He kicked things off with some Q&A to help guide where he’d take the talk.

For Zach, empa­thy is the abil­ity to mutu­ally expe­ri­ence the thoughts, emo­tions, and direct expe­ri­ence of oth­ers. It’s not exactly sym­pa­thy, though the words are often used inter­change­ably. It’s pos­si­ble to be sym­pa­thetic with­out being empa­thetic. To write empa­thetic API docs, you have to under­stand and share your users needs. The eas­i­est way to do this is to be an API user your­self. Dog­food your own product.

To gain empa­thy, gain expe­ri­ence. There’s no short­cut around it; you have to use your own API. Using other APIs will help you iden­tify the strengths and weak­nesses of your own API.

To write with empa­thy you first have to know who your read­ers and your API users are. To learn this at Moz they ran a cus­tomer sur­vey; but they ended up with data that wasn’t a large enough sam­ple size to be sta­tis­ti­cally valid. While not sta­tis­ti­cally valid it was use­ful. Once you’ve learned who your read­ers are, write for them. Write to that audi­ence, their back­ground, and their skills. Fre­quently you’re writ­ing for other rea­sons than your read­ers. Dead­lines come in to play. Busi­ness incen­tives force their way in.

They found their read­ers wanted sam­ple code and writ­ten doc­u­men­ta­tion, in that order. Ulti­mately code sam­ples are writ­ten doc­u­men­ta­tion as well.

The best thing you can do in your doc­u­men­ta­tion is to tell true sto­ries. If someone’s in your docs you don’t need to mar­ket to them; just estab­lish trust­wor­thi­ness. To help guide this, ask why does this API mat­ter? True, trust­wor­thy sto­ries help con­vey that information.

Your API docs also have to pro­vide a friendly UI. It needs to be approach­able for your tar­get read­ers. One approach to doing this is to use single-page docs with stronger typog­ra­phy.

You also must lav­ishly doc­u­ment fail­ure states and edge cases. You can tech­ni­cally push this off to your sup­port team but doc­u­ment­ing it will save you so many headaches.

Empa­thetic API docs chal­lenge you to deepen your tech­ni­cal skills and care for the needs of oth­ers. Deeply tech­ni­cal mate­r­ial requires you to be empa­thetic. Empa­thetic API docs delight your read­ers excep­tion­ally. It turns moments they wouldn’t expect in to pleas­ant learn­ing experiences.

Write the Docs: Jared Bhatti — The Getting Stopped Experience: Improving Your Content’s First Impression

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Jared works at Google and attended last year’s con­fer­ence as well. Coin­ci­den­tally he was start­ing a new project at Google work­ing with their cloud prod­ucts. Google’s cloud plat­form encom­passes every­thing from com­put­ing to stor­age to app ser­vices. Each seg­ment has its own get­ting started expe­ri­ence. But each seg­ment can also be com­bined with oth­ers for another, unique get­ting started experience.

He started with a hands-on exer­cise to get every­one think­ing about what chal­lenges they face in get­ting users started with a prod­uct. At Google they found users had a hard time find­ing infor­ma­tion and the results that did come up were com­plex and dense. On top of that there were poor con­nec­tions between the team and the user com­mu­ni­ties. They were expect­ing users to come to them with help­ful information.

They looked at get­ting started as a 3-part jour­ney. Some­one has an idea and then goes through 1) find­ing help­ful con­tent 2) get­ting some­thing work­ing 3) get­ting the envi­sioned project working.

The first step, find­ing help­ful con­tent, is unfor­tu­nately where lots of peo­ple get stuck. With Google Cloud Stor­age users were get­ting irrel­e­vant Google search results, each of which took mul­ti­ple clicks to find the details for. They com­pared this to Ama­zon Web Ser­vice results which were high­light rel­e­vant and gen­er­ally one-click away from the get­ting started expe­ri­ence. To fix this they did a few things. They started run­ning searches on their own prod­ucts. They worked to under­stand how users were actu­ally find­ing their con­tent. And, they looked at what com­peti­tors were doing far bet­ter than they were. With com­peti­tors they looked at large and small; fre­quently it was the small com­peti­tors who were doing really inter­est­ing stuff.

In terms of get­ting some­thing done once you’ve started Jared focused on Google App Engine. The first thing Jared did when join­ing the team was try to spin up an app. 24 steps later he was done. 24. Way too many steps for a first-time user. There were so many dead ends you could run in to before you even started. Luck­ily every­one on the team rec­og­nized this as a large prob­lem. They launched 21 projects around this in just a few months.

As they col­lected data they looked what tools and resources were being most uti­lized and use­ful dur­ing a get­ting started process. They focused on pain points that new devel­op­ers face. Through ques­tion­naires, researcher obser­va­tions, pre­sen­ta­tions, and infor­mal con­ver­sa­tions they gath­ered data. They adopted user-first prin­ci­ples as well:

  1. The user is always in control.
  2. Give the user what they want quickly.
  3. Reduce fric­tion. Everywhere.
  4. Show, don’t tell.

The trick is to give users con­sis­tent value from your appli­ca­tion and prod­uct. They brought the ‘Hello World’ expe­ri­ence down to 4 key steps. It dropped down to just 15 min­utes of work for the inter­ested user. When Jared first went through the steps that same process took 4 hours. They then took that a step fur­ther and brought a wid­get in to the tech­ni­cal doc­u­men­ta­tion that lets users gen­er­ate a ‘Hello World’ wid­get in just 30 seconds.

Lastly, in the third act of get­ting their prod­uct work­ing, users look to launch a com­plex appli­ca­tion. Essen­tially ship­ping what their orig­i­nal idea was. For this they focused on com­mu­nity out­reach and hap­pi­ness sur­veys of cus­tomers. It was about bring­ing exter­nal feed­back in to the prod­uct devel­op­ment process. They also started a GitHub repos­i­tory with fully fleshed out appli­ca­tions that peo­ple can down­load, tear apart, and launch themselves.

Jared left us with a few ques­tions to con­sider for our own products:

  • How are your users get­ting to your docs?
  • How long does it take to get to ‘Hello World’?
  • What met­rics are you collecting?
  • How are you inter­act­ing with your larger community?

Write the Docs: L.S. Cook — Scale for support without losing personality

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Leona kicked talks off after lunch. She views sup­port, doc­u­men­ta­tion, and in-product design as all fun­da­men­tally con­nected. She asked, as writ­ers, how we can not shut out those who learn best through non-written for­mats. Peo­ple who are non-technical by trade are learn­ing soft­ware through the prod­ucts and doc­u­men­ta­tion we create.

One hat for help­ing users is not bet­ter than the other. From engi­neer­ing on up the stack, it’s all about help­ing users on some level. The public-facing con­tent you have, though, is what gives users a per­son­al­ity to equate with your brand.

With in-product design there’s a lot to con­sider. You need to think about what device some­one is using, the visual order of the design, and the over­all flow. For docs par­tic­u­larly, the white­space and visual order can greatly impact how some­one reads and scans the infor­ma­tion you present. Prin­ci­ples of good design can apply to other spheres as well.

Leona argued that sup­port is the only that deserves the title of “ninja.” They’re the strongest link to triage and approach any 1:1 inter­ac­tion with a cus­tomer as an oppor­tu­nity for learning.

Doc­u­men­ta­tion and tuto­ri­als are not syn­onyms. Tuto­ri­als are a unique type of con­tent, and some­thing that’s not used enough in the world of docs, we should change that. These tuto­ri­als are one oppor­tu­nity for com­pa­nies to make money. They can hook your users in to your prod­uct and really illus­trate how the prod­uct will solve their problems.

Write the Docs: Britta Gustafson — Strategies to fight documentation inertia

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Britta wrapped up the talks before lunch by talk­ing about doc­u­men­ta­tion iner­tia. The talk was about what she learned through reviv­ing the com­mu­nity wiki she works on.

For a few years the wiki had an out­dated tagline and a list of ideas on what to add but no men­tion of the exist­ing con­tent. Now, though, the main page has a state­ment of goals, lists of inter­est­ing arti­cles, infor­ma­tion on how to get started, and links to get help with edit­ing. 60 great vol­un­teers helped con­tribute to this renovation.

Britta works for the 3-person com­pany that makes Cydia. The wiki in ques­tion is the doc­u­men­ta­tion for using this jail­bro­ken OS. The wiki con­tent is all done on a volunteer-basis to fur­ther the com­mu­nity. How do you get started writ­ing about these crazy jail­bro­ken cus­tomiza­tions, though?

That’s where doc­u­men­ta­tion comes in. For a long time, though, Cydia had very lit­tle doc­u­men­ta­tion. Peo­ple had to just fig­ure it out as they went along. Britta was con­tribut­ing to Open­Hatch, though. It’s a project for learn­ing how to con­tribute to open source projects. Through this she got expe­ri­ence help­ing devel­op­ers with devel­op­ment prob­lems, even though she her­self is not a developer.

That gave her the con­fi­dence to start work­ing more closely with the devel­op­ers in her own com­mu­nity at Cydia. She threw a few strate­gies out there to help with this.

  • Show up and be some­one who cares. If you hang out where the devel­op­ers are that’s the first step.
  • Talk to new­com­ers and begin­ners. Find where the pain points are and write them down.
  • Ask encour­ag­ing ques­tions to counter anti-documentation con­cerns. Write down advice for those who don’t read the docs.
  • Think of doc­u­men­ta­tion as a mar­ket­ing prob­lem. Make the home­page entic­ing to click and put the best stuff right up front.
  • Make writ­ing more fun. Show that there are peo­ple read­ing it. This can even be imme­di­ately copy-editing new writ­ing as it’s contributed.
  • Make the wiki fun by mak­ing it look active. No one wants to con­tribute to a silent project.
  • Under­stand that the docs sys­tem needs docs. Not every edit­ing tool is under­stand­able to the unfamiliar.
  • Leave some rough edges for oth­ers to smooth out. Easy tasks hook other contributors.

Write the Docs: Scot Marvin — Wabi-Sabi Writing

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Scot cov­ered how we can use wabi-sabi in our writ­ing. He first started read­ing about wabi-sabi when his first child was born. He found the first rule of wabi-sabi seemed to be that you don’t talk about wabi-sabi. It has a respect for syn­tac­tic ambi­gu­ity which goes against much of our West­ern ideals. Essen­tially, though, it’s beauty in imper­ma­nence, imper­fec­tion, and incompleteness.

Our Hel­lenic ideals push us toward per­fec­tion, but ulti­mately we cre­ate very lit­tle of per­fec­tion. Wabi-sabi embraces the rus­tic and the sim­ple. It allows for tran­quil­ity but also asym­me­try. It’s not about being a Lud­dite. It’s also not com­pla­cency. Allow­ing for imper­fec­tion does not mean accept­ing the generic. You strive for excel­lence, not perfection.

Scot works in an agile soft­ware envi­ron­ment. Essen­tially agile means find­ing out where you are, tak­ing small steps toward your goal, and then adjust­ing your under­stand­ing based on what learned. It, like wabi-sabi, is not about cre­at­ing per­fec­tion. It’s a repeat­able process that lets you improve and strive toward excellence.

He tries to abide by four sug­ges­tions in his writing.

  1. Less Faulkner, more Hem­ing­way. They are the two extremes of Amer­ica writ­ing. Write in sim­ple, clear sen­tences. Your doc­u­men­ta­tion should be clear and concise.
  2. Less Coltrane, more Davis. Or, strive for tran­quil writ­ing. Miles Davis was known for his eco­nom­i­cal restraint when it comes to music. Scot strives to put sup­port out of busi­ness with his restrained documentation.
  3. Less Ver­sailles, more Ryōan-ji. Work toward sim­ple writ­ing. Your docs shouldn’t be the equiv­a­lent of the gar­dens at Ver­sailles. This can also be thought of as the, “Don’t get too cute” principle.
  4. Done is beau­ti­ful. We can’t truly com­plete any­thing, but we can get it done. Incom­plete, imper­fect writ­ing is okay. Incre­men­tally make it bet­ter; don’t just wait for per­fect completion.

Write the Docs: Lauren Rother — We Strongly Recommend You Write Best Practices

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Lau­ren is a tech writer at Pup­pet Labs. She focuses on the Forge, a repos­i­tory of com­mu­nity mod­ules for Pup­pet. At Pup­pet the docs team falls under engi­neer­ing, and the writ­ers are embed­ded within the var­i­ous devel­op­ment teams.

The chal­lenge they face is that users have dif­fer­ent needs, envi­ron­ments, work­flows, and most impor­tantly, lev­els of expe­ri­ence. The exper­i­ment Pup­pet tried was the beginner’s guide to mod­ules. They’re in the process of writ­ing an advanced guide, which would incor­po­rate what they learned from the beginner’s guide.

In the first draft they took inspi­ra­tion from text­books. They brain-dumped every­thing in to a Google Doc which they then sorted in to 4 sec­tions. They wanted to present heaps of infor­ma­tion so that every­one who needs a foun­da­tional knowl­edge of mod­ules would be sat­is­fied. It was imme­di­ately appar­ent that the doc­u­ment was over­whelm­ing. It was hard to approach and the first sec­tion in the doc­u­ment was dis­pro­por­tion­ately huge.

What they real­ized was that the first thing they needed to do was cre­ate sign posts within each sec­tion along with sub-sections. The took a more orientation-based approach to writ­ing the doc­u­ment. This sec­ond take at the doc­u­ment pro­duced some­thing much more read­able and com­pre­hen­si­ble. Large para­graphs were bro­ken up and edits were geared toward ori­ent­ing the user. It was still really dif­fi­cult to nav­i­gate and locate dis­tinct items, though. You con­stantly were going back and search­ing for where you left off; it wasn’t easy. In short, the doc at this point was really read­able but not usable.

They real­ized they’d been work­ing on a trea­tise, not a guide­book. They dropped the aca­d­e­mic stand­point and text­book approach. What they really wanted was a doc that peo­ple would want to look at more than once, not some­thing they’d only read because of its chal­leng­ing nature.

In hind­sight it seems really obvi­ous that a step-by-step approach was nec­es­sary. It didn’t occur to them to take this approach, though, until they were 2 drafts in. The final ver­sion has a table of con­tents, head­ings, steps, intro­duc­tory sen­tences and exam­ples. All around it was a huge improvement.

Lau­ren couldn’t help from think­ing about future improve­ments, though. There wasn’t enough get-your-hands-dirty details in the beginner’s guide. It was too high-level at times. She’s tak­ing some of these lessons and apply­ing them to the advanced guide to modules.

Write the Docs: Brian L. Troutwine — Instrumentation as Living Documentation: Teaching Humans About Complex Systems

I’m at Write the Docs today in Port­land and will be post­ing notes from ses­sions through­out the day. These are all posted right after a talk fin­ishes so they’re rough around the edges.

Brian kicked off the slate of talks after the morn­ing break. He’s a soft­ware engi­neer who focuses on computer-to-computer pro­gram­ming. It’s not as much end-user soft­ware. He pri­mar­ily works on real-time and dis­trib­uted sys­tems. A lot of this is for adver­tis­ing com­pa­nies and takes the form of real-time bid­ding on ads.

Adver­tis­ing has become an engineering-driven indus­try. It’s low­ered the cost of adver­tis­ing and allowed peo­ple to spend as lit­tle as $10 a month for their busi­ness adver­tis­ing. The nature of the prob­lem in this world is it super low latency (less than 100ms per trans­ac­tion) and global, highly con­cur­rent nature. The com­plex sys­tems to exe­cute this have com­plex orga­ni­za­tions around them. They’re tightly cou­pled to exter­nal sys­tems and have non-linear feed­back built-in to them.

Bad things hap­pen when these com­plex sys­tems fail. You’ve built them to solve wicked prob­lems and when they fail they some­times cre­ate even worse prob­lems then they orig­i­nally solved. Ulti­mately, “our abil­ity to cre­ate large and com­plex sys­tems fools us into believ­ing that we’re also enti­tled to under­stand them.” To work toward that under­stand­ing we write documentation.

The issue is that com­plex sys­tems are fiendishly dif­fi­cult to com­mu­ni­cate about. The gap of under­stand­ing is dif­fi­cult to bridge in doc­u­men­ta­tion. Any mis­com­mu­ni­ca­tions around com­plex sys­tems are acci­dents in the mak­ing. Doc­u­men­ta­tion can help to reduce these acci­dents. It can help give every­one a greater shared under­stand­ing of the sys­tem. With­out know­ing how a sys­tem should behave you can­not really under­stand how it shouldn’t behave.

An issue with under­stand­ing through doc­u­men­ta­tion is that the docs get out of date. Com­plex sys­tems evolve and writ­ten words “rot” as the sys­tem moves on. Some­times this falls on engi­neers for fail­ing to update what they change. Or, engi­neers can be unaware of the sys­tem as it is actu­ally used. This con­tributes to mis­un­der­stand­ing through docs.

Nor­mal acci­dents hap­pen as well. Every sys­tem has, intrin­sic to itself, some fail­ure. No mat­ter how much we try we can never get rid of them, we can only choose how to react in the pres­ence of such an accident.

How can we build bet­ter sys­tems that dis­trib­ute knowl­edge and bet­ter avoid acci­dents in real-time? Brian’s argu­ment is for instru­men­ta­tion. This reflects the real­ity of the sys­tem as it exists. Instru­men­ta­tion allows the users and the engi­neers to explore the sys­tem as it actu­ally exists. This explo­ration, done hon­estly, guides us to a new, bet­ter under­stand­ing of the sys­tem. Instru­men­ta­tion also democ­ra­tizes the orga­ni­za­tion around a com­plex system.

Trou­bleshoot­ing com­plex sys­tems can some­times be lim­ited by not hav­ing enough infor­ma­tion. If you’re rely­ing upon exter­nal ser­vices there’s a level of opaque­ness that you can­not see through. But, instru­men­ta­tion is not a panacea. It can also cre­ate an over­load of infor­ma­tion that floods your abil­ity to make sense of the sys­tem. Too much infor­ma­tion ham­pers interpretation.

On a darker note instru­men­ta­tion can be used for unde­sir­able pur­poses. Sur­veil­lance is a form of this. Com­plex orga­ni­za­tions with exces­sive instru­men­ta­tion can do evil things.

What we should do is write doc­u­men­ta­tion with the instru­men­ta­tion in mind. Pro­ce­dure man­u­als and visu­al­iza­tions, for exam­ple, reduce the amount of com­plex knowl­edge required. The more con­tex­tual lay­ers you add, the more you reduce the “big boards with blinky lights” issue. Instru­men­ta­tion is like a suit. It needs to fit your mind. Cross-checks and doc­u­mented error mar­gins mit­i­gate instru­ment defi­ciency. Check­lists with ref­er­ences to instru­men­ta­tion at deci­sion points is key. It pin­points where atten­tion should flow and how to process it toward informed decisions.

So, instru­men­ta­tion addresses the prob­lems of doc­u­men­ta­tion and doc­u­men­ta­tion addresses the prob­lems of instru­men­ta­tion. Together they help guide and ensure the qual­ity and under­stand­ing of com­plex systems.