Meetings: Where Work Goes to Die

February 14, 2012

How many meetings did you have today? This week? This month?

Now ask yourself how many of those meetings were worthwhile, versus the work that you could have accomplished in that same time.

Meetings, the practical alternative to work

This might lead one to wonder why we even have meetings at all.

At GitHub we don't have meetings. We don't have set work hours or even work days. We don't keep track of vacation or sick days. We don't have managers or an org chart. We don't have a dress code. We don't have expense account audits or an HR department.

Now, I'm sure Tom was being facetious when he said that GitHub doesn't have meetings, because I sure as heck saw meeting rooms when I recently visited their offices to give a talk. Who knows, maybe they use them to store all the extra forks.

Although some meetings are inevitable, even necessary, the principle he's advocating here is an important one. Meetings should be viewed skeptically from the outset, as risks to productivity. We have meetings because we think we need them, but all too often, meetings are where work ends up going to die. I have a handful of principles that I employ to keep my meetings useful:

  1. No meeting should ever be more than an hour, under penalty of death.

    The first and most important constraint on any meeting is the most precious imaginable resource at any company: time. If you can't fit your meeting in about an hour, there is something deeply wrong with it, and you should fix that first. Either it involves too many people, the scope of the meeting is too broad, or there's a general lack of focus necessary to keep the meeting on track. I challenge anyone to remember anything that happens in a multi-hour meeting. When all else fails, please keep it short!

  2. Every meeting should have a clearly defined mission statement.

    What's the mission statement of your meeting? Can you define the purpose of your meeting in a single succinct sentence? I hesitate to recommend having an "agenda" and "agenda items" because the word agenda implies a giant, tedious bulleted list of things to cover. Just make sure the purpose of the meeting is clear to everyone; the rest will take care of itself.

  3. Do your homework before the meeting.

    Since your meeting has a clearly defined mission statement, everyone attending the meeting knows in advance what they need to talk about and share, and has it ready to go before they walk into the room. Right? That's how we can keep the meeting down to an hour. If you haven't done your homework, you shouldn't be in the meeting. If nobody has done their homework, the meeting should be cancelled.

  4. Make it optional.

    "Mandatory" meetings are a cop-out. Everyone in the meeting should be there because they want to be there, or they need to be there. One sure way to keep yourself accountable for a meeting is to make everyone optional. Imagine holding a meeting that people actually wanted to attend, because it was … useful. Or interesting. Or entertaining. Now make it happen!

  5. Summarize to-dos at the end of the meeting.

    If your meeting never happened, what would the consequences be? If the honest answer to that is almost nothing, then perhaps your meeting has no reason to exist. Any truly productive meeting causes stuff to happen as a direct result of the decisions made in that meeting. You, as a responsible meeting participant, are responsible for keeping track of what you need to do – and everyone in the room can prove it by summarizing their to-do list for everyone's benefit before they leave the meeting.

It's not that we shouldn't have meetings, but rather, we need to recognize the inherent risks of meetings and strive to make the (hopefully) few meetings we do have productive ones. Let's work fast, minimize BS, and get to the point.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Posted by Jeff Atwood    54 Comments

Farewell Stack Exchange

February 6, 2012

I am no longer a part of Stack Exchange.

I still have much literal and figurative stock in the success of Stack Exchange, of course, but as of March 1st I will no longer be part of the day to day operations of the company, or the Stack Exchange sites, in any way.

It's been almost exactly 4 years since I chose my own adventure. In those four years, we accomplished incredible things together. Stack Overflow is now an enormous bustling city, a hugely positive influence on the daily lives of programmers around the world, a place to learn from and teach your peers. And the entire Stack Exchange network, born out of the seed of Stack Overflow, is a reference model of high signal, low noise, no-nonsense Q&A that makes the internet better for all of us. I could quote traffic figures, but to me that's not what it's all about. I prefer to think of it building something awesome, because I know that if you build it, they will come.

Stackoverflow-stackexchange-logos

And they did. I'll be damned if we didn't change our little corner of the Internet for the better. Possibly permanently. This is more than I could have ever hoped for, and I am honored to have been a founding and guiding part of it for the last four years. But I don't need to be a part of it forever – nor should I be, if I've been doing my job correctly. Stack Exchange was always about designing software and creating recipes for self-governing communities who love a particular topic. It is an honor to be a "just" a citizen of this community again, because as a citizen, I too have the power to shape its future. Just like you do.

Startup life is hard on families. We just welcomed two new members into our family, and running as fast as you can isn't sustainable for parents of multiple small children. The death of Steve Jobs, and his subsequent posthumous biography, highlighted the risks for a lot of folks:

For a long time, work was my only thing. I worked evenings, weekends, and Christmas. At those rare times when I wasn’t at work in body, I was there in spirit, unable to speak or think of much else. I wanted so badly to climb the mountain that I stopped asking why I was doing it.

I admire Steve for the mountains he climbed. At the same time, I wonder if he missed the whole point, becoming the John Henry of our time. He won the race, but at what cost?

Me? I may turn out to be a failure in business, but I refuse to fail my kids.

I've followed Brad Wardell's success for a long time, and he had a very similar reaction to Jobs' death.

In the last several years, the company has been successful enough to generate a substantial amount of capital. And with it, I have been fortunate to bring in people with great talent. And so I started thinking of all the amazing things we would do. I would put in crazy hours to do it, of course, but we would go and do amazing things.

Then Steve Jobs died.

And suddenly I realized something. What is the objective here? My oldest child just turned 15. My other two are no longer little either. And I have been missing out on them. And my wife.

For all the success and amazing accomplishments of Steve Jobs, in the end, nothing could save him. Death can come at any time. And I realized that if I found myself on death’s door, I would regret deeply not having spent more time with my kids when they were…well, kids.

You may have more discipline than I do. But for me, the mission is everything; I'm downright religious about it. Stack Overflow and Stack Exchange have been wildly successful, but I finally realized that success at the cost of my children is not success. It is failure.

I've met so many amazing people through Stack Exchange. First, the incredibly talented team of people who work for the company, many of whom I personally recruited. As far as I'm concerned, you are among the best in the world at what you do. That's why we hired you, and it has been an honor to serve with you. But more than that, the broader community that formed around a shared vision of making the Internet better through these beautiful public parks of curated, creative commons Q&A. I have continually been humbled by the brilliant minds that saw fit to work alongside us towards this goal, who selflessly contributed their own time and effort because they just plain loved this stuff as much as we do.

I will miss you all terribly.

What's next for me? I honestly don't know. I do know that I love the Internet, and I remain passionate as ever about making the Internet better – but right now I need to be with my family. In six months, perhaps I'll be ready to choose another adventure. I have total confidence that the team at Stack Exchange, and the thriving community that makes it so great, will carry Stack Exchange onward. After all, our shared voyage never ends, it just takes different forms.

Come, my friends.
'Tis not too late to seek a newer world.
Push off, and sitting well in order smite
the sounding furrows; for my purpose holds
To sail beyond the sunset, and the baths
Of all the western stars, until I die.
It may be that the gulfs will wash us down;
It may be that we shall touch the Happy Isles,
And see the great Achilles, whom we knew.
Though much is taken, much abides; and though
We are not now that strength which in old days
Moved earth and heaven, that which we are, we are —
One equal temper of heroic hearts,
Made weak by time and fate, but strong in will
To strive, to seek, to find, and not to yield.

Farewell, Stack Exchange. I hope you can understand that if I was hard on you at times, it was because I wanted you to be the best you could possibly be.

It was because I loved you.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.
Posted by Jeff Atwood    239 Comments

Listen to Your Community, But Don't Let Them Tell You What to Do

February 3, 2012

You know how interviewers love asking about your greatest weakness, or the biggest mistake you've ever made? These questions may sound formulaic, maybe even borderline cliche, but be careful when you answer: they are more important than they seem.

So when people ask me what our biggest mistake was in building Stack Overflow I'm glad I don't have to fudge around with platitudes. I can honestly and openly point to a huge, honking, ridiculously dumb mistake I made from the very first day of development on Stack Overflow – and, worse, a mistake I stubbornly clung to for a solid nine month period after that over the continued protestations of the community. I even went so far as to write a whole blog post decrying its very existence.

For the longest time, I had an awfully Fight Club-esque way of looking at this: the first rule of Stack Overflow was that you didn't discuss Stack Overflow! After all, we were there to learn about programming with our peers, not learn about a stupid website. Right?

Fight-club-soap

I didn't see the need for a meta.

Meta is, of course, the place where you go to discuss the place. Take a moment and think about what that means. Meta is for people who care so deeply about their community that they're willing to go one step further, to come together and spend even more of their time deciding how to maintain and govern it. So, in a nutshell, I was telling the people who loved Stack Overflow the most of all to basically … f**k off and go away.

As I said, not my finest hour.

In my defense, I did eventually figure this out, thanks to the continued prodding of the community. Although we'd used an external meta site since beta, we eventually launched our very own meta.stackoverflow in June 2009, ten months after public beta. And we fixed this very definitively with Stack Exchange. Every Stack Exchange site we launch has a meta from day one. We now know that meta participation is the source of all meaningful leadership and governance in a community, so it is cultivated and monitored closely.

I also paid penance for my sins by becoming the top user of our own meta. I've spent the last 2 years and 7 months totally immersed in the morass of bugs, feature requests, discussions, and support that is our meta. As you can see in my profile, I've visited meta 901 unique days in that time frame, which is disturbingly close to every day. I consider my meta participation stats a badge of honor, but more than that, it's my job to help build this thing alongside you. We explicitly do everything in public on Stack Exchange – it's very intentionally the opposite of Ivory Tower Development.

Along the way I've learned a few lessons about building software with your community, and handling community feedback.

1. 90% of all community feedback is crap.

Let's get this out of the way immediately. Sturgeon's Law can't be denied by any man, woman, child … or community, for that matter. Meta community, I love you to death, so let's be honest with each other: most of the feedback and feature requests you give us are just not, uh, er … actionable, for a zillion different reasons.

But take heart: this means 10% of the community feedback you'll get is awesome! I guarantee you'll find ten posts that are pure gold, that have the potential to make the site clearly better for everyone … provided you have the intestinal fortitude to look at a hundred posts to get there. Be prepared to spend a lot of time, and I mean a whole freaking lot of time, mining through community feedback to extract those rare gems. I believe every community has users savvy enough to produce them in some quantity, and they're often startlingly wonderful.

2. Don't get sweet talked into building a truck.

You should immediately triage the feedback and feature requests you get into two broad buckets:

We need power windows in this car!

or

We need a truck bed in this car!

The former is, of course, a reasonable thing to request adding to a car, while the latter is a request to change the fundamental nature of the vehicle. The malleable form of software makes it all too tempting to bolt that truck bed on to our car. Why not? Users keep asking for it, and trucks sure are convenient, right?

Don't fall into this trap. Stay on mission. That car-truck hybrid is awfully tempting to a lot of folks, but then you end up with a Subaru Brat. Unless you really want to build a truck after all, the users asking for truck features need to be gently directed to their nearest truck dealership, because they're in the wrong place.

3. Be honest about what you won't do.

It always depressed me to see bug trackers and feedback forums with thousands of items languishing there in no man's land with no status at all. That's a sign of a neglected community, and worse, a dishonest relationship with the community. It is sadly all too typical. Don't do this!

I'm not saying you should tell your community that their feedback sucks, even when it frequently does. That'd be mean. But don't be shy about politely declining requests when you feel they don't make sense, or if you can't see any way they could be reasonably implemented. (You should always reserve the right to change your mind in the future, of course.) Sure, it hurts to be rejected – but it hurts far more to be ignored. I believe very, very strongly that if you're honest with your community, they will ultimately respect you more for that.

All relationships are predicated on honesty. If you're not willing to be honest with your community, how can you possibly expect them to respect you … or continue the relationship?

4. Listen to your community, but don't let them tell you what to do.

It's tempting to take meta community requests as a wholesale template for development of your software or website. The point of a meta is to listen to your community, and act on that feedback, right? On the contrary, acting too directly on community feedback is incredibly dangerous, and the reason many of these community initiatives fail when taken too literally. I'll let Tom Preston-Werner, the co-founder of GitHub, explain:

Consider a feature request such as “GitHub should let me FTP up a documentation site for my project.” What this customer is really trying to say is “I want a simple way to publish content related to my project,” but they’re used to what’s already out there, and so they pose the request in terms that are familiar to them. We could have implemented some horrible FTP based solution as requested, but we looked deeper into the underlying question and now we allow you to publish content by simply pushing a Git repository to your account. This meets requirements of both functionality and elegance.

Community feedback is great, but it should never be used as a crutch, a substitute for thinking deeply about what you're building and why. Always try to identify what the underlying needs are, and come up with a sensible roadmap.

5. Be there for your community.

Half of community relationships isn't doing what the community thinks they want at any given time, but simply being there to listen and respond to the community. When the co-founder of Stack Exchange responds to your meta post – even if it wasn't exactly what you may have wanted to hear – I hope it speaks volumes about how committed we are to really, truly building this thing alongside our community.

Regardless of whether money is changing hands or not, you should love discovering some small gem of a community request or bugfix on meta that makes your site or product better, and swooping in to make it so. That's a virtuous public feedback loop: it says you matter and we care and everything just keeps on getting better all in one delightful gesture.

And isn't that what it's all about?

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.
Posted by Jeff Atwood    24 Comments

The One Button Mystique

February 1, 2012

I enjoy my iPhone, but I can't quite come to terms with one aspect of its design: Apple's insistence that there can be only ever be one, and only one, button on the front of the device.

Iphone-4s-button

I also own a completely buttonless Kindle Fire, and you'll get no argument from me that there should be at least one obvious "Jesus Handle" button on the front of any gadget. I do wonder why Amazon decided to make the Fire buttonless, when every other Kindle they ship has a home button. Amazon has a track record of making some awfully rough version 1.0 devices; I'm sure they'll add a home button in a version or two. And, hey, at only $199 I'm willing to cut them a little slack. For now.

Even Apple is no stranger to buttonless devices. Consider the oddly buttonless third generation iPod Shuffle, where you had to double and even triple click the controls on the headphones to do basic things like advance tracks. Oh, and by the way, this also made every set of headphones you own obsolete, at least for use with this model. The fourth gen shuffle rapidly switched back to physical controls on the device, and the fifth gen went to touch controls on the device, as expected.

Ipod-shuffle-3g-vs-4g

Microsoft is just as guilty. I sometimes struggle with the otherwise awesome Xbox 360 Wireless Microphone. It has only a power button and some lights.

Xbox-360-wireless-microphone

In its defense, for the most part it does just work when you pick it up and start singing (badly, in my case), but I admit to being slightly perplexed every time I have to sync it with an Xbox, or figure out what's going on with it. Can you blame me?

When you turn on the microphone, the built-in lights shine to display the microphone status as follows:

  • Power on: lights flash green one time every second
  • Connecting: lights flash green four times every second
  • Connection complete: lights flash blue, and then stops

When your battery power is low, the built-in lights shine to display the battery charge status as follows:

  • Low: Lights flash amber one time every three seconds
  • Critical: Lights flash amber one time every second

When your microphone moves out of the wireless range of your console, the lights flash green one time every second. The lights can also change color together with supported game titles.

If we can agree that no buttons is clearly a bad idea, I think it follows that one button is problematic in its own way. I have the same issue with the single button on the iPhone that I do with the single button mouse – it may be OK-ish at the very beginning, but over time it leads to absurd, almost comical overloading of functionality. Consider how many different things the single button on the face of an iPhone now controls:

How-to-use-the-home-button-excerpt
(diagram courtesy Andrew Durdin, source)

The iPhone home button? Why, it's easy! You have your choice of…

  • single-click
  • double-click
  • triple-click
  • click and hold
  • click and pause and click again

All of which have different meanings at different times, of course. In particular I spend a lot of time double-clicking to get to the active apps list, and I often mis-tap which kicks me over to the home screen. I have so many apps installed on my iPhone that search is the only rational way to navigate. This means I search a lot, which requires clicking once to get to the default home page, pausing, then clicking again. Sometimes I click too long, which is then detected as click-and-hold, and I get the voice search app which I am … er, not a fan of, to put it mildly.

I've gotten to the point where I dread using the home button on my iPhone because it Makes Me Think. And I get it wrong a significant percentage of the time. This isn't the way it's supposed to be.

You might be expecting me to turn into a rabid Windows Phone or Android fanboy about now and snarkily note how they get it right. I'm not sure they do. Either of them. They all manage to suck in their own special way.

Phone-windows-and-android-buttons

When there's one button on the device, at least it's clear what that button is supposed to do, right? Well, sometimes.

There is one theme I agree with here – the clearly marked back button on both Android and Windows phones, just like a web browser. I mostly use my iPhone as a platform for the Internet, and the simplicity of the Internet is its primary strength: a collection of obvious things to click, and an easy, giant honking back button so you never get lost in its maze of twisty passages, all alike. It is true that browsers have a home button, but the latest versions of Chrome, Firefox, and Internet Explorer have all but pushed that home button off the UI in favor of the ginormous back button. While I'll tentatively agree that not all phone apps have to behave like the Internet, the Internet is becoming more and more of a platform for bona fide applications every day. The back button is a UI paradigm that works like gangbusters for webapps, and I'd argue strongly in favor of that being a hard button on a device.

But once you add three buttons, thinking starts to creep in again. Am I pressing the correct button? That's never good. And I don't even know what that third button is supposed to be on the Android phone! I could possibly be in favor of the hard search button on the Windows phone, I suppose, but I'd rather see good, consistent use of two buttons on the face of a device before willy-nilly adding Yet Another Button. I think there's a reason the industry has more or less standardized on a two-button mouse, for example. (Yes, there is that pesky middle button, but it's a nice to have, not an essential part of the experience.)

What about the one finger solution? Even with touch devices, one finger does not seem to be enough; there's a curious overloading of the experience over time.

On the iPad, there are a number of system-wide gestures, such as swiping left or right with four fingers to switch between apps. Four-finger swipes? That's convoluted, but imagine a virtual mixing console with horizontal sliders. Quickly move four of them at once...and you switch apps. Application designers have to work around these, making sure that legitimate input methods don't mimic the system-level gestures.

The worst offender is this: swipe down from the top of the screen to reveal the Notification Center (a window containing calendar appointments, the weather, etc.). A single-finger vertical motion is hardly unusual, and many apps expect such input. The games Flight Control and Fruit Ninja are two prime examples. Unintentionally pulling down the Notification Center during normal gameplay is common. A centered vertical swipe is natural in any paint program, too. Do app designers need build around allowing such controls? Apparently, yes.

Yes, our old friend overloading is now on the touch scene in spades: for all but the simplest use of a tablet, you will inevitably find yourself double-tapping, tapping and holding, swiping with two fingers, and so on.

Apple's done a great job of embodying simplicity and clean design, but I often think they go too far, particularly at the beginning. For example, the first Mac didn't even have cursor keys. Everything's a design call, and somewhat subjective, but like Goldilocks, I'm going to maintain that the secret sauce here is not to get the porridge too cold (no buttons) or too hot (3 or more buttons), but just right. I'd certainly be a happier iPhone user if I didn't have to think so much about what's going to happen when I press my home button for the hundredth time in a day.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.
Posted by Jeff Atwood    45 Comments

Defeating SOPA and PIPA Isn't Enough

January 18, 2012

SOPA and PIPA are two pieces of proposed legislation designed to "stop" Internet piracy… in the most hamfisted way imaginable. As Mitchell Baker explains:

Assume there's a corner store in your neighborhood that rents movies. But the movie industry believes that some or even all of the videos in that store are unauthorized copies, so that they're not being paid when people watch their movies. What should be done?

SOPA/PIPA do not aim at the people trying to get to the store, or even the store itself. The solution under the proposed bills is to make it as difficult as possible to find or interact with the store:

  1. Maps showing the location of the store must be changed to hide it.
  2. The road to the store must be blocked off so that it is difficult to physically get to there.
  3. Directory services must delist the store’s phone number and address.
  4. Credit card companies would have to cease providing services to the store.
  5. Local newspapers would no longer be allowed to place ads for the video store.
  6. To make sure it all happens, any person or organization who doesn’t do this is subject to penalties. Even publishing a newsletter that tells people where the store is would be prohibited by this legislation.

Just substitute "corner store" with "website" and I think you can see where this is going. These bills are so rife with potential for abuse and misuse, so clearly dangerous to the very fabric of the Internet, that frankly I have a hard time getting worked up about them. The Internet is under constant siege by large companies, and will be for the forseeable future. This is nothing new. These bills will be defeated, because they must be.

Instead, I'm scratching my head and wondering how such boneheaded bills made it this far in Congress. I can think of a few reasons:

  • Average people don't understand how the Internet works and thus can't comprehend the danger.
  • Nobody pays attention to what our government does until it hits them in the pocketbook (or below the belt).
  • These bills were pushed through by highly paid lobbyists for the entertainment industry.

I often bemoan the state of Slacktivism on the internet, where changing your Facebook or Twitter picture is considered a valid and effective form of protest. But this time, I am happy to say, was indeed different.

Perhaps because of the obvious danger of these bills, geek websites and communities banded together weeks ago to protect themselves and the greater Internet. Like many other technical communities, we wrote about it on our blog, talked about it on our podcast, and even put up a little banner on Stack Overflow for a day. Users were encouraged to call, fax, and write their representatives in Congress and express their concerns. And they did, in droves! But outside of our technical geek ghettos, there was precious little mainstream coverage of this dangerous legislation.

That is, until major sites like Wikipedia, Google, and Craigslist joined the bandwagon today. Most notably, Wikipedia actually went dark for all of today, January 18th, rendering all of English language Wikipedia inaccessible. That turned the tide, and transformed SOPA/PIPA into something that average people would actually talk about and care about. There's no better way to raise awareness of the danger these bills pose than blacklisting one of the greatest resources the Internet has ever produced.

Wikipedia

While SOPA/PIPA are still alive -- barely -- for now, I think it's safe to say that they are well on their way to defeat. I'm glad to be a part of that, however tiny, but I cannot in good conscience celebrate.

Yes, we likely succeeded in defeating these two specific bills and galvanizing the political will of major Internet communities, including our very own Stack Exchange. These are good and noble and just and necessary things. They are things to be proud of. But instead of celebrating, let's take this time to reflect and ask ourselves a deeper question: how is it that these dangerous bills came to exist in the first place?

First, and again, this is a critical battle to wage and win. SOPA is just the latest, but in many ways, the most absurd campaign in the endless saga of America’s copyright wars. It will be yet another failed attempt in a failed war, and I obviously believe it should be opposed.

But second, and as you describe, this isn’t my war anymore. Not because my heart isn’t in it, but because I don’t believe we will win that war (or better, win the peace and move on) — even if we can win battles like this one — until the more basic corruption that is our government gets addressed. That’s the fight I have spent the last 4 years working on. That’s where I’ll be for at least the next 6.

For this is what I know: We will never (as in not ever) win the war you care about until we win the war against this corruption of our Republic.

Republic-lost-cover

Of course, as my book, Republic, Lost: How Money Corrupts Congress -- and a Plan to Stop It, describes, this is an insanely difficult, possibly impossible, fight. But whether difficult or not, it is the fight that must be waged.

We have done much. But in our celebratory enthusiasm, please take a moment to hear out Mr. Lessig, and appreciate just how far we have yet to go.

So yes, join us in fighting the obvious insanity of legislation like SOPA and PIPA that threaten the open, unfettered Internet. But please, please also join us in attacking the far more pernicious problem of lobbyist money subtly corrupting our government. If we don't deal with that, we will never stop fighting bills like SOPA and PIPA.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.
Posted by Jeff Atwood    43 Comments

Building Social Software for the Anti-Social

December 19, 2011

In November, I delivered the keynote presentation at Øredev 2011. It was the second and probably final presentation in the series I call Building Social Software for the Anti-Social.

I've spent almost four years thinking about the Q&A format, and these two presentations are the culmination of that line of thought. In them I present ten "scary ideas", ideas which are counterintuitive for most folks. These are the building blocks we used to construct Stack Overflow, and by extension, Server Fault, Super User, and the rest of the Stack Exchange network.

  1. Radically lower the bar for participation.
  2. Trusting (some of) your users.
  3. Life is the world’s biggest MMORPG.
  4. Bad stuff happens.
  5. Love trumps money.
  6. Rules can be fun and social.
  7. All modern website design is game design.
  8. Thoughtful game design creates sustainable communities.
  9. The community isn’t always right.
  10. Some moderation required.

It's not the same experience as attending the actual live presentation, of course, but you can certainly get the gist of it by viewing the slides for these two presentations online:

Building Social Software for the Anti-Social

Building Social Software for the Anti-Social: Part II

The Øredev organizers hired ImageThink to draw each presentation on a whiteboard live on stage as it was presented. I was skeptical that this would work, but the whiteboard visualizations came out great for all the presentations. Here's the two whiteboard drawings ImageThink created during my presentation. (Yes, they had two artists on stage "live whiteboarding", one on the left side, and one on the right side.)

Imagethink-oredev-atwood-1-small

Imagethink-oredev-atwood-2-small

It's not a bad approximation of what was covered. If you're curious about live whiteboard visualizations, ImageThink posted a great set of links on their blog that I highly recommend.

After four years, we've mostly figured out what works and what doesn't work for our particular brand of low noise, high signal Q&A at Stack Exchange. But the title Social Software for the Anti-Social is only partially tongue in cheek. If you want to learn anything online, you have to design your software to refocus and redirect people's natural social group impulses, and that's what these presentations attempt to explain. I hope you enjoy them!

Update: Part II is now available as a full talk, with audio and video courtesy of Oredev. Watch it now!

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Posted by Jeff Atwood    22 Comments

Gifts for Geeks, 2011 Edition

December 13, 2011

Between founding Stack Overflow (and later, running Stack Exchange) and having a child, I haven't had much time to blog about the holidays for a few years now. The last Gifts for Geeks I did was in 2008. Those recommendations are still as valid as ever, but I just couldn't muster the enthusiasm to do it every year.

I've also come to realize, especially after having a child, that the goal in life is not to own a lot of "stuff", but rather, to free yourself from everything except that which is essential, and that which you love.

I'm still working on this, and I probably will be until I die. That said, there are a few essential things I think any self respecting geek should have, things I use all the time and I truly love – and I feel it's my responsibility to let my fellow geeks, and the spouses and significant others of geeks, know about them. Otherwise you might end up with yet another WiFi Detecting Shirt as a gift this year, and that'd just be … sad, for everyone involved. So consider this a public service, and feel free to share this post, lest you show up to work in January and find yourself and all your coworkers wearing Wifi Detecting Shirts.

As I wrote in What's On Your Utility Belt? I've been carrying LED flashlights since 2005, and just in that time the average LED flashlight has gone from bright, to very bright, to amazingly bright, to ridiculously blinding laser-like bright. You can thank Haitz's Law for that:

[Haitz's Law] states that every decade, the cost per lumen (unit of useful light emitted) falls by a factor of 10, the amount of light generated per LED package increases by a factor of 20, for a given wavelength (color) of light. It is considered the LED counterpart to Moore's law, which states that the number of transistors in a given integrated circuit doubles every 18 to 24 months. Both laws rely on the process optimization of the production of semiconductor devices.

Or, as I like to call it, "why you will be regularly blinded by flashlights for the rest of your natural life." But on the plus side, it also means that today even inexpensive LED flashlights are plenty bright for all but the most niche applications. You no longer have to pay a big premium to get one that's usefully bright. LED lights are so awesome, in fact, that I own and recommend no less than three form factors:

  1. Fenix HL21 ($35)

    Fenix-hl21

    If you do any kind of DIY work at all, at some point you're going to want a focused light exactly where you are looking. If you can get over the "hey, I have this lamp strapped to my head and I look like a dork" factor, headlamps are ridiculously convenient. I had a much less bright (~40 lumens) headlamp and switching to this 90 lumen HL21 was a major improvement. I use this thing all the time. Looking cool is overrated.

  2. Fenix E21 ($35)

    Fenix-e21

    The E21 is much smaller than your typical full-size flashlight but it is every bit as bright as those giant police-baton like Maglites. It runs off two ubiquitous AA batteries, and has a pleasingly simple design, with an obvious switch in the rear and only two configurable light levels: low (48 lumens) and high (150 lumens). This is a flashlight you could buy your parents without baffling them. We own three, and each of our cars has one in the glove box. This is, in my opinion, what LED lights were meant to be.

  3. Fenix LD01R4 ($40)

    Fenix-ld01

    The latest revision of the LD01 is the proverbial Every Day Carry; a compact single AAA flashlight. As long as you have your keys with you, you'll never without a reliable, bright enough light. Twist the cap to balance between runtime and light output; the three modes are 85 lumens for 1 hour, 28 lumens for 3.5 hours, and 9 lumens for 11 hours. Pretty incredible from a single AAA battery! Oh, and I recommend a lithium AAA battery because they run longer and are 1/3 lighter than other types of batteries. Normally I wouldn't care, but the reduced weight is surprisingly noticeable in something you'll have in your pocket all the time.

All these LED lights have one thing in common: batteries. It's unavoidable. Because you're a responsible geek, of course you use modern rechargeable battery technology. And as I wrote in Adventures in Rechargeable Batteries, sophisticated battery chargers are like geek catnip.

Lacrosse Techology BC-900 AlphaPower battery charger

This is the LaCrosse BC1000 ($60), and it's a ton of fun to mess around with. Also, it recharges batteries. It might seem a little spendy, but it can do miraculous things like bring old nearly-dead rechargeable batteries back to life. And it comes with a bunch of actually useful accessories in the box:

  • Nylon carrying bag
  • 4 AA and 4 AAA rechargeable NiMH batteries
  • 4 C size battery adapters
  • 4 D size battery adapters

Yep, you can simulate C and D cells by putting the AA and AAA batteries inside the shells. The only battery type not represented here is the 9 volt. I own two of these LaCrosse chargers, and given the stupid number of AA and AAA powered devices in the house I'm thinking of buying a third. If you're a geek, you almost certainly have 99 battery problems, but armed with this baby, recharging ain't one. And don't forget the low self-discharge NiMH batteries, while you're at it.

Ah, the dremel. I think this Canadian forumgoer expressed it best:

It truly is hard for me to express the joy I feel when I am forced to break out the dremel; the last resort, the "Trojan Horse" of tools. In a dark place when all other tools abandon me and leave me heartbroken, the dremel always provides a loving shoulder to help complete my tasks. The dremel is a very selfless tool, he/she has no purpose to which they cling, yet is always willing to assist its fellow tools in completing theirs...

Drill strip a screw? The dremel can help... The jigsaw leave some nasty edges? dremel can restore them. I like to think of the dremel as the Jesus of tools.

They say Jesus performed many miracles and although it's not thoroughly documented, I believe his first miracle was, in fact, the dremel blueprint (he was a carpenter after all). The good Lord presented me with an image in a dream... I would like to share it.

Jesus-dremel

If you don't own a dremel, I'm sorry, but I'm going to have to ask you to turn in your geek card. The dremel is truly the swiss army knife of DIY projects. Any DIY project.

I use my Dremel about once every few months, mostly for things that I probably shouldn't even be attempting. But that's the beauty of the Dremel. It doesn't judge; it just helps you get s**t done, by any means necessary. I don't recommend buying a big Dremel kit to start, because it's hard to tell which accessories you'll actually want or need until you begin using this insanely versatile tool. I suggest starting with the entry-level high power Dremel kit ($90).

Dremel-4000

Finally, I have to put in a mention for an updated version of what is probably the most frequently used thing on my keychain, with the biggest bang for the gram other than my front door key -- the Leatherman Squirt PS4 ($24).

Leatherman-squirt-ps4

That's right, you no longer have to face the terrible existential conundrum of choosing between pliers or scissors. The new PS4 model now includes both pliers and scissors. This is nothing less than a Christmas miracle, people! (Oh yeah, and get this awesome tiny carabiner to attach it to your keychain so you can easily detach it when you need to bust it out.)

So that's it this year. Nothing extravagant. Nothing too expensive. No frills. Just essential stuff I love and use regularly. I hope you, or someone you love, will love them too.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Posted by Jeff Atwood    31 Comments

Fast Approximate Anti-Aliasing (FXAA)

December 7, 2011

Anti-aliasing has an intimidating name, but what it does for our computer displays is rather fundamental. Think of it this way -- a line has infinite resolution, but our digital displays do not. So when we "snap" a line to the pixel grid on our display, we can compensate by imagineering partial pixels along the line, pretending we have a much higher resolution display than we actually do. Like so:

2d-anti-aliasing

As you can see on these little squiggly black lines I drew, anti-aliasing produces a superior image by using grey pixels to simulate partial pixels along the edges of the line. It is a hack, but as hacks go, it's pretty darn effective. Of course, the proper solution to this problem is to have extremely high resolution displays in the first place. But other than tiny handheld devices, I wouldn't hold your breath for that to happen any time soon.

This also applies to much more complex 3D graphics scenes. Perhaps even more so, since adding motion amplifies the aliasing effects of all those crawling lines that make up the edges of the scene.

No-aa-vs-4x-aa

But anti-aliasing, particularly at 30 or 60 frames per second in a complex state of the art game, with millions of polygons and effects active, is not cheap. Per my answer here, you can generally expect a performance cost of at least 25% for proper 4X anti-aliasing. And that is for the most optimized version of anti-aliasing we've been able to come up with:

  1. Super-Sampled Anti-Aliasing (SSAA). The oldest trick in the book - I list it as universal because you can use it pretty much anywhere: forward or deferred rendering, it also anti-aliases alpha cutouts, and it gives you better texture sampling at high anisotropy too. Basically, you render the image at a higher resolution and down-sample with a filter when done. Sharp edges become anti-aliased as they are down-sized. Of course, there's a reason why people don't use SSAA: it costs a fortune. Whatever your fill rate bill, it's 4x for even minimal SSAA.

  2. Multi-Sampled Anti-Aliasing (MSAA). This is what you typically have in hardware on a modern graphics card. The graphics card renders to a surface that is larger than the final image, but in shading each "cluster" of samples (that will end up in a single pixel on the final screen) the pixel shader is run only once. We save a ton of fill rate, but we still burn memory bandwidth. This technique does not anti-alias any effects coming out of the shader, because the shader runs at 1x, so alpha cutouts are jagged. This is the most common way to run a forward-rendering game. MSAA does not work for a deferred renderer because lighting decisions are made after the MSAA is "resolved" (down-sized) to its final image size.

  3. Coverage Sample Anti-Aliasing (CSAA). A further optimization on MSAA from NVidia [ed: ATI has an equivalent]. Besides running the shader at 1x and the framebuffer at 4x, the GPU's rasterizer is run at 16x. So while the depth buffer produces better anti-aliasing, the intermediate shades of blending produced are even better.

Pretty much all "modern" anti-aliasing is some variant of the MSAA hack, and even that costs a quarter of your framerate. That's prohibitively expensive, unless you have so much performance you don't even care, which will rarely be true for any recent game. While the crawling lines of aliasing do bother me, I don't feel anti-aliasing alone is worth giving up a quarter of my framerate and/or turning down other details to pay for it.

But that was before I learned that there are some emerging alternatives to MSAA. And then, much to my surprise, these alternatives started showing up as actual graphics options in this season's PC games -- Battlefield 3, Skyrim, Batman: Arkham City, and so on. What is this FXAA thing, and how does it work? Let's see it in action:

No AA 4x MSAA FXAA
Noaa-closeup-1 Msaa-closeup-1 Fxaa-closeup-1

(this is a zoomed fragment; click through to see the full screen)

FXAA stands for Fast Approximate Anti-Aliasing, and it's an even more clever hack than MSAA, because it ignores polygons and line edges, and simply analyzes the pixels on the screen. It is a pixel shader program documented in this PDF that runs every frame in a scant millisecond or two. Where it sees pixels that create an artificial edge, it smooths them. It is, in the words of the author, "the simplest and easiest thing to integrate and use".

Fxaa-algorithm

FXAA has two major advantages:

  1. FXAA smooths edges in all pixels on the screen, including those inside alpha-blended textures and those resulting from pixel shader effects, which were previously immune to the effects of MSAA without oddball workarounds.
  2. It's fast. Very, very fast. Version 3 of the FXAA algorithm takes about 1.3 milliseconds per frame on a $100 video card. Earlier versions were found to be double the speed of 4x MSAA, so you're looking at a modest 12 or 13 percent cost in framerate to enable FXAA -- and in return you get a considerable reduction in aliasing.

The only downside, and it is minor, is that you may see a bit of unwanted edge "reduction" inside textures or in other places. I'm not sure if it's fair to call this a downside, but FXAA can't directly be applied to older games; games have to be specifically coded to call the FXAA pixel shader before they draw the game's user interface, otherwise it will happily smooth the edges of on-screen HUD elements, too.

The FXAA method is so good, in fact, it makes all other forms of full-screen anti-aliasing pretty much obsolete overnight. If you have an FXAA option in your game, you should enable it immediately and ignore any other AA options.

FXAA is an excellent example of the power of simple hacks and heuristics. But it's also a great demonstration of how attacking programming problems from a different angle -- that is, rather than thinking of the screen as a collection of polygons and lines, think of it as a collection of pixels -- can enable you to solve computationally difficult problems faster and arguably better than anyone thought possible.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Posted by Jeff Atwood    31 Comments

Bias Lighting

November 7, 2011

I've talked about computer workstation ergonomics before, but one topic I didn't address is lighting. We computer geeks like it dark. Really dark. Ideally, we'd be in a cave. A cave … with an internet connection.

You-read-my-doormat

The one thing that we can't abide is direct overhead lighting. Every time the overhead light gets turned on in this room, I feel like a Gremlin shrieking Bright light! Bright light! Oh, how it burns!

But there is a rational basis for preferring a darkened room. The light setup in a lot of common computing environments causes glare on the screen:

If your room's lit, as most are, by fittings hanging from the ceiling, you'll be wanting to set up your monitor so that you don't see reflections of the lights in it. The flat screens on many modern monitors (like the excellent Samsung I review here) help, because they reflect less of the room behind you. And anti-reflective screen coatings are getting better and better too. But lots of office workers still just can't avoid seeing one or more ceiling fluoros reflected in their screen.

A good anti-reflective coating can reduce most such reflections to annoyance level only. But if you can see lights reflected in your screen, you can probably also directly see lights over the top of your monitor. Direct line of sight, or minimally darkened reflected line of sight, to light sources is going to give you glare problems.

Glare happens when there are small things in your field of vision that are much brighter than the general scene. Such small light sources can't be handled well by your irises; your eyes' pupil size is matched to the overall scene illumination, and so small light sources will appear really bright and draw lines on your retinas. The more of them there are, and the brighter they are, the more work your eyes end up doing and the sooner they'll get tired.

While a darkened room is better for viewing most types of computer displays, it has risks of its own. It turns out that sitting in a dark room staring at a super bright white rectangle is … kind of bad for your eyes, too. It doesn't help that most LCDs come from the factory with retina-scorching default brightness levels. To give you an idea of how crazy the defaults truly are, the three monitors I'm using right now have brightness set to 25/100. Ideally, your monitors shouldn't be any brighter than a well-lit book. Be sure to crank that brightness level down to something reasonable.

You don't want total darkness, what you want is some indirect lighting – specifically bias lighting. It helps your eyes compensate and adapt to bright displays.

"[Bias lighting] works because it provides enough ambient light in the viewing area that your pupils don't have to dilate as far. This makes for less eyestrain when a flashbang gets thrown your way or a bolt of lightning streams across the screen," he told Ars. "Because the display is no longer the only object emitting light in the room, colors and black levels appear richer than they would in a totally black environment. Bias lighting is key in maintaining a reference quality picture and reducing eye-strain."

Bias lighting is the happy intersection of indirect lighting and light compensation. It reduces eye strain and produces a better, more comfortable overall computing display experience.

Bias-lighting

The good news is that it's trivially easy to set up a bias lighting configuration these days due to the proliferation of inexpensive and bright LEDs. You can build yourself a bias light with a clamp and a fluorescent bulb, or with some nifty IKEA LED strips and double-sided foam tape. It really is that simple: just strap some lights to the back of your monitors.

I'm partial to the IKEA Dioder and Ledberg technique myself; I currently have an array of Ledbergs behind my monitors. But if you don't fancy any minor DIY work, you can try the Antec Halo 6 LED Bias Lighting Kit. It also has the benefit of being completely USB powered.

Of course, lighting conditions are a personal preference, and I'd never pitch bias lighting as a magic bullet. But there is science behind it, it's cheap and easy to try, and I wish more people who regularly work in front of a computer knew about bias lighting. If nothing else, I hope this post gets people to turn their LCD monitors down from factory brightness level infinity to something a tad more gentle on the old Mark I Eyeball.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Posted by Jeff Atwood    51 Comments

On Parenthood

October 24, 2011

Our son was born March 12th, 2009. He's a little over two and a half years old. Now, I am the wussiest wuss to ever wuss up the joint, so take everything I'm about to say with a grain of salt – but choosing to become a parent is the hardest thing I have ever done. By far. Everything else pales in comparison.

My feelings on this matter are complex. I made a graph. You know, for the children.

Children

That one percent makes all the difference.

It's difficult to explain children to people who don't yet have children, because becoming a parent is an intensely personal experience. Every child is different. Every parent is different. Every culture has their own way of doing things. The experience is fundamentally different for every new parent in the world, yet children are the one universally shared thing that binds our giant collective chain letter of human beings together, regardless of nationality and language. How do you explain the unexplainable?

Well, having children changes you. Jonathan Coulton likens it to becoming a vampire.

I was having a conversation with a friend who had recently become a parent, and she reminded me of something I had forgotten about since my daughter was born. She was describing this what-have-I-done feeling – I just got everything perfect in my life, and then I went and messed it all up by having a baby. I don’t feel that way anymore, but the thought certainly crossed my mind a few times at the beginning. Eventually you just fall in love and forget about everything else, but it’s not a very comfortable transition. I compare the process to becoming a vampire, your old self dies in a sad and painful way, but then you come out the other side with immortality, super strength and a taste for human blood. At least that’s how it was for me. At any rate, it’s complicated.

Maybe tongue in cheek, but not that far from the truth, honestly. Your children, they ruin everything in the nicest way.

Before Henry was born, I remembered Scott Hanselman writing this odd blurb about being a parent:

You think you love you wife when you marry her. Then you have a baby and you realize you'd throw your wife yourself under a bus to save your baby. You can't love something more.

Nuts to that, I thought. Hanselman's crazy. Well, obviously he doesn't love his wife as much as I love mine. Sniff. Babies, whatever, sure, they're super cute on calendars, just like puppies and kittens. Then I had a baby. And by God, he was right. I wouldn't just throw myself under a bus for my baby, I'd happily throw my wife under that bus too – without the slightest hesitation. What the hell just happened to me?

As an adult, you may think you've roughly mapped the continent of love and relationships. You've loved your parents, a few of your friends, eventually a significant other. You have some tentative cartography to work with from your explorations. You form ideas about what love is, its borders and boundaries. Then you have a child, look up to the sky, and suddenly understand that those bright dots in the sky are whole other galaxies.

You can't possibly know the enormity of the feelings you will have for your children. It is absolutely fucking terrifying.

When I am holding Henry and I tickle him, I can feel him laughing all the way to his toes. And I realize, my God, I had forgotten, I had completely forgotten how unbelievably, inexplicably wonderful it is that any of us exist at all. Here I am with this tiny, warm body so close to me, breathing so fast he can barely catch up, sharing his newfound joy of simply being alive with me. The sublime joy of this moment, and all the other milestones – the first smile, the first laugh, the first "dada" or "mama", the first kiss, the first time you hold hands. The highs are so incredibly high that you'll get vertigo and wonder if you can ever reach that feeling again. But you peak ever higher and higher, with dizzying regularity. Being a new parent is both terrifying and exhilarating, a constant rollercoaster of extreme highs and lows.

It's also a history lesson. The first four years of your life. Do you remember them? What's your earliest memory? It is fascinating watching your child claw their way up the developmental ladder from baby to toddler to child. All this stuff we take for granted, but your baby will painstakingly work their way through trial and error: eating, moving, walking, talking. Arms and legs, how the hell do they work? Turns out, we human beings are kind of amazing animals. There's no better way to understand just how amazing humans are than the front row seat a child gives you to observe it all unfold from scratch each and every day, from literal square zero. Children give the first four years of your life back to you.

I wasn't sure how to explain meeting new people to Henry, so I decided to just tell him we've met a new "friend" every time. Now, understand that this is not at all the way I view the world. I'm extremely wary of strangers, and of new people in general with their agendas and biases and opinions. I've been burned too many times. But Henry is open to every person he meets by default. Each new person is worth greeting, worth meeting as a new experience, as a fellow human being. Henry taught me, without even trying to, that I've been doing it all wrong. I realized that I'm afraid of other people, and it's only my own fear preventing me from opening up, even a little, to new people that I meet. I really should view every new person I meet as a potential friend. I'm not quite there yet; it's still a work in progress. But with Henry's help, I think I can. I had absolutely no idea my child would end up teaching me as much as I'm teaching him.

Having a child is a lot like running a marathon. An incredible challenge, but a worthwhile and transformative experience. It leaves you feeling like you truly accomplished something for all that effort. After all, you've created something kind of amazing: a person.

Bob: It gets a whole lot more complicated when you have kids.

Charlotte: It's scary.

Bob: The most terrifying day of your life is the day the first one is born.

Charlotte: Nobody ever tells you that.

Bob: Your life, as you know it... is gone. Never to return. But they learn how to walk, and they learn how to talk, and you want to be with them. And they turn out to be the most delightful people you will ever meet in your life.

It's scary and it's wonderful in equal measure. So why not have another baby? Or so we thought.

Atwood-babbies

Turns out, we're having two babies. Both are girls, due in mid-February 2012.

I've been told several times that you should never be crazy enough to let the children outnumber you. I hope to ultimately win the War of the Lady Babies, but when it comes to children, I think all anyone can ever realistically hope for is a peaceful surrender.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.
Posted by Jeff Atwood    244 Comments

Multiple Video Cards

October 17, 2011

Almost nobody should do what I am about to describe – that is, install and use more than one video card. Nobody really needs that much graphics performance. It's also technically complex and a little expensive. But sometimes you gotta say to hell with rationality and embrace the overkill.

Why? Battlefield 3, that's why.

Battlefield-3-caspian

I've been a fan of the series from the earliest days of Battlefield 1942, and I lost hundreds of hours to Battlefield 2 and Battlefield: Bad Company 2. I even wrote about the original Battlefield 2 demo on this very blog six years ago. So, yeah, I'm a superfan from way back. As much as I was anticipating Battlefield 3, I have to say the open beta convinced me it is everything I always wanted, and more. Glorious sandbox warfare on an enormous, next-generation destructible battlefield is a beautiful thing.

Since PC was the lead platform for Battlefield 3, it is the rare current game that isn't dumbed down to PS3 and Xbox 360 console levels; it is a truly next-generation engine designed to scale over the next few years of PC performance.

Battlefield-3-graphics-levels

This also means it's going to be rough on current PCs; at a minimum, you'll need a fast dual core CPU, and a modern video card with 512mb or more video memory. It only goes up from there. Way up. Like most games, Battlefield 3 is far more limited by video card performance than CPU performance. This is normally the place where I'd trot out my standard advice urging you to buy one of the new hotness video cards released this holiday season. But unfortunately due to difficulties with the 40nm to 28nm process transition for ATI and NVIDIA, there aren't any new hotness video cards this year.

So what's a poor performance addicted Battlefield superfan to do? Double down and add another video card for more performance, that's what. Both ATI and NVIDIA have offered mature multi-GPU support for a few years now, and they've mostly settled on a simple Alternate Frame Rendering (AFR) strategy where each video card alternates between frames to share the graphics rendering work.

Alternate-frame-rendering-diagram

The little arrow there is a bridge attachment that you place on both cards so they can synchronize their work. Yes, there is a bit of overhead, but it scales surprisingly well, producing not quite double the performance but often in the area of 1.8x or so. Certainly enough to make it worth your while. You can technically add up to four video cards in this manner, but as with multiple CPUs your best bang for the buck is adding that second one; the third, fourth, and beyond provide increasingly diminished returns.

The good news is that the market crash in BitCoin GPU mining (if you don't know what this is, don't ask… please) means there is a glut of recent video cards up for sale on eBay right now. I have the same AMD Radeon HD 5870 that I've had since early 2010. I picked up another 5870 on eBay for a mere $170. This is a great video card, well ahead of its time when it was originally released, and even now only 10% slower than the fastest video card AMD makes. I simply dropped the second card in my system and installed the bridge connector.

Dual-radeon-gpus

You may recognize this computer as a further tweaked version of my last build (which is still awesome, by the way, and highly recommended). Anyway, for this to work, you'll need to establish a few things about your computer before rushing out and buying that second video card.

  1. A motherboard that has two video card capable PCI Express slots. Most aftermarket and enthusiast motherboards have this, but if you bought a system from say, Dell, it's less clear.
  2. A power supply with enough headroom to drive two video cards. Warning: modern gaming video cards are major power hogs -- they easily pull 100 to 200 watts under load. Each. Sometimes more than that! Be absolutely sure you have a quality power supply rated for a minimum of 600 watts. Each video card will have two power connectors, either 6 or 8 pin. Check that your power supply offers enough connectors, or that you have converters on hand.
  3. A case with sufficient airflow to dissipate the 400 to 800 watts of CPU and GPU heat that you'll be generating. Understand that this is serious amounts of heat while gaming, way more than even the highest of high end PCs would normally produce. Yes, it is possible to do this quietly (at least in the typical idle case), but it will take some engineering work.

Beyond that, I found there are some additional peculiarities of multi-GPU systems that you need to be aware of.

  • Make sure that the two cards you use are not only of the exact same family (minor vendor differences are OK) but also have identical clock and memory speeds. It's not supposed to matter, but I found that it did and I had to flash one of my cards to set it to default speeds to match the other card.
  • Do not attempt to overclock your system while getting the multiple GPUs up and running. In particular, be extremely careful not to mess with the bus speed as timings are critical when dealing with two GPUs on the PCI Express bus synchronizing their work. Trust me on this one.
  • Do a clean video driver remove and install the very very latest video drivers after putting the second card in. I recommend Driver Sweeper to remove any traces of your old drivers before rebooting.

Don't say I didn't warn you about this stuff, because I said it would be technically complex in the first paragraph. But after a (lot) of teething pains, I'm happy to report that multiple GPUs really does work as advertised. I can crank up games to the absolute maximum settings on my 27" monitor and get nearly constant 60 frames per second. As you can see in the below example, we go from 44 fps to 77 fps in Battlefield: Bad Company 2.

Battlefield-bad-company-2-gpu-scaling

Now, Battlefield 3 (beta) is so very bleeding edge that I can't quite get it to max settings even with two GPUs in play. But I can now run very high settings, much higher than I could with a single GPU.

To be honest, it's unlikely I will continue with multiple GPUs through 2012 when the next-generation video cards are released. With every new video card introduction, you're supposed to get about the same performance in the new card as you did with two previous generation video cards working together. So at best this is a sort of sneak preview, cheating time by pretending we have a next-generation video card today. There are obvious efficiencies involved in performing that parallelization on a single GPU die rather than through two distinct video cards sitting on the PCI bus.

There's also the issue of micro-stuttering. I personally haven't found that to be a big problem unless you're pushing the graphics settings beyond what even two cards can reliably deliver. But if the frame rate dips low enough, the overhead of synchronization between the cards can interfere with overall frame rate in a perceptible way.

A single fast video card is always going to be the simpler, easier, and cheaper route. But multiple video cards sure is nifty tech in its own right, and it wasn't too expensive to get started with at $170. In the meantime, I'm having a ball playing with it, and I am dying to test my configuration with the final release of Battlefield 3 on October 25th. Join me if you like!

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Posted by Jeff Atwood    40 Comments

Serving at the Pleasure of the King

October 15, 2011

I enjoy my iPhone tremendously; I think it's the most important product Apple has ever created and one they were born to make. As a consumer who has waited far too long for the phone industry to get the swift kick in the ass it so richly deserved, I'm entirely on Apple's side here.

But as a software developer, I am deeply ambivalent about an Apple dominated future. Apple isn't shy about cultivating the experience around their new iOS products and the App Store. There are unusually strict, often mysterious rules around what software developers can and cannot do -- at least if they want entry into the App Store. And once you're in, the rules can and will change at any time. Apple has cracked down several times already:

The developers involved are contractually prevented from even discussing specifically what happened to them by the terms of the app store. Those frustrating, inconsistent, opaque App Store experiences led developers to coin parodies such as Apple's Three Laws of Developers.

  1. A developer may not injure Apple or, through inaction, allow Apple to come to harm.
  2. A developer must obey any orders given to it by Apple, except where such orders would conflict with the First Law.
  3. A developer must protect its own existence as long as such protection does not conflict with the First or Second Law.

It is absolutely clear who is in charge when you submit an application to the App Store. Apple developers serve at the pleasure of the king.

Louis-xiv-painting

In Apple's defense, this is done in the name of protecting the consumers from malicious, slimy, or defective applications. Sort of like Nintendo's Seal of Approval, I guess.

The court of the king is a lucrative place to be, but equally dangerous. While upgrading my iPhone to iOS 5 – an excellent upgrade, by the way – I was surprised to discover the following blurb in the feature notes:

Safari Reader displays web articles sans ads or clutter so you can read without distractions. Reading List lets you save interesting articles to peruse later [like the popular Instapaper application], while iCloud keeps your list updated across all your devices.

Apple has since changed the page, but at the time I read it, there was a direct linked reference to Instapaper, the popular "save this webpage to read later" application which Reading List is a clone of. I distinctly remember this mention, because I was shocked that they would be so open and overt about replacing a beloved third-party application. Perhaps it made Apple uncomfortable too; maybe that's why they pulled the Instapaper text and link.

If Microsoft added a feature to Windows that duplicated a popular application's functionality, developers would be screaming bloody murder and rioting in the, er, blogs and web forums. But in the Mac world, if the king deems it necessary, then so it must be.

When iOS 5 and Lion ship, Apple will show a much larger percentage of iOS-device owners that saving web pages to read later is a useful workflow and can dramatically improve the way they read.

If Reading List gets widely adopted and millions of people start saving pages for later reading, a portion of those people will be interested in upgrading to a dedicated, deluxe app and service to serve their needs better. And they’ll quickly find Instapaper in the App Store.

I've met Marco Arment, the developer of Instapaper, and I like Marco. He's even been a guest on the Stack Exchange podcast. This is a nice, optimistic interpretation, but the reality is a little scarier. I'm struggling to understand why anyone would buy Instapaper when they can click a button in Safari and have that web page delivered to any of their Macs or iOS devices for later reading via iCloud.

Ah, but wait – what about offline support? Yes, that's something only Instapaper can deliver! Or can it?

A common scenario: an Instapaper customer is stocking up an iPad for a long flight. She syncs a bunch of movies and podcasts, downloads some magazines, and buys a few new games, leaving very little free space. Right before boarding, she remembers to download the newest issue of The Economist. This causes free space to fall below the threshold that triggers the [new iOS 5 space] cleaner, which — in the background, unbeknownst to her — deletes everything that was saved in Instapaper. Later in the flight, with no internet connectivity, she goes to launch Instapaper and finds it completely empty.

That's the problem with kings, you see. Their rule is absolute law, but they can be capricious, erratic, and impulsive. If you're lucky enough to live under the rule of a fair and generous king, then you'll do well. But historically speaking, monarchies have proven to be … unreliable.

Louis-xiv-convenience-store

I tend to agree with Marco that this is, in the big scheme of things, a minor technical problem. A private application cache not subject to iCloud syncing and space limitations would fix it. But it speaks volumes that Marco – a dedicated subject of the king – apparently had no idea this change was coming until it was on top of him. It's negatively impacting his Instapaper business and his customers. It's also concerning that this issue wasn't resolved or at least raised as a serious concern during the lengthy iOS 5 beta. Perhaps Apple's legendary secrecy is to blame. I honestly don't know.

As a consumer, I like that Apple is perfectly willing to throw its software developers under a bus to protect me (or, more cynically, Apple itself). But as a software developer, I'm not sure I can cope with that and I am unlikely to ever develop anything for an iOS device as a result. If you choose to deliver software in the Apple ecosystem, this is simply the tradeoff you've chosen to make. Apple developers serve at the pleasure of the king.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.
Posted by Jeff Atwood    57 Comments

The Gamification

October 12, 2011

When Joel Spolsky and I set out to design the Stack Exchange Q&A engine in 2008 – then known as Stack Overflow – we borrowed liberally and unapologetically from any online system that we felt worked. Some of our notable influences included:

  • Reddit and Digg voting
  • Xbox 360 achievements
  • Wikipedia editing
  • eBay karma
  • Blogs and blog comments
  • Classic web bulletin boards

All these elements were folded up into the Stack Exchange Q&A engine, so that we might help people create useful artifacts on the internet while learning with and among their peers. You know the old adage that good artists copy, great artists steal? That quote is impossible to source, but it means we were repurposing these elements we liked.

So, what do Picasso and T.S. Eliot mean? They say, in the briefest of terms: take old work to a new place. Steal the Google site, strip down what works (fast load, nonexistent graphics, small quirky changes that delight) and use the parts on your own site. Look at the curve of a Coke Bottle and create a beautiful landscape painting with it. Take the hairline pinstriping on the side of somebody’s car, reimagine it on your print job. Find inspiration in the world you live in, where nothing is truly new so that everything has the potential to be innovative.

Unfortunately, the elements we liked were often buried in mounds of stuff that we ... sort of hated. So extracting just the good parts and removing the rest was part of the mission. If you're lucky enough to have a convenient villain to position yourself against, that might be all you need.

Traditional web bulletin board systems have a design that was apparently permanently frozen in place circa 2001 along with Windows XP. Consider this typical forum thread.

Web-forum-thread-screenshot

Here is the actual information from that forum thread.

Web-forum-thread-information-screenshot

Based on the original size of those screenshots, only 18 percent of that forum thread page is content. The other 82 percent is lost to signatures, avatars, UI doohickeys, and other web forum frippery that has somehow become accepted as "the way things are done". I regularly participate in several expert niche bulletin boards of various types today, and they're all built the same way. Nobody complains.

But they should.

This is the status quo that we're up against. Yes, we fixed it for programmers with Stack Overflow, but why stop there? We want to liberate all the brilliant experts stuck in these horrible Soviet-era concrete block housing forums all over the web. We'd like to introduce them to the focused, no-nonsense Stack Exchange Way, a beautiful silo of pure Q&A signal without all the associated web forum gunk.

There's only one teeny-tiny obstacle in our way. As a great programmer I worked with once said:

It's the damn users. They've ruined every program I've ever created.

Every web forum is the way it is because users wanted it that way. Yes, the design of the forum software certainly influences behavior, but the classic 2001-era web forum paradigm assumed that what users wanted made sense for the rest of the larger internet. As it turns out, groups are their own worst enemy. What groups want, and what the rest of the world needs, are often two very different things. Random discussion is fine for entertainment, but it's not particularly useful, nor does it tend to generate the kind of artifacts that will be relevant a few years from now like Wikipedia does. So then the problem becomes how do you encourage groups to do what's best for the world rather than their own specific, selfish needs?

When I looked at this problem, I felt I knew the answer. But there wasn't a word for it in 2008. Now there is: Gamification.

Gamification is the use of game design techniques and mechanics to solve problems and engage audiences. […] Gamification works by … taking advantage of humans' psychological predisposition to engage in gaming. The technique can encourage people to perform chores that they ordinarily consider boring, such as completing surveys, shopping, or reading web sites.

I had no idea this Wikipedia article even existed until a few months ago, but we are featured prominently in it. It is true that all our stolen ideas about reputation systems, achievements, identity, and vote scoring are in place specifically to encourage the adoption of the brave new no-nonsense, all-signal Stack Exchange Q&A model. Without those incentive systems, when left to their own devices, what you get is … well, every forum ever created. Broken by design.

Yes, we have ulterior motives, but let me explain why I think gaming elements are not tacked on to the Stack Exchange Q&A engine, but a natural and essential element of the design from day one.

Learning is (supposed to be) fun

I've had this concept in my head way before the web emerged, long before anyone coined the term "Gamification" in 2010. In fact, I'd trace my inspiration for this all the way back to 1983.

Beagle Brothers: Our programs are FUN to use. Our instructions are CLEAR and complete.

For programmers, everything we know is pretty much guaranteed to be obsolete in 10 years if we're lucky, and 5 years if we aren't. It's changing all the time. The field of programming is almost by definition one of constant learning. Programming is supposed to be fun – and it is, if you're doing it right. Nobody taught me that better than the Beagle Bros on my Apple II. Why can't learning in every other subject matter be just as enjoyable?

Games are learning aids

There's a long, rich history of programmers as gamers. Oftentimes, the whole reason we became programmers in the first place is because we wanted to move beyond being a mere player and change the game, control it, modify its parameters, maybe even create our own games.

Basic-computer-games

We used games to learn how to program. To a programmer, a game is a perfectly natural introduction to real programming problems. I'd posit that any field can use games as an introduction to the subject matter – and as a reinforcement to learning.

Games help people work toward a goal

It's something of a revelation to me that solid game design can defeat the Greater Internet F**kwad Theory. Two great examples of this are Counter-Strike and Team Fortress. Both games are more than ten years old, but they're still actively being played right now, by tens of thousands of people, all anonymous … and playing as cohesive teams!

The game's objectives and rules are all cleverly constructed to make working together the most effective way to win. None of these players know each other; the design of the game forces players to work together, whether they want to or not. It is quite literally impossible to win as a single lone wolf.

Counter-strike-italy-start

I haven't ever quite come out and said it this way, but … I played a lot of Counter-Strike from 1998 to 2001, and Stack Overflow is in many ways my personal Counter-Strike. It is a programmer in Brazil learning alongside a programmer in New Jersey. Not because they're friends – but because they both love programming. The design of Stack Overflow makes helping your fellow programmers the most effective way to "win" and advance the craft of software development together.

And I say we all win when that happens, no matter which profession we're talking about.

I feel a little responsible for "Gamification", since we're often cited as an example (even, much to my chagrin, on Wikipedia). I wanted to clear up exactly why we made those choices, and specifically that all the gaming elements are there in service of a higher purpose. I play the Stack Exchange game happily alongside everyone else, collecting reputation and badges and rank and upvotes, and I am proud to do so, because I believe it ultimately helps me become more knowledgeable and a better communicator while also improving the very fabric of the web for everyone. I hope you feel the same way.

(If you'd like to learn more about the current state of Gamification, I highly recommend Sebastian Deterding's page, and specifically his Meaningful Play: Getting Gamification Right presentation.)

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Posted by Jeff Atwood    39 Comments

Cutting the Gordian Knot of Web Identity

September 6, 2011

Perhaps you've seen this recent XKCD about password choice?

To anyone who understands information theory and security and is in an infuriating argument with someone who does not (possibly involving mixed case), I sincerely apologize.

It prompted a spirited debate – even on our very own Security Stack Exchange – about the merits of the argument presented there. Now, to be clear, I'm completely on Randall's side here; I'm all for passphrases over passwords, and I have been for years.

But this is merely one symptom of a much larger disease: identity on the internet. Every time you touch a website that actually cares who the heck you are – and this is an increasingly large list of sites as the web matures – you have to, sigh, "log in". And logging in inevitably requires you to create a username and a password. Over and over and over and over. And, oh by the way, you'll be logging in again each and every time on every browser and every computer and every device you own. It's a great system. And by "great" I mean fracking terrible.

This isn't just tedious busywork for us, the poor web users, it's also downright dangerous as I explained in The Dirty Truth About Web Passwords. It's a near-impossible problem, an intractable Gordian Knot. So I'm going to answer one comic with another.

Watchmen-gordian-knot

The problem is not choosing better passwords for the dozens or hundreds of web sites we have to log in. The problem is passwords.

Thus, the only real cure to the disease of identity on the web is to get rid of passwords altogether.

Yes, you read that correctly. But "Jeff", you might say, "how can we possibly log in to websites without our beloved, mile-long list of site-specific usernames and passwords?" I'm so glad you asked! Try to make time in your busy schedule of account and password creation to read a few more paragraphs into this post and I'll attempt to explain my crazy scheme.

We could use our internet driver's license and log in to a particular website using our existing Google, Facebook, Twitter, or OpenID credentials. This works, but it assumes a lot; is the website enlightened enough to accept third party logins, or is there a political agenda (or delusions of grandeur) preventing them from recognizing any form of identity other than their own? To be fair, accepting third party identity is hard and undeniably adds complexity. There are a million ways to get it wrong, and only a handful of sites that get it right. I like to think Stack Exchange is one of the websites that gets this right, but I'll fully acknowledge that it is challenging to get there. Unfortunately, the path of least resistance for web identity leads inexorably to one sad, depressing, dysfunctional place:

Username-password-input

Yep. Get used to it. Username. Password. For every single website you'll ever visit. On every single device you'll ever own. Forever. Until the end of time. Oh God.

Lately I've begun to hope there might be a viable solution, even outside the third-party logins I've championed for the last 3 years. A way of absolving users of username and password selection. Like Alexander's solution to the Gordian Knot, it might be a bit scary in its … absolutism. But anything has to be better than the unspeakable terror of a million logins on a million different websites on a million different devices. Right? Right?

(Warning, Extreme Hand Waviness Ahead: while I do honestly believe the techniques described below would work, I am glossing over many of the technical implementation details. This is absolutely not intended to be a spec but an "I Have a Dream" outline. Feel free to help me clarify and expand on the necessary details by blogging a more technical plan for this in any form you like.)

Imagine, if you will, visiting a new website and deciding you'd like to create an account there. You click the "Create New Account" link and then …

  • The website presents a secure account creation page decorated with specific meta tags that indicate this page supports automated account creation with a standard(ish) set of user info HTML form fields.
  • The browser, seeing these meta tags in the page, does not present the page to the user but retrieves the user's standard information fields like name, email address, etcetera from some form of secure https cloud storage, and readies them. The browser will also automatically select a completely random, cryptographically secure password for the new account.
  • The browser must, unfortunately, prompt the user with a confirm dialog containing a CAPTCHA at this point to ensure that the signup process isn't being scripted in any way, and that a real human being actually wants to create an account on this website. While we're there, we might as well confirm the identity data we're about to send to the website (though hopefully the defaults should suffice). Once confirmed, the user credentials and password will be sent to the site and stored securely in the cloud.
  • The website redirects the newly created account to an appropriate page.

There may be some more information that the browser (or the site) needs to ask the user for in there somewhere. But account creation is a one-time event, and in the typical case where you're signing up for some simple website, your preferred defaults should suffice. Caveats aside, look what we have wrought: you clicked "Create New Account", completed a single captcha and clicked OK – now you're logged in to your brand new account on any website.

Once you have an account, it's even simpler. Imagine clicking the "Sign In" link, and then …

  • The website presents a secure login page decorated with specific meta tags that indicate this page supports automated login with a standard set of username and password HTML form fields.
  • The browser, seeing these meta tags in the page, does not present the page to the user but retrieves the user's credentials from some form of secure https cloud storage, and sends them to the site.
  • The site receives the credentials via https, validates them, and returns a valid login cookie to the browser.
  • The browser redirects the now logged-in user to the page they originally wanted to see as a logged in user.

From the perspective of this weary citizen of the web, at least, a miracle just happened. You clicked "Sign In", and you're immediately signed in without having to look at a single stinking username and password field!

Seems like magic, yes? Gotta be a catch, yes? Well, there is. Two catches, to be precise.

  1. Web browsers will have to be rewritten to understand basic identity protocols. I suppose it could be a browser plugin as well, but I'd rest a lot easier knowing basic identity protocols are officially "baked in" to the browser and supported by the Powers That Be, perhaps even as accepted W3C standards. And yes, you will need to log in to your browser at a minimum.
  2. You have to trust "The Cloud" at least a little. There has to be some trusted, secure location on the internet for all your usernames, passwords, and basic identity information to reside. Otherwise this scheme can't possibly work, because how would you log in from your (insert favorite device name here) if it has no access to the secret, hidden list of account information and automagically generated secure passwords created on your behalf?

Identity is fundamental to the modern internet; more and more websites need to know something about who you are to work. The current status quo of thousands of websites with thousands of differing ideas about identity and passwords and account information is beyond broken. We want – no, we demand – that the browser understand and standardize identity the same way it does HTML and CSS. Maybe it's crazy, but I dream of the day when I never need to see another password field in my browser ever again.

I hope you can too.

[advertisement] What's your next career move? Stack Overflow Careers has the best job listings from great companies, whether you're looking for opportunities at a startup or Fortune 500. You can search our job listings or create a profile and let employers find you.

Posted by Jeff Atwood    126 Comments

Nobody's Going to Help You, and That's Awesome

July 27, 2011

I'm not into self-help. I don't buy self-help books, I don't read productivity blogs, and I certainly don't subscribe to self-proclaimed self-help guru newsletters. Reading someone else's advice on the rather generic concept of helping yourself always struck me as a particularly misguided idea.

Apparently I'm not the only person to reach this conclusion, either.

I spent two years reading all the self-help books I could find. As of a year ago, I had read 340 self-help books. Because I’m insane.

My conclusion from all that reading?

95% of self-help books are complete bullshit.

To be clear, I am all for self-improvement. Reading books, blogs, and newsletters by people who have accomplished great things is a fine way to research your own path in life. But these people, however famous and important they may be, are not going to help you.

Unfortunately that's not the answer he wanted. To him, my answer [that nobody was going to help him become successful] was really discouraging. To me, if I was receiving that answer from someone else, it would be really encouraging.

I like being reminded that nobody's going to help me - that it's all up to me. It puts my focus back on the things I can control - not waiting for outside circumstances.

Take it from The Ultimate Productivity Blog:

Ultimate-productivity-blog

Reading self-help advice from other people, however well-intentioned, is no substitute for getting your own damn work done. The sooner you come to terms with this, the better off you'll be.

Get out there and do stuff because you fundamentally enjoy it and because it makes you better. As a writer, as an analyst, as a techie, whatever. Learn to love practicing the fundamentals and do it better each time. Over time, quality does lead to success, but you have to be patient. Really patient. Turns out, "overnight" success takes years. Maybe even decades. This is not a sprint, it's a marathon. Plan accordingly.

For example, I don't care if anyone reads what I write here. I'm writing to satisfy myself first and foremost. If others read it and benefit from it, fantastic -- that's a welcome side effect. If I worry about who is reading, why they're reading, or if anyone is even reading at all, I'd be too paralyzed to write! That'd be the least productive outcome of all.

That's not to say that some introspection about the nature of your work isn't useful. It is. Even the weary self-help student I quoted above concluded that 5% of self-help advice surprisingly wasn't bullshit. The one book he recommended without hesitation? 59 Seconds: Think a Little, Change a Lot.

59-seconds-book-cover

Despite my deep reservations about the genre, I ordered this book based on his recommendation and a number of credible references to it I noticed on the Skeptic Stack Exchange.

Why does this self-help book work when so many others fail? In a word, science! The author goes out of his way to find actual published scientific research documenting specific ways we can make small changes in our behavior to produce better outcomes for ourselves and those around us. It's powerful stuff, and the book is full of great, research backed insights like this one:

A group of participants were asked to select a negative experience. One group of participants were then asked to have a long chat with a supportive experimenter about the event, while a second group were invited to chat about a far more mundane topic - a typical day.

Participants who had spent time talking about their traumatic event thought the chat had been helpful. However, the various questionnaires told a very different story. In reality the chat had no significant impact at all. They might just as well have been chatting about a typical day.

In several studies, participants who have experienced a traumatic event have been encouraged to spend just a few minutes each day writing in a diary-type account of their deepest thoughts and feelings about it. For example, in one study participants who had just been made redundant were asked to reflect upon their deepest thoughts and feelings about their job loss, including how it had affected both their personal and professional lives. Although these types of exercises were both speedy and simple, the results revealed a remarkable boost in their psychological and physical well-being, including a reduction in health problems and an increase in self-esteem and happiness.

The results left psychologists with something of a mystery. Why would talking about a traumatic experience have almost no effect but writing about it yield such significant benefits? From a psychological perspective, talking and writing are very different. Talking can often be somewhat unstructured, disorganized, even chaotic. In contrast, writing encourages the creation of a story line and structure that help people make sense of what has happened and work towards a solution. In short, talking can add to a sense of confusion, but writing provides a more systematic, solution-based approach.

Therefore, the real world change you would make based on this advice – the proverbial 59 seconds on the book jacket – is to avoid talking through traumatic experiences in favor of writing about them. Not because some self-help guru said so, but because the published research data tells us that talking doesn't work and writing does. Not exactly intuitive, since talking through our problems with a friend always feels like the right thing to do, but I have certainly documented many times over the value of writing through a problem.

59 Seconds is so good, in fact, it has rekindled my hopes that our new Stack Exchange Productivity Q&A can work. I'd love for our productivity site to be founded on a scientific basis, and not the blind cult of personality I've come to expect from the self-help industry.

Remember, nobody's going to help you … except science, and if you're willing to put in the required elbow grease each and every day – yourself.

Posted by Jeff Atwood    65 Comments

Building a PC, Part VII: Rebooting

July 18, 2011

I've had more or less the same PC, with various updates, since 2007. I've written about most of it here:

While the advice in those original articles is still quite sound, my old 2007 era case was feeling mighty creaky. I needed a new chassis. I also wanted a motherboard that supported native 6 Gbps SATA for the latest generation of SSDs that truly benefit from them. The buzz around the Sandy Bridge based Core i7-2600k was nearly deafening, and I've fallen completely in love with my last HTPC build based on the same technology. (Oh, and even if you already read that article, read it again because I added new PicoPSU and case information that takes it from awesome to sublime – on the order of 17 watts idle!)

So I decided it was time to build myself a nice Sandy Bridge system. What I ended up with is easily the best case and motherboard combination I've ever laid hands on. Read on!

I cut out a lot of the initial research work by relying on my old, dear friends at Tech Report and their current workstation recommendations:

As for the case, I was impressed by the Tech Report review of the Corsair 600T, which even comes in a heart-stopping pseudo stormtrooper white. WANT.

Corsair-600t-special-edition

When it comes to power supplies, I'm crazy about efficiency, and fortunately there are now lots of so-called "80 Plus Gold" PSUs out there now, offering a staggering 90% efficiency under most loads. Power supply efficiency is important, because the rest of that heat is dumped back into your case. The less efficient your PSU, the more heat buildup you'll have under load. I chose the Seasonic X-760 – which, when bench tested, indeed delivered the promised 90% efficiency – but any high quality 80 Plus Gold model will generally do.

The CPU (and possibly, depending on your tastes, the video card) is the biggest heat generator inside your PC. The better and more efficient the CPU cooler, the quieter your whole system can be. This also affects how much you can overclock. I chose the Thermalright Venomous-X Silent Edition on the basis of it being the current top dog for efficiency, and because it had a simple mounting system. Large coolers can be a real bear to install. And did I mention it comes with an especially quiet fan out of the box, too?

Once I had all the parts in hand, it was a simple matter of building it up, as documented in my previous post series. I adore this Corsair case; it is an absolute joy to work in. Everything in it is cleverly designed, from the rear cable routing area with rubber grommets all over the place for easily passing cables back and forth, to the tool-less 2.5" and 3.5" bays, to the super easily removable side panels. It's like they read a giant list of all my prior complaints with every PC case I've ever used and fixed every. single. last. one of them.

The end result is what you see here:

New-sandy-bridge-build-internal

There are some significant tweaks visible in the above picture that I do recommend:

  1. Use tin snips to remove the rear exhaust grill. We don't need it back there, and the exhaust airflow is critical. Fan grills affect low-speed fan airflow more than you'd think:
    Wire grills also have an effect: ~20%. This was checked with an anemometer on several different fans of 80, 92 and 120mm size, at full and lower speeds. The airflow reduction went as high as 24% but it was never below 19%. At 12V, the reduction in airflow with most fans will be relatively harmless, though there is an increase in turbulence noise (audible to me). But at the low airflow rates SPCR members run fans, I think the airflow reduction is significant.
  2. Install a 140mm rear exhaust fan. The Noctua NF-P14 is expensive at $25 but is cleverly designed to give you 140mm of super-quiet fan in the space a 120mm fan would normally take. It just barely fits in the rear exhaust fan slot with a little nudging. But it does fit; it's the beige fan in the above picture. It also comes with its own speed reducers and accessories.

  3. Use fan speed reducers on all the fans. The case has two 200mm fans, and the 140mm fan we're adding. I couldn't get the Asus motherboard's "QFan" fan control system to work, as it seems to require 4-pin fans, and all the ones I had (including the ones that came with the case) are 3-pin. While I do prefer dynamic, temperature based control when I can get it, the next best thing is to use hardware to slow down the fans. I like the Zalman-ZM-RC56 resistor connector as the simplest solution, but it's getting hard to find for some reason. The Zalman Fan Mate 2 will also work, and allows you to individually adjust the speed of each fan. The case also has a built in fan controller – that's the knob you see on the front top – but I found it too limited in range for my purposes.

  4. Add acoustic foam to taste. Between inexpensive eggcrate foam and thin, adhesive backed open-cell foam, you can easily reduce that last 10-20% of fan noise to a very pleasant white noise. It works well in the areas pictured, and also on the interior of the side panel "facing" the fans. See item 6 in my Building a Quiet PC post for details.

And then, of course, the overclockening. What kind of geek would I be if I didn't attempt to turn this baby up to 11? This is another area where Sandy Bridge is a revelation: so long as you buy one of the blessed "K" series processors, overclocking is as simple as setting the multiplier to the desired value. It is ridiculously simple. And my results, for once, were immediately as good as the ones everyone else was crowing about: 4.4 GHz totally stable!

Sandy-bridge-overclock

(beware: there is one nasty little issue with the Asus motherboard's auto-overclock feature. The PLL Overvoltage setting, which auto-overclock "helpfully" enables, completely bollixes up resuming from sleep. Just turn it off, and all is well. I don't even want to tell you how long it took me to figure that one out.)

The total package with a nice SSD delivers a near-perfect Windows Experience score:

Sandy-bridge-windows-experience-score

I won't lie to you. This is not a compact build. It's big! Those roomy side areas come at a cost, and that makes it a very wide case. But that's to be expected for a desktop powerhouse machine. And since my last case lasted me from 2007-2011, I'll happily accept a little bulk for something that's easy to work on and upgrade over time.

New-sandy-bridge-build-external

It's a fantastic new reboot of my system, and I didn't expect to be this excited about the final result. This is not merely an incremental improvement over what I had, it's much quieter, easier to work on, and when overclocked to 4.4 GHz, noticeably faster too. (I do slightly mourn the loss of 8 GB of RAM, but I'll survive.)

In this build, I already had hard drives, DVD drive, a sound card, and so forth … but for completeness' sake I'll list everything here if you want to mirror this setup. Realize that some of this comes down to personal taste, so I'm just listing what I recommend. Feel free to change anything out, and bear in mind that Sandy Bridge has decent default onboard video as well.

Remember, if you can put together a LEGO kit, you can build this totally sweet PC for yourself, too. Good luck and happy building!

Posted by Jeff Atwood    59 Comments

Performance is a Feature

June 20, 2011

We've always put a heavy emphasis on performance at Stack Overflow and Stack Exchange. Not just because we're performance wonks (guilty!), but because we think speed is a competitive advantage. There's plenty of experimental data proving that the slower your website loads and displays, the less people will use it.

[Google found that] the page with 10 results took 0.4 seconds to generate. The page with 30 results took 0.9 seconds. Half a second delay caused a 20% drop in traffic. Half a second delay killed user satisfaction.

In A/B tests, [Amazon] tried delaying the page in increments of 100 milliseconds and found that even very small delays would result in substantial and costly drops in revenue.

I believe the converse of this is also true. That is, the faster your website is, the more people will use it. This follows logically if you think like an information omnivore: the faster you can load the page, the faster you can tell whether that page contains what you want. Therefore, you should always favor fast websites. The opportunity cost for switching on the public internet is effectively nil, and whatever it is that you're looking for, there are multiple websites that offer a similar experience. So how do you distinguish yourself? You start by being, above all else, fast.

Do you, too, feel the need – the need for speed? If so, I have three pieces of advice that I'd like to share with you.

1. Follow the Yahoo Guidelines. Religiously.

The golden reference standard for building a fast website remains Yahoo's 13 Simple Rules for Speeding Up Your Web Site from 2007. There is one caveat, however:

There's some good advice here, but there's also a lot of advice that only makes sense if you run a website that gets millions of unique users per day. Do you run a website like that? If so, what are you doing reading this instead of flying your private jet to a Bermuda vacation with your trophy wife?

So … a funny thing happened to me since I wrote that four years ago. I now run a network of public, community driven Q&A web sites that do get millions of daily unique users. (I'm still waiting on the jet and trophy wife.) It does depend a little on the size of your site, but if you run a public website, you really should pore over Yahoo's checklist and take every line of it to heart. Or use the tools that do this for you:

We've long since implemented most of the 13 items on Yahoo's list, except for one. But it's a big one: Using a Content Delivery Network.

The user's proximity to your web server has an impact on response times. Deploying your content across multiple, geographically dispersed servers will make your pages load faster from the user's perspective. But where should you start?

As a first step to implementing geographically dispersed content, don't attempt to redesign your web application to work in a distributed architecture. Depending on the application, changing the architecture could include daunting tasks such as synchronizing session state and replicating database transactions across server locations. Attempts to reduce the distance between users and your content could be delayed by, or never pass, this application architecture step.

Remember that 80-90% of the end-user response time is spent downloading all the components in the page: images, stylesheets, scripts, Flash, etc. This is the Performance Golden Rule. Rather than starting with the difficult task of redesigning your application architecture, it's better to first disperse your static content. This not only achieves a bigger reduction in response times, but it's easier thanks to content delivery networks.

As a final optimization step, we just rolled out a CDN for all our static content. The results are promising; the baseline here is our datacenter in NYC, so the below should be read as "how much faster did our website get for users in this area of the world?"

Cdn-performance-test-world-map

In the interests of technical accuracy, static content isn't the complete performance picture; you still have to talk to our servers in NYC to get the dynamic content which is the meat of the page. But 90% of our visitors are anonymous, only 36% of our traffic is from the USA, and Yahoo's research shows that 40 to 60 percent of daily vistors come in with an empty browser cache. Optimizing this cold cache performance worldwide is a huge win.

Now, I would not recommend going directly for a CDN. I'd leave that until later, as there are a bunch of performance tweaks on Yahoo's list which are free and trivial to implement. But using a CDN has gotten a heck of a lot less expensive and much simpler since 2007, with lots more competition in the space from companies like Amazon's, NetDNA, and CacheFly. So when the time comes, and you've worked through the Yahoo list as religiously as I recommend, you'll be ready.

2. Love (and Optimize for) Your Anonymous and Registered Users

Our Q&A sites are all about making the internet better. That's why all the contributed content is licensed back to the community under Creative Commons and always visible regardless of whether you are logged in or not. I despise walled gardens. In fact, you don't actually have to log in at all to participate in Q&A with us. Not even a little!

The primary source of our traffic is anonymous users arriving from search engines and elsewhere. It's classic "write once, read – and hopefully edit – millions of times." But we are also making the site richer and more dynamic for our avid community members, who definitely are logged in. We add features all the time, which means we're serving up more JavaScript and HTML. There's an unavoidable tension here between the download footprint for users who are on the site every day, and users who may visit once a month or once a year.

Both classes are important, but have fundamentally different needs. Anonymous users are voracious consumers optimizing for rapid browsing, while our avid community members are the source of all the great content that drives the network. These guys (and gals) need each other, and they both deserve special treatment. We design and optimize for two classes of users: anonymous, and logged in. Consider the following Google Chrome network panel trace on a random Super User question I picked:

  requests data transferred DOMContentLoaded onload
Logged in (as me) 29 233.31 KB 1.17 s 1.31 s
Anonymous 22 111.40 KB 768 ms 1.28 s

We minimize the footprint of HTML, CSS and Javascript for anonymous users so they get their pages even faster. We load a stub of very basic functionality and dynamically "rez in" things like editing when the user focuses the answer input area. For logged in users, the footprint is necessarily larger, but we can also add features for our most avid community members at will without fear of harming the experience of the vast, silent majority of anonymous users.

3. Make Performance a Point of (Public) Pride

Now that we've exhausted the Yahoo performance guidance, and made sure we're serving the absolute minimum necessary to our anonymous users – where else can we go for performance? Back to our code, of course.

When it comes to website performance, there is no getting around one fundamental law of the universe: you can never serve a webpage faster than it you can render it on the server. I know, duh. But I'm telling you, it's very easy to fall into the trap of not noticing a few hundred milliseconds here and there over the course of a year or so of development, and then one day you turn around and your pages are taking almost a full freaking second to render on the server. It's a heck of a liability to start 1 full second in the hole before you've even transmitted your first byte over the wire!

That's why, as a developer, you need to put performance right in front of your face on every single page, all the time. That's exactly what we did with our MVC Mini Profiler, which we are contributing back to the world as open source. The simple act of putting a render time in the upper right hand corner of every page we serve forced us to fix all our performance regressions and omissions.

Mvc-mini-profiler-question-page

(Note that you can click on the SQL linked above to see what's actually being run and how long it took in each step. And you can use the share link to share the profiler data for this run with your fellow developers to shame them diagnose a particular problem. And it works for multiple AJAX requests. Have I mentioned that our open source MVC Mini Profiler is totally freaking awesome? If you're on a .NET stack, you should really check it out. )

In fact, with the render time appearing on every page for everyone on the dev team, performance became a point of pride. We had so many places where we had just gotten a little sloppy or missed some tiny thing that slowed a page down inordinately. Most of the performance fixes were trivial, and even the ones that were not turned into fantastic opportunities to rearchitect and make things simpler and faster for all of our users.

Did it work? You bet your sweet ILAsm it worked:

Google-webmaster-crawl-stats-download-time

That's the Google crawler page download time; the experimental Google Site Performance page, which ostensibly reflects complete full-page browser load time, confirms the improvements:

Google-webmaster-site-performance-overview

While server page render time is only part of the performance story, it is the baseline from which you start. I cannot emphasize enough how much the simple act of putting the page render time on the page helped us, as a development team, build a dramatically faster site. Our site was always relatively fast, but even for a historically "fast" site like ours, we realized huge gains in performance from this one simple change.

I won't lie to you. Performance isn't easy. It's been a long, hard road getting to where we are now – and we've thrown a lot of unicorn dollars toward really nice hardware to run everything on, though I wouldn't call any of our hardware choices particularly extravagant. And I did follow my own advice, for the record.

I distinctly remember switching from AltaVista to Google back in 2000 in no small part because it was blazing fast. To me, performance is a feature, and I simply like using fast websites more than slow websites, so naturally I'm going to build a site that I would want to use. But I think there's also a lesson to be learned here about the competitive landscape of the public internet, where there are two kinds of websites: the quick and the dead.

Which one will you be?

Posted by Jeff Atwood    40 Comments

Geek Transportation Systems

June 10, 2011

On my first visit to the Fog Creek Software offices in 2008, I was surprised to see programmers zooming around the office on scooters. I didn't realize that scooters were something geeks would be into, but it sure looked like fun, albeit borderline dangerous fun, on the 25th floor of an office building in Manhattan.

It turns out that having children is a great excuse reason to get into fun things like scooters. I didn't know much about scooters for adults, so being an obsessive geek, of course I had to research the heck out of this topic. My research turned up the Xootr MG as a top choice.

Xootr-mg-scooter

News flash: scooters are fun. Really fun!

But per my research (and now, personal experience) scooters are also surprisingly practical forms of transportation in certain situations, namely when …

  • you need to travel 1-3 miles
  • the route is not too hilly
  • it is not raining or wet
  • the route is mostly paved or has sidewalks
  • you are comfortable being "that awkward looking guy on a scooter"

Scooters are very primitive machines; it is both their greatest strength and their greatest weakness. It's arguably the simplest personal wheeled vehicle there is. In these short distance scenarios, scooters tend to win over, say, bicycles because there's less setup and teardown necessary – you don't have to lock up a scooter, nor do you have to wear a helmet (though I highly recommend one). Just hop on and go! You get almost all the benefits of gravity and wheeled efficiency with a minimum of fuss and maintenance. And yes, it's fun, too!

I'm just a scooter newbie, but the Xootr MG has a few characteristics I liked a lot, including rock-solid construction, a front brake (not super efficient, but reasonably effective when combined with the rear foot fender brake), and a wide, comfortable platform for your feet. But it does take some effort to kick around – don't forget to alternate your legs – and the ride can be rough at times depending on the surface. Large bumps and very uneven surfaces are wreck material. And going uphill on a scooter, beyond the absolute wussiest and mildest of grades, is simply out of the question.

For longer distances, or if the terrain is rougher or hillier, a scooter might work, but it'd be a tough way to travel. What you need in those cases is a small, portable bicycle – one you can take with you. I've dabbled in foldable bicycles before, and we own two Dahon folding bicycles. They're great, versatile and inexpensive bikes.

New-foldable-bikes

Dahon makes fine traditional folding bicycles, but they are not quite as pick-up-and-go as I would like for short trips. As an experiment, I purchased something I've had my eye on for a long time: the Strida LT folding bicycle. Or, as I like to call it, my "mid-life crisis vehicle".

Strida-green-side

(also pictured: some cool accessories that I recommend for Strida owners: a Cateye Reflex rear LED light on the rack, Knog beetle silicone front LED light on the handlebars, and a Sunlite Bicycle bungie cargo net.)

The appeal of the Strida is that it folds down to an incredibly small size.

Strida-green-folded

It's almost a pogo stick in folded form. I took my Strida on a short trip into San Francisco for a speaking gig in the city, which involved riding on BART, and the Strida in practice is everything I dreamed a modern ultra-portable folding bicycle could be:

  • front and rear disc brakes; superb stoppers
  • belt drive so no grease on your hands or pants
  • built in fenders in case you encounter puddles or rain
  • comfortable, full size(ish) upright riding position
  • super-easy, crazy fast folding: five seconds, no kidding!
  • when folded, the bike can be propped by the rear rack (as pictured) or strolled along by rolling it on its wheels.

The Strida may look odd, and perhaps it is odd, but I found it to be shockingly close to an ideal go-anywhere do-anything convenience bicycle. It isn't perfect, of course:

  • My only real beef with the Strida: the seat adjustment is horrendously kludgey. Adjusting the seat height on a Strida is painfully awkward even in the garage; on the go it's not an option.
  • It is a small wheel bicycle, with all the unavoidable physical compromises that entails. It'll always be a little twitchy and not something you would want to go on a 10 or 20 mile ride with.
  • It's a single speed, and you're not supposed to stand out of the saddle for power pedaling at any time. The frame and belt drive won't take it. On anything other than a moderate uphill you'll need to hop off and walk. (There is a slightly fancier Strida that has two internal hub gears, but I know nothing about it.)
  • Because the fold involves a ball joint, it is possible to permanently damage the bike if you aren't careful when you fold and force it. I doubt this is a real concern for anyone who has folded a Strida more than once, but if a ham-fisted friend tries to fold your Strida to "test it out", you might be in trouble.

None of these criticisms apply to the Dahon, so hopefully you can get a sense of the dividing line between an ultra-folder and a plain old folding bicycle.

Being a geek, it's not like I spend a lot of time outdoors. But when I do venture outside, I like to travel in a manner befitting a geek. That is, with my utility belt fully equipped, and in the dorkiest, most efficient vehicle possible for a trip of that particular length. Scooters, folding bicycles, recumbents, pogo sticks … whatever it takes. If you, too, would like to geek out around town, consider adding the Xootr MG scooter and Strida LT folding bicycle to your stable of geek transportation systems.

Posted by Jeff Atwood    52 Comments

Suspension, Ban or Hellban?

June 4, 2011

For almost eight months after launching Stack Overflow to the public, we had no concept of banning or blocking users. Like any new frontier town in the wilderness of the internet, I suppose it was inevitable that we'd be obliged to build a jail at some point. But first we had to come up with some form of government.

Stack Overflow was always intended to be a democracy. With the Stack Exchange Q&A network, we've come a long way towards that goal:

  • We create new communities through the open, democratic process defined at Area 51.
  • Our communities are maintained and operated by the most avid citizens within that community. The more reputation you have, the more privileges you earn.
  • We hold yearly moderator elections once each community is large enough to support them.

We strive mightily to build self organizing, self governing communities of people who are passionate about a topic, whether it be motor vehicles or homebrewing or musical instruments, or … whatever. Our general philosophy is power to the people.

Power-to-the-people

But in the absence of some system of law, the tiny minority of users out to do harm – intentionally or not – eventually drive out all the civil community members, leaving behind a lawless, chaotic badland.

Our method of dealing with disruptive or destructive community members is simple: their accounts are placed in timed suspension. Initial suspension periods range from 1 to 7 days, and increase exponentially with each subsequent suspension. We prefer the term "timed suspension" to "ban" to emphasize that we do want users to come back to their accounts, if they can learn to refrain from engaging in those disruptive or problematic behaviors. It's not so much a punishment as a time for the user to cool down and reflect on the nature of their participation in our community. (Well, at least in theory.)

Timed suspension works, but much like democracy itself, it is a highly imperfect, noisy system. The transparency provides ample evidence that moderators aren't secretly whisking people away in the middle of the night. But it can also be a bit too … entertaining for some members of the community, leading to hours and hours of meta-discussion about who is suspended, why they are suspended, whether it was fair, what the evidence is, how we are censoring people, and on and on and on. While a certain amount of introspection is important and necessary, it can also become a substitute for getting stuff done. This might naturally lead one to wonder – what if we could suspend problematic users without anyone knowing they had been suspended?

There are three primary forms of secretly suspending users that I know of:

  1. A hellbanned user is invisible to all other users, but crucially, not himself. From their perspective, they are participating normally in the community but nobody ever responds to them. They can no longer disrupt the community because they are effectively a ghost. It's a clever way of enforcing the "don't feed the troll" rule in the community. When nothing they post ever gets a response, a hellbanned user is likely to get bored or frustrated and leave. I believe it, too; if I learned anything from reading The Great Brain as a child, it's that the silent treatment is the cruelest punishment of them all.

    I've always associated hellbanning with the Something Awful Forums. Per this amazing MetaFilter discussion, it turns out the roots of hellbanning go much deeper – all the way back to an early Telnet BBS system called Citadel, where the "problem user bit" was introduced around 1986. Like so many other things in social software, it keeps getting reinvented over and over again by clueless software developers who believe they're the first programmer smart enough to figure out how people work. It's supported in most popular forum and blog software, as documented in the Drupal Cave module.

    (There is one additional form of hellbanning that I feel compelled to mention because it is particularly cruel – when hellbanned users can see only themselves and other hellbanned users. Brrr. I'm pretty sure Dante wrote a chapter about that, somewhere.)

  2. A slowbanned user has delays forcibly introduced into every page they visit. From their perspective, your site has just gotten terribly, horribly slow. And stays that way. They can hardly disrupt the community when they're struggling to get web pages to load. There's also science behind this one, because per research from Google and Amazon, every page load delay directly reduces participation. Get slow enough, for long enough, and a slowbanned user is likely to seek out greener and speedier pastures elsewhere on the internet.

  3. An errorbanned user has errors inserted at random into pages they visit. You might consider this a more severe extension of slowbanning – instead of pages loading slowly, they might not load at all, return cryptic HTTP errors, return the wrong page altogether, fail to load key dependencies like JavaScript and images and CSS, and so forth. I'm sure your devious little brains can imagine dozens of ways things could go "wrong" for an errorbanned user. This one is a bit more esoteric, but it isn't theoretical; an existing implementation exists in the form of the Drupal Misery module.

Because we try to hew so closely to the real world model of democracy with Stack Exchange, I'm not quite sure how I feel about these sorts of reality-altering tricks that are impossible in the world of atoms. On some level, they feel disingenuous to me. And it's a bit like wishing users into the cornfield with superhuman powers far beyond the ken of normal people. But I've also spent many painful hours trapped in public dialog about users who were, at best, just wasting everyone's time. Democracy is a wonderful thing, but efficient, it ain't.

That said, every community is different. I've personally talked to people in charge of large online communities – ones you probably participate in every day – and part of the reason those communities haven't broken down into utter chaos by now is because they secretly hellban and slowban their most problematic users. These solutions do neatly solve the problem of getting troublesome users to "voluntarily" decide to leave a community with a minimum of drama. It's hard to argue with techniques that are proven to work.

I think everyone has a right to know what sort of jail their community uses, even these secret, invisible ones. But keep in mind that whether it's timed suspensions, traditional bans, or exotic hellbans and beyond, the goal is the same: civil, sane, and safe online communities for everyone.

Posted by Jeff Atwood    73 Comments

The Infinite Version

May 23, 2011

One of the things I like most about Google's Chrome web browser is how often it is updated. But now that Chrome has rocketed through eleven versions in two and a half years, the thrill of seeing that version number increment has largely worn off. It seems they've picked off all the low hanging fruit at this point and are mostly polishing. The highlights from Version 11, the current release of Chrome?

HTML5 Speech Input API. Updated icon.

Exciting, eh? Though there was no shortage of hand-wringing over the new icon, of course.

Chrome's version number has been changing so rapidly lately that every time someone opens a Chrome bug on a Stack Exchange site, I have to check my version against theirs just to make sure we're still talking about the same software. And once -- I swear I am not making this up -- the version incremented while I was checking the version.

another nanosecond, another Chrome version.

That was the day I officially stopped caring what version Chrome is. I mean, I care in the sense that sometimes I need to check its dogtags in battle, but as a regular user of Chrome, I no longer think of myself as using a specific version of Chrome, I just … use Chrome. Whatever the latest version is, I have it automagically.

For the longest time, web browsers have been strongly associated with specific versions. The very mention of Internet Explorer 6 or Netscape 4.77 should send a shiver down the spine of any self-respecting geek. And for good reason! Who can forget what a breakout hit Firefox 3 was, or the epochs that Internet Explorer 7, 8 and 9 represent in Microsoft history. But Chrome? Chrome is so fluid that it has transcended software versioning altogether.

Chrome-infinite-version

This fluidity is difficult to achieve for client software that runs on millions of PCs, Macs, and other devices. Google put an extreme amount of engineering effort into making the Chrome auto-update process "just work". They've optimized the heck out of the update process.

Rather then push put a whole new 10MB update [for each version], we send out a diff that takes the previous version of Google Chrome and generates the new version. We tried several binary diff algorithms and have been using bsdiff up until now. We are big fans of bsdiff - it is small and worked better than anything else we tried.

But bsdiff was still producing diffs that were bigger than we felt were necessary. So we wrote a new diff algorithm that knows more about the kind of data we are pushing - large files containing compiled executables. Here are the sizes for the recent 190.1 -> 190.4 update on the developer channel:

  • Full update: 10 megabytes
  • bsdiff update: 704 kilobytes
  • Courgette update: 78 kilobytes

The small size in combination with Google Chrome's silent update means we can update as often as necessary to keep users safe.

Google's Courgette -- the French word for Zucchini, oddly enough -- is an amazing bit of software optimization, capable of producing uncannily small diffs of binary executables. To achieve this, it has to know intimate details about the source code:

The problem with compiled applications is that even a small source code change causes a disproportional number of byte level changes. When you add a few lines of code, for example, a range check to prevent a buffer overrun, all the subsequent code gets moved to make room for the new instructions. The compiled code is full of internal references where some instruction or datum contains the address (or offset) of another instruction or datum. It only takes a few source changes before almost all of these internal pointers have a different value, and there are a lot of them - roughly half a million in a program the size of chrome.dll.

The source code does not have this problem because all the entities in the source are symbolic. Functions don't get committed to a specific address until very late in the compilation process, during assembly or linking. If we could step backwards a little and make the internal pointers symbolic again, could we get smaller updates?

Since the version updates are relatively small, they can be downloaded in the background. But even Google hasn't figured out how to install an update while the browser is running. Yes, there are little alert icons to let you know your browser is out of date, and you eventually do get nagged if you are woefully behind, but updating always requires the browser to restart.

Please-restart-google-chrome

Web applications have it far easier, but they have version delivery problems, too. Consider WordPress, one of the largest and most popular webapps on the planet. We run WordPress on multiple blogs and even have our own WordPress community. WordPress doesn't auto-update to each new version, but it makes it as painless as I've seen for a webapp. Click the update link on the dashboard and WordPress (and its add-ons) update to the latest version all by themselves. There might be the briefest of interruptions in service for visitors to your WordPress site, but then you're back in business with the latest update.

Wordpress-update

WordPress needs everyone to update to the latest versions regularly for the same reasons Google Chrome does -- security, performance, and stability. An internet full of old, unpatched WordPress or Chrome installations is no less dangerous than an internet full of old, unpatched Windows XP machines.

These are both relatively seamless update processes. But they're nowhere near as seamless as they should be. One click updates that require notification and restart aren't good enough. To achieve the infinite version, we software engineers have to go a lot deeper.

Twitter-google-docs-infinite-version

Somehow, we have to be able to automatically update software while it is running without interrupting the user at all. Not if -- but when -- the infinite version arrives, our users probably won't even know. Or care. And that's how we'll know we've achieved our goal.

Posted by Jeff Atwood    68 Comments

Who Needs a Sound Card, Anyway?

May 4, 2011

The last sound card I purchased was in 2006, and that's only because I'm (occasionally) a bleeding edge PC gamer. The very same card was still in my current PC until a few days ago. It's perhaps too generous to describe PC sound hardware as stagnant; it's borderline irrelevant.

The default, built-in sound chips on most motherboards have evolved from "totally crap" to "surprisingly decent" in the last 5 years. But besides that, in this era of ubiquitous quad core CPUs nearing 4 GHz, it'd be difficult to make a plausible case that you need a discrete set of silicon to handle sound processing, even for the very fanciest of 3D sound algorithms and HRTFs.

That said, if you enjoy music even a little, I still strongly recommend investing in a quality set of headphones. As I wrote in 2005's Headphone Snobbery:

Am I really advocating spending two hundred dollars on a set of headphones? Yes. Yes I am. Now, you could spend a lot more. This is about extracting the maximum bang for your buck:

  1. Unlike your computer, or your car, your headphones will never wear out or become obsolete. I hesitate to say lifetime, but they're multiple decade investments at the very least.
  2. The number one item that affects the music you hear is the speakers. Without a good set of headphones, everything else is irrelevant.
  3. The right headphones can deliver sound equivalent to extremely high-end floorstanding speakers worth thousands of dollars.

If you're the type of person who is perfectly happy listening to 64 kilobit MP3s through a $5 set of beige headphones, that's fine. There's nothing wrong with that. Keep on scrolling; this post is not for you.

I realize that there's a fine line between audiophile and bats**t insane -- and that line better not be near any sources of interference! But nice headphones require powerful, reasonably clean output to deliver the best listening experience. This isn't high end audio crackpot snake oil, it's actual physics.

I'll let the guys at headroom explain:

You may have heard of a headphone's "impedance." Impedance is the combined resistance and reactivity the headphones present to the amplifier as an electrical load. High impedance cans will usually need more voltage to get up to a solid listening level, so they will often benefit from an amp, especially with portable players that have limited voltage available from their internal batteries. But low impedance cans may require more current, and will lower the damping factor between the amp and headphones. So while low impedance headphones may be driven loud enough from a portable player, the quality of sound may be dramatically improved with an amp.

The size of your headphone will give you some clues to whether an amp may be warranted. Most earbud and in ear headphones are typically very efficient and are less likely to benefit strongly from an amp. Many larger headphones will benefit, or even require, a headphone amp to reach listenable volume levels with portable players.

Thus, once you have a set of nice headphones, you do need some kind of amplified output for them. Something like the Boostaroo, or a Total BitHead. And if you're on a laptop these outboard solutions might be your only options.

Total-bithead

But desktops offer the option of adding a sound card. The good news is that arguably the best sound card on the planet, the Xonar DG, is all of 30 measly bucks. It's a big step up in fundamental sound quality from even the best current integrated HD audio motherboard sound chips, per this Tech Report review.

RightMark Audio Analyzer audio quality, 16-bit/44.1kHz
freq response noise level range THD THD + Noise IMD + Noise crosstalk IMD at 10kHz overall
Realtek ALC892 HD 5 4 4 3 1 3 5 3 4
Xonar DG 5 6 6 5 4 6 6 6 5

It also includes a little something extra of particular interest to us music loving programmers with nice headphones:

Built-in headphone amplification is something you won't find on a motherboard, but it's featured in both Xonars. On the DG, Asus has gone with Texas Instruments' DRV601RTJR, which is optimized for headphone impedances of 32-150 Ω according to the card's spec sheet. The Xense gets something considerably fancier: a TI amp capable of pushing headphones with impedances up to 600 Ω. Of course, the headphones bundled with the card are rated for an impedance of only 150 Ω. Mid-range stereo cans like Sennheiser's excellent HD 555s, which we use for listening tests, have a rated impedance of just 50 Ω. You don't need big numbers for high-quality sound.

The headphone amplification options are a bit buried in the Xonar driver user interface. To get there, select headphone mode, then click the little hammer icon to bring up the headphone amp gain settings.

Xonar-dg-audio-center-headphones

After my last upgrade, I was truly hoping I could get away with just the on-board Realtek HD audio my motherboard provides. I resisted mightily -- but the drop in headphone output quality with the onboard stuff was noticeable. Not to mention that I had to absolutely crank the volume to get even moderate loudness with my fancy-ish Sennheiser HD 600 headphones. The Xonar DG neatly solves both of these problems.

As you probably expected, the answer to the question "Who needs a sound card?" is "Almost nobody." Except those of us who invested in quality headphones. Rather than spending $30 or $150 on an outboard headphone amp, spend $30 on the Xonar DG to get a substantial sound quality upgrade and a respectable headphone amp to boot.

Posted by Jeff Atwood    59 Comments

The Hot/Crazy Solid State Drive Scale

May 2, 2011

As an early advocate of solid state hard drives …

… I feel ethically and morally obligated to let you in on a dirty little secret I've discovered in the last two years of full time SSD ownership. Solid state hard drives fail. A lot. And not just any fail. I'm talking about catastrophic, oh-my-God-what-just-happened-to-all-my-data instant gigafail. It's not pretty.

I bought a set of three Crucial 128 GB SSDs in October 2009 for the original two members of the Stack Overflow team plus myself. As of last month, two out of three of those had failed. And just the other day I was chatting with Joel on the podcast (yep, it's back), and he casually mentioned to me that the Intel SSD in his Thinkpad, which was purchased roughly around the same time as ours, had also failed.

Portman Wills, friend of the company and generally awesome guy, has a far scarier tale to tell. He got infected with the SSD religion based on my original 2009 blog post, and he went all in. He purchased eight SSDs over the last two years … and all of them failed. The tale of the tape is frankly a little terrifying:

  • Super Talent 32 GB SSD, failed after 137 days
  • OCZ Vertex 1 250 GB SSD, failed after 512 days
  • G.Skill 64 GB SSD, failed after 251 days
  • G.Skill 64 GB SSD, failed after 276 days
  • Crucial 64 GB SSD, failed after 350 days
  • OCZ Agility 60 GB SSD, failed after 72 days
  • Intel X25-M 80 GB SSD, failed after 15 days
  • Intel X25-M 80 GB SSD, failed after 206 days

You might think after this I'd be swearing off SSDs as unstable, unreliable technology. Particularly since I am the world's foremost expert on backups.

Well, you'd be wrong. I just went out and bought myself a hot new OCZ Vertex 3 SSD, the clear winner of the latest generation of SSDs to arrive this year. Storage Review calls it the fastest SATA SSD we've seen.

Beta firmware or not though, the Vertex 3 is a scorcher. We'll get into the details later in the review, but our numbers show it as clearly the fastest SATA SSD to hit our bench.

ocz-vertex-3

While that shouldn't be entirely surprising, it's not just faster like, "Woo, it edged out the prior generation SF-1200 SSDs, yeah!" It's faster like, "Holy @&#% that's fast," boasting 69% faster results in some of our real-world tests.

Solid state hard drives are so freaking amazing performance wise, and the experience you will have with them is so transformative, that I don't even care if they fail every 12 months on average! I can't imagine using a computer without a SSD any more; it'd be like going back to dial-up internet or 13" CRTs or single button mice. Over my dead body, man!

It may seem irrational, but … well, I believe the phenomenon was explained best on the television show How I Met Your Mother by Barney Stinson, a character played brilliantly by geek favorite Neil Patrick Harris:

Barney: There's no way she's above the line on the 'hot/crazy' scale.

Ted: She's not even on the 'hot/crazy' scale; she's just hot.

Robin: Wait, 'hot/crazy' scale?

Barney: Let me illustrate!

The-hot-crazy-scale1

Barney: A girl is allowed to be crazy as long as she is equally hot. Thus, if she's this crazy, she has to be this hot. You want the girl to be above this line. Also known as the 'Vickie Mendoza Diagonal'. This girl I dated. She played jump rope with that line. She'd shave her head, then lose 10 pounds. She'd stab me with a fork, then get a boob job. [pause] I should give her a call.

Thing is, SSDs are so scorching hot that I'm willing to put up with their craziness. Consider that just in the last two years, their performance has doubled. Doubled! And the latest, fastest SSDs can even saturate existing SATA interfaces; they need brand new 6 Gbps interfaces to fully strut their stuff. No CPU or memory upgrade can come close to touching that kind of real world performance increase.

Just make sure you have a good backup plan if you're running on a SSD. I do hope they iron out the reliability kinks in the next 2 generations … but I've spent the last two months checking out the hot/crazy solid state drive scale in excruciating detail, and trust me, you want one of these new Vertex 3 SSDs right now.

Posted by Jeff Atwood    126 Comments

Working with the Chaos Monkey

April 25, 2011

Late last year, the Netflix Tech Blog wrote about five lessons they learned moving to Amazon Web Services. AWS is, of course, the preeminent provider of so-called "cloud computing", so this can essentially be read as key advice for any website considering a move to the cloud. And it's great advice, too. Here's the one bit that struck me as most essential:

We’ve sometimes referred to the Netflix software architecture in AWS as our Rambo Architecture. Each system has to be able to succeed, no matter what, even all on its own. We’re designing each distributed system to expect and tolerate failure from other systems on which it depends.

If our recommendations system is down, we degrade the quality of our responses to our customers, but we still respond. We’ll show popular titles instead of personalized picks. If our search system is intolerably slow, streaming should still work perfectly fine.

One of the first systems our engineers built in AWS is called the Chaos Monkey. The Chaos Monkey’s job is to randomly kill instances and services within our architecture. If we aren’t constantly testing our ability to succeed despite failure, then it isn’t likely to work when it matters most – in the event of an unexpected outage.

Which, let's face it, seems like insane advice at first glance. I'm not sure many companies even understand why this would be a good idea, much less have the guts to attempt it. Raise your hand if where you work, someone deployed a daemon or service that randomly kills servers and processes in your server farm.

Now raise your other hand if that person is still employed by your company.

Who in their right mind would willingly choose to work with a Chaos Monkey?

Angry-monkey-family-guy

Sometimes you don't get a choice; the Chaos Monkey chooses you. At Stack Exchange, we struggled for months with a bizarre problem. Every few days, one of the servers in the Oregon web farm would simply stop responding to all external network requests. No reason, no rationale, and no recovery except for a slow, excruciating shutdown sequence requiring the server to bluescreen before it would reboot.

We spent months -- literally months -- chasing this problem down. We walked the list of everything we could think of to solve it, and then some:

  • swapping network ports
  • replacing network cables
  • a different switch
  • multiple versions of the network driver
  • tweaking OS and driver level network settings
  • simplifying our network configuration and removing TProxy for more traditional X-FORWARDED-FOR
  • switching virtualization providers
  • changing our TCP/IP host model
  • getting Kernel hotfixes and applying them
  • involving high-level vendor support teams
  • some other stuff that I've now forgotten because I blacked out from the pain

At one point in this saga our team almost came to blows because we were so frustrated. (Well, as close to "blows" as a remote team can get over Skype, but you know what I mean.) Can you blame us? Every few days, one of our servers -- no telling which one -- would randomly wink off the network. The Chaos Monkey strikes again!

Even in our time of greatest frustration, I realized that there was a positive side to all this:

  • Where we had one server performing an essential function, we switched to two.
  • If we didn't have a sensible fallback for something, we created one.
  • We removed dependencies all over the place, paring down to the absolute minimum we required to run.
  • We implemented workarounds to stay running at all times, even when services we previously considered essential were suddenly no longer available.

Every week that went by, we made our system a tiny bit more redundant, because we had to. Despite the ongoing pain, it became clear that Chaos Monkey was actually doing us a big favor by forcing us to become extremely resilient. Not tomorrow, not someday, not at some indeterminate "we'll get to it eventually" point in the future, but right now where it hurts.

Now, none of this is new news; our problem is long since solved, and the Netflix Tech Blog article I'm referring to was posted last year. I've been meaning to write about it, but I've been a little busy. Maybe the timing is prophetic; AWS had a huge multi-day outage last week, which took several major websites down, along with a constellation of smaller sites.

Notably absent from that list of affected AWS sites? Netflix.

When you work with the Chaos Monkey, you quickly learn that everything happens for a reason. Except for those things which happen completely randomly. And that's why, even though it sounds crazy, the best way to avoid failure is to fail constantly.

(update: Netflix released their version of Chaos Monkey on GitHub. Try it out!)

Posted by Jeff Atwood    49 Comments

Revisiting the Home Theater PC

March 28, 2011

It's been almost three years since I built my home theater PC. I adore that little machine; it drives all of our family entertainment and serves as a general purpose home media server and streaming box. As I get older, I find that I'm no longer interested in having a home full of PCs whirring away. I only want one PC in my house on all the time, and I want it to be as efficient and versatile as possible.

My old low-power Athlon X2 based HTPC generally worked great, but still struggled with some occasional 1080p content. And when you have a toddler in the house, believe me, you need reliable 1080p playback. Only the finest in children's entertainment for my spawned process, I say!

When I recently had to transcode Megamind down to 720p to get it to play back without stuttering or pausing at times… I knew my current HTPC's days were numbered.

Megamind-evil-overlord-small

(Megamind is hilarious and highly recommended, by the way; it's far better than its Metacritic and Rotten Tomatoes percentages would seem to indicate.)

Now that Intel has finally released their Sandy Bridge CPUs -- the first with integrated GPUs -- I was eager to revisit and rebuild. The low power Core i3-2100T is the one I had my eye on, with a miserly TDP of 35 watts. Combine that with a decent Mini-ITX motherboard and a few other essential parts, and you're good to go:

CPUIntel Core i3-2100T$135
MotherboardASRock H67M ITX$100
RAMCorsair 4GB DDR3$45
Case + PSUAntec ISK 300-65$70
HDD750GB 2.5"$70

Now, I am fudging a bit here. This is just the basic level of hardware to get a functional home theater PC. I didn't actually buy a case, PSU, or even hard drive for that matter; I recycled many of my old existing parts, so my personal outlay was all of 300 bucks. I'm including the fuller part list as courtesy recommendations in case you're starting from scratch. You also might want to add a Blu-Ray drive, and perhaps a Windows 7 Home Premium license ($99) for its excellent 10-foot Windows Media Center interface.

Asrock-mini-itx-h67-motherboard

The magical part here is the extreme level of hardware integration: the CPU has a GPU and memory controller on die, and the motherboard has optical digital out and HDMI out built in. It's delightfully simple to build and downright cheap. Just assemble it, install your OS of choice (sorry, Apple fans), then plug it into your receiver and television and boot it up.

My results? I'll just get right to the good part, but please bear in mind each step is about twice as powerful as the one before:

2005~$1000512 MB RAM, single core CPU80 watts idle
2008~$5202 GB RAM, dual core CPU45 watts idle
2011~$4204 GB RAM, dual core CPU + GPU22 watts idle

I know I get way too excited about this stuff, but … holy crap, 22 tesla-lovin' watts at idle!

Kill-a-watt-2500t

The kill-a-watt never lies. To be fair, it's more like 25 watts idle with torrents in the background. This little box is remarkably efficient; even when playing back a 1080p video it's not unusual to see CPU usage well under 50%, which equates to around 30-35 watts in practice. Under full, artificial multithreaded Prime95 load, it tops out at an absolute peak of 55 watts.

(Update: I ended up replacing my old Seasonic ECO 300 SFX power supply with a Pico PSU-90 plus 60 watt adapter kit. That got the idle power down from 22 watts to 17 watts, a solid savings of 22%. Recommended!)

This is a killer setup, but don't take my word for it. There is an excruciatingly in-depth review of essentially the same system at Missing Remote, with a particular eye toward home theater duties. Spoiler: they loved the hell out of it too. And it compromises almost nothing in performance, with a Windows Experience score of 5.1 -- that would be a solid 5.8 if you factored out desktop Aero performance.

Windows-experience-score

(Also, in case you're wondering, I intentionally dropped the analog cable tuner. All modern cable is now digital, which means awkward DRM-ed up the wazoo CableCard systems. I've cancelled cable altogether; I'd rather take that $60+ per month and use it to support innovative companies who will deliver media through the internet, like Netflix, Hulu, etcetera. Or as I like to call it: the future, unless the media congolomerates with vaults full of cash manage to subvert net neutrality.)

When all is said and done, I have a new always-on, does-anything home theater box that is twice as fast as the one I built in 2008, while consuming less than half the power.

I've been a computer nerd since age 8, and I just turned 40. I should be jaded by computer hardware pornography by now, but I still find this progress amazing. At this rate, I can't wait to find out what my 2014 home theater PC will look like.

Posted by Jeff Atwood    72 Comments

The Importance of Net Neutrality

February 14, 2011

Although I remain a huge admirer of Lawrence Lessig, I am ashamed to admit that I never fully understood the importance of net neutrality until last week. Mr. Lessig described network neutrality in these urgent terms in 2006:

At the center of the debate is the most important public policy you've probably never heard of: "network neutrality." Net neutrality means simply that all like Internet content must be treated alike and move at the same speed over the network. The owners of the Internet's wires cannot discriminate. This is the simple but brilliant "end-to-end" design of the Internet that has made it such a powerful force for economic and social good: All of the intelligence and control is held by producers and users, not the networks that connect them.

Fortunately, the good guys are winning. Recent legal challenges to network neutrality have been defeated, at least under US law. I remember hearing about these legal decisions at the time, but I glossed over them because I thought they were fundamentally about file sharing and BitTorrent. Not to sound dismissive, but someone's legal right to download a complete video archive of Firefly wasn't exactly keeping me up at night.

But network neutrality is about far more than file sharing bandwidth. To understand what's at stake, study the sordid history of the world's communication networks – starting with the telegraph, radio, telephone, television, and onward. Without historical context, it's impossible to appreciate how scarily easy it is for common carriage to get subverted and undermined by corporations and government in subtle (and sometimes not so subtle) ways, with terrible long-term consequences for society.

That's the genius of Tim Wu's book The Master Switch: The Rise and Fall of Information Empires.

The-master-switch-cover

One of the most fascinating stories in the book is that of Harry Tuttle and AT&T.

Harry Tuttle was, for most of his life, president of the Hush-a-Phone Corporation, manufacturer of the telephone silencer. Apart from Tuttle, Hush-a-Phone employed his secretary. The two of them worked alone out of a small office near Union Square in New York City. Hush-a-Phone's signature product was shaped like a scoop, and it fit around the speaking end of a receiver, so that no one could hear what the user was saying on the telephone. The company motto emblazoned on its letterhead stated the promise succinctly: "Makes your phone private as a booth."

If the Hush-a-Phone never became a household necessity, Tuttle did a decent business, and by 1950 he would claim to have sold 125,000 units. But one day late in the 1940s, Henry Tuttle received alarming news. AT&T had launched a crackdown on the Hush-a-Phone and similar products, like the Jordaphone, a creaky precursor of the modern speakerphone, whose manufacturer had likewise been put on notice. Bell repairmen began warning customers that Hush-a-Phone use was a violation of a federal tariff and that, failing to cease and desist, they risked termination of their telephone service.

Was AT&T merely blowing smoke? Not at all: the company was referring to a special rule that was part of their covenant with the federal government. It stated: No equipment, apparatus, circuit or device not furnished by the telephone company shall be attached to or connected with the facilities furnished by the telephone company, whether physically, by induction, or otherwise.

Tuttle hired an attorney, who petitioned the FCC for a modification of the rule and an injunction against AT&T's threats. In 1950 the FCC decided to hold a trial (officially a "public hearing") in Washington, D.C., to consider whether AT&T, the nation's regulated monopolist, could punish its customers for placing a plastic cup over their telephone mouthpiece.

The story of the Hush-a-Phone and its struggle with AT&T, for all its absurdist undertones, offers a window on the mindset of the monopoly at its height, as well as a picture of the challenges facing even the least innovative innovator at that moment.

Absurdist, indeed – Harry Tuttle is also not-so-coincidentally the name of a character in the movie Brazil, one who attempts to work as a renegade, outside oppressive centralized government systems. Often at great peril to his own life and, well, that of anyone who happens to be nearby, too.

Harry-tuttle-brazil

But the story of Harry Tuttle isn't just a cautionary tale about the dangers of large communication monopolies. Guess who was on Harry Tuttle's side in his sadly doomed legal effort against the enormously powerful Bell monopoly? No less than an acoustics professor by the name of Leo Beranek, and an expert witness by the name of J.C.R. Licklider.

If you don't recognize those names, you should. J.C.R. Licklider went on to propose and design ARPANET, and Leo Beranek became one of the B's in Bolt, Beranek and Newman, who helped build ARPANET. In other words, these gentlemen went on from battling the Bell monopoly in court in the 1950s to designing a system in 1968 that would ultimately defeat it: the internet.

The internet is radically unlike all the telecommunications networks that have preceded it. It's the first national and global communication network designed from the outset to resist mechanisms for centralized control and monopoly. But resistance is not necessarily enough; The Master Switch makes a compelling case that, historically speaking, all communication networks start out open and then rapidly swing closed as they are increasingly commercialized.

Just as our addiction to the benefits of the internal combustion engine led us to such demand for fossil fuels as we could no longer support, so, too, has our dependence on our mobile smart phones, touchpads, laptops, and other devices delivered us to a moment when our demand for bandwidth – the new black gold – is insatiable. Let us, then, not fail to protect ourselves from the will of those who might seek domination of those resources we cannot do without. If we do not take this moment to secure our sovereignty over the choices that our information age has allowed us to enjoy, we cannot reasonably blame its loss on those who are free to enrich themselves by taking it from us in a manner history has foretold.

It's up to us to be vigilant in protecting the concepts of common carriage and network neutrality on the internet. Even devices that you may love, like an iPad, Kindle, or Xbox, can easily be turned against you – if you let them.

Posted by Jeff Atwood    124 Comments

How to Write Without Writing

February 4, 2011

I have a confession to make: in a way, I founded Stack Overflow to trick my fellow programmers.

Before you trot out the pitchforks and torches, let me explain.

Over the last 6 years, I've come to believe deeply in the idea that that becoming a great programmer has very little to do with programming. Yes, it takes a modicum of technical skill and dogged persistence, absolutely. But even more than that, it takes serious communication skills:

The difference between a tolerable programmer and a great programmer is not how many programming languages they know, and it's not whether they prefer Python or Java. It's whether they can communicate their ideas. By persuading other people, they get leverage. By writing clear comments and technical specs, they let other programmers understand their code, which means other programmers can use and work with their code instead of rewriting it. Absent this, their code is worthless.

That is of course a quote from my co-founder Joel Spolsky, and it's one of my favorites.

In defense of my fellow programmers, communication with other human beings is not exactly what we signed up for. We didn't launch our careers in software development because we loved chatting with folks. Communication is just plain hard, particularly written communication. How exactly do you get better at something you self-selected out of? Blogging is one way:

People spend their entire lives learning how to write effectively. It isn't something you can fake. It isn't something you can buy. You have to work at it.

That's exactly why people who are afraid they can't write should be blogging.

It's exercise. No matter how out of shape you are, if you exercise a few times a week, you're bound to get fitter. Write a small blog entry a few times every week and you're bound to become a better writer. If you're not writing because you're intimidated by writing, well, you're likely to stay that way forever.

Even with the best of intentions, telling someone "you should blog!" never works. I know this from painful first hand experience. Blogging isn't for everyone. Even a small blog entry can seem like an insurmountable, impenetrable, arbitrary chunk of writing to the average programmer. How do I get my fellow programmers to blog without blogging, to write without writing?

By cheating like hell, that's how.

Consider this letter I received:

I'm not sure if you have thought about this side effect or not, but Stack Overflow has taught me more about writing effectively than any class I've taken, book I've read, or any other experience I have had before.

I can think of no other medium where I can test my writing chops (by writing an answer), get immediate feedback on its quality (particularly when writing quality trumps technical correctness, such as subjective questions) and see other peoples attempts as well and how they compare with mine. Votes don't lie and it gives me a good indicator of how well an email I might send out to future co-workers would be received or a business proposal I might write.

Over the course of the past 5 months all the answers I've been writing have been more and more refined in terms of the quality. If I don't end up as the top answer I look at the answer that did and study what they did differently and where I faltered. Was I too verbose or was I too terse? Was I missing the crux of the question or did I hit it dead on?

I know that you said that writing your Coding Horror blog helped you greatly in refining your writing over the years. Stack Overflow has been doing the same for me and I just wanted to thank you for the opportunity. I've decided to setup a coding blog in your footsteps and I just registered a domain today. Hopefully that will go as well as writing on SO has. There are no tougher critics than fellow programmers who scrutinize every detail, every technical remark and grammar structure looking for mistakes. If you can effectively write for and be accepted by a group of programmers you can write for anyone.

Joel and I have always positioned Stack Overflow, and all the other Stack Exchange Q&A sites, as lightweight, focused, "fun size" units of writing.

Yes, by God, we will trick you into becoming a better writer if that's what it takes – and it always does. Stack Overflow has many overtly gamelike elements, but it is a game in service of the greater good – to make the internet better, and more importantly, to make you better. Seeing my fellow programmers naturally improve their written communication skills while participating in a focused, expert Q&A community with their peers? Nothing makes me prouder.

Beyond programming, there's a whole other community of peers out there who grok how important writing is, and will support you in sharpening your saw, er, pen. We have our own, too.

Writers Stack Exchange

If you're an author, editor, reviewer, blogger, copywriter, or aspiring writer of any kind, professional or otherwise – check out writers.stackexchange.com. Becoming a more effective writer is the one bedrock skill that will further your professional career, no matter what you choose to do.

But mostly, you should write. I thought Jon Skeet summed it up particularly well here:

Everyone should write a lot – whether it's a blog, a book, Stack Overflow answers, emails or whatever. Write, and take some care over it. Clarifying your communication helps you to clarify your own internal thought processes, in my experience. It's amazing how much you find you don't know when you try to explain something in detail to someone else. It can start a whole new process of discovery.

The process of writing is indeed a journey of discovery, one that will last the rest of your life. It doesn't ultimately matter whether you're writing a novel, a printer review, a Stack Overflow answer, fan fiction, a blog entry, a comment, a technical whitepaper, some emo LiveJournal entry, or even meta-talk about writing itself. Just get out there and write!

Posted by Jeff Atwood    49 Comments

Lived Fast, Died Young, Left a Tired Corpse

January 31, 2011

It's easy to forget just how crazy things got during the Web 1.0 bubble in 2000. That was over ten years ago. For context, Mark Zuckerberg was all of sixteen when the original web bubble popped.

Web-bubble-graph-nasdaq

There's plenty of evidence that we're entering another tech bubble. It's just less visible to people outside the tech industry because there's no corresponding stock market IPO frenzy. Yet.

There are two films which captured the hyperbole and excess of the original dot com bubble especially well.

Startup-com-dvd-movie

The first is the documentary Startup.com. It's about the prototypical web 1.0 company: one predicated on an idea that made absolutely no sense, which proceeded to flame out in a spectacular and all too typical way for the era. This one just happened to occur on digital film. The govworks.com website described in the documentary, the one that burned through $60 million in 18 months, is now one of those ubiquitous domain squatter pages. A sign of the times, perhaps.

The second film was one I had always wanted to see, but wasn't able to until a few days ago: Code Rush. For a very long time, Code Rush was almost impossible to find, but the activism of Andy Baio nudged the director to make the film available under Creative Commons. You can now watch it online — and you absolutely should.

Remember when people charged money for a web browser? That was Netscape.

Code Rush is a PBS documentary recorded at Netscape from 1998 - 1999, focusing on the open sourcing of the Netscape code. As the documentary makes painfully clear, this wasn't an act of strategy so much as an act of desperation. That's what happens when the company behind the world's most ubiquitous operating system decides a web browser should be a standard part of the operating system.

Everyone in the documentary knows they're doomed; in fact, the phrase "we're doomed" is a common refrain throughout the film. But despite the gallows humor and the dark tone, parts of it are oddly inspiring. These are engineers who are working heroic, impossible schedules for a goal they're not sure they can achieve — or that they'll even survive as an organization long enough to even finish.

The most vivid difference between Startup.com and Code Rush is that Netscape, despite all their other mistakes and missteps, didn't just burn through millions of dollars for no discernable reason. They produced a meaningful legacy:

  • Through Netscape Navigator, the original popularization of HTML and the internet itself.
  • With the release of the Netscape source code on March 31st, 1998, the unlikely birth of the commercial open source movement.
  • Eventually producing the first credible threat to Internet Explorer in the form of Mozilla Firefox 1.0 in 2004.

Do you want money? Fame? Job security? Or do you want to change the world … eventually? Consider how many legendary hackers went on to brilliant careers from Netscape: Jamie Zawinski, Brendan Eich, Stuart Parmenter, Marc Andreeseen. The lessons of Netscape live on, even though the company doesn't. Code Rush is ultimately a meditation on the meaning of work as a programmer.

I'd like to think that when Facebook – the next Google and Microsoft rolled into one – goes public in early 2012, the markets will react rationally. More likely, people will all collectively lose their damn minds again and we'll be thrust into a newer, bigger, even more insane tech bubble than the first one.

Yes, you will have incredibly lucrative job offers in this bubble. That's the easy part. As Startup.com and Code Rush illustrate, the hard part is figuring out why you are working all those long hours. Consider carefully, lest the arc of your career mirror that of so many failed tech bubble companies: lived fast, died young, left a tired corpse.

Posted by Jeff Atwood    31 Comments

24 Gigabytes of Memory Ought to be Enough for Anybody

January 20, 2011

Are you familiar with this quote?

640K [of computer memory] ought to be enough for anybody. — Bill Gates

It's amusing, but Bill Gates never actually said that:

I've said some stupid things and some wrong things, but not that. No one involved in computers would ever say that a certain amount of memory is enough for all time … I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again.

One of the few killer features of the otherwise unexciting Intel Core i7 platform upgrade* is the subtle fact that Core i7 chips use triple channel memory. That means three memory slots at a minimum, and in practice most Core i7 motherboards have six memory slots.

The price of DDR3 ram has declined to the point that populating all six slots of memory with 4 GB memory is, well, not cheap -- but quite attainable at $299 and declining.

24 gigabytes of DDR3 RAM

Twenty-four gigabytes of system memory for a mere $299! That's about $12.50 per gigabyte.

(And if you don't have a Core i7 system, they're not expensive to build, either. You can pair an inexpensive motherboard with even the slowest and cheapest triple channel compatible i7-950, which is plenty speedy – and overclocks well, if you're into that. Throw in the 24 GB of ram, and it all adds up to about $800 total. Don't forget the power supply and CPU cooler, though.)

Remember when one gigabyte of system memory was considered a lot? For context, our first "real" Stack Overflow database server had 24 GB of memory. Now I have that much in my desktop … just because I can. Well, that's not entirely true, as we do work with some sizable databases while building the Stack Exchange network.

24-gigabytes-ram-in-use

I guess having 24 gigabytes of system memory is a little extravagant, but at these prices -- why not? What's the harm in having obscene amounts of memory, making my system effectively future proof?

I have to say that in 1981, making those decisions, I felt like I was providing enough freedom for 10 years. That is, a move from 64k to 640k felt like something that would last a great deal of time. Well, it didn't – it took about only 6 years before people started to see that as a real problem. — Bill Gates

To me, it's more about no longer needing to think about memory as a scarce resource, something you allocate carefully and manage with great care. There's just .. lots. As Clay Shirky once related to me, via one of his college computer science professors:

Algorithms are for people who don't know how to buy RAM.

I mean, 24 GB of memory should be enough for anybody… right?

* it's only blah on the desktop; on the server the Nehalem architecture is indeed a monster and anyone running a server should upgrade to it, stat.

Posted by Jeff Atwood    85 Comments

Trouble In the House of Google

January 3, 2011

Let's look at where stackoverflow.com traffic came from for the year of 2010.

Stack-overflow-2010-traffic-by-source

When 88.2% of all traffic for your website comes from a single source, criticizing that single source feels … risky. And perhaps a bit churlish, like looking a gift horse in the mouth, or saying something derogatory in public about your Valued Business Partnertm.

Still, looking at the statistics, it's hard to avoid the obvious conclusion. I've been told many times that Google isn't a monopoly, but they apparently play one on the internet. You are perfectly free to switch to whichever non-viable alternative web search engine you want at any time. Just breathe in that sweet freedom, folks.

Sarcasm aside, I greatly admire Google. My goal is not to be acquired, because I'm in this thing for the long haul – but if I had to pick a company to be acquired by, it would probably be Google. I feel their emphasis on the information graph over the social graph aligns more closely with our mission than almost any other potential suitor I can think of. Anyway, we've been perfectly happy with Google as our de-facto traffic sugar daddy since the beginning. But last year, something strange happened: the content syndicators began to regularly outrank us in Google for our own content.

Syndicating our content is not a problem. In fact, it's encouraged. It would be deeply unfair of us to assert ownership over the content so generously contributed to our sites and create an underclass of digital sharecroppers. Anything posted to Stack Overflow, or any Stack Exchange Network site for that matter, is licensed back to the community in perpetuity under Creative Commons cc-by-sa. The community owns their contributions. We want the whole world to teach each other and learn from the questions and answers posted on our sites. Remix, reuse, share – and teach your peers! That's our mission. That's why I get up in the morning.

Jeff Atwood: Teaching peers is one of the best ways to develop mastery

However, implicit in this strategy was the assumption that we, as the canonical source for the original questions and answers, would always rank first. Consider Wikipedia – when was the last time you clicked through to a page that was nothing more than a legally copied, properly attributed Wikipedia entry encrusted in advertisements? Never, right? But it is in theory a completely valid, albeit dumb, business model. That's why Joel Spolsky and I were confident in sharing content back to the community with almost no reservations – because Google mercilessly penalizes sites that attempt to game the system by unfairly profiting on copied content. Remixing and reusing is fine, but mass-producing cheap copies encrusted with ads … isn't.

I think of this as common sense, but it's also spelled out explicitly in Google's webmaster content guidelines.

However, some webmasters attempt to improve their page's ranking and attract visitors by creating pages with many words but little or no authentic content. Google will take action against domains that try to rank more highly by just showing scraped or other auto-generated pages that don't add any value to users. Examples include:

Scraped content. Some webmasters make use of content taken from other, more reputable sites on the assumption that increasing the volume of web pages with random, irrelevant content is a good long-term strategy. Purely scraped content, even from high-quality sources, may not provide any added value to your users without additional useful services or content provided by your site. It's worthwhile to take the time to create original content that sets your site apart. This will keep your visitors coming back and will provide useful search results.

In 2010, our mailboxes suddenly started overflowing with complaints from users – complaints that they were doing perfectly reasonable Google searches, and ending up on scraper sites that mirrored Stack Overflow content with added advertisements. Even worse, in some cases, the original Stack Overflow question was nowhere to be found in the search results! That's particularly odd because our attribution terms require linking directly back to us, the canonical source for the question, without nofollow. Google, in indexing the scraped page, cannot avoid seeing that the scraped page links back to the canonical source. This culminated in, of all things, a special browser plug-in that redirects to Stack Overflow from the ripoff sites. How totally depressing. Joel and I thought this was impossible. And I felt like I had personally failed all of you.

The idea that there could be something wrong with Google was inconceivable to me. Google is gravity on the web, an omnipresent constant; blaming Google would be like blaming gravity for my own clumsiness. It wasn't even an option. I started with the golden rule: it's always my fault. We did a ton of due diligence on webmasters.stackexchange.com to ensure we weren't doing anything overtly stupid, and uber-mensch Matt Cutts went out of his way to investigate the hand-vetted search examples contributed in response to my tweet asking for search terms where the scrapers dominated. Issues were found on both sides, and changes were made. Success!

Despite the semi-positive resolution, I was disturbed. If these dime-store scrapers were doing so well and generating so much traffic on the back of our content – how was the rest of the web faring? My enduring faith in the gravitational constant of Google had been shaken. Shaken to the very core.

Throughout my investigation I had nagging doubts that we were seeing serious cracks in the algorithmic search foundations of the house that Google built. But I was afraid to write an article about it for fear I'd be claimed an incompetent kook. I wasn't comfortable sharing that opinion widely, because we might be doing something obviously wrong. Which we tend to do frequently and often. Gravity can't be wrong. We're just clumsy … right?

I can't help noticing that we're not the only site to have serious problems with Google search results in the last few months. In fact, the drum beat of deteriorating Google search quality has been practically deafening of late:

Anecdotally, my personal search results have also been noticeably worse lately. As part of Christmas shopping for my wife, I searched for "iPhone 4 case" in Google. I had to give up completely on the first two pages of search results as utterly useless, and searched Amazon instead.

People whose opinions I respect have all been echoing the same sentiment -- Google, the once essential tool, is somehow losing its edge. The spammers, scrapers, and SEO'ed-to-the-hilt content farms are winning.

Like any sane person, I'm rooting for Google in this battle, and I'd love nothing more than for Google to tweak a few algorithmic knobs and make this entire blog entry moot. Still, this is the first time since 2000 that I can recall Google search quality ever declining, and it has inspired some rather heretical thoughts in me -- are we seeing the first signs that algorithmic search has failed as a strategy? Is the next generation of search destined to be less algorithmic and more social?

It's a scary thing to even entertain, but maybe gravity really is broken.

Posted by Jeff Atwood    159 Comments

The Dirty Truth About Web Passwords

December 14, 2010

This weekend, the Gawker network was compromised.

This weekend we discovered that Gawker Media's servers were compromised, resulting in a security breach at Lifehacker, Gizmodo, Gawker, Jezebel, io9, Jalopnik, Kotaku, Deadspin, and Fleshbot. If you're a commenter on any of our sites, you probably have several questions.

It's no Black Sunday or iPod modem firmware hack, but it has release notes -- and the story it tells is as epic as Beowulf:

So, here we are again with a monster release of ownage and data droppage. Previous attacks against the target were mocked, so we came along and raised the bar a little. How's this for "script kids"? Your empire has been compromised, your servers, your databases, online accounts and source code have all been ripped to shreds!

You wanted attention, well guess what, You've got it now!

Read those release notes. It'll explain how the compromise unfolded, blow by blow, from the inside.

Gawker is operated by Nick Denton, notorious for the unapologetic and often unethical "publish whatever it takes to get traffic" methods endorsed on his network. Do you remember the iPhone 4 leak? That was Gawker. Do you remember the article about bloggers being treated as virtual sweatshop workers? That was Gawker. Do you remember hearing about a blog lawsuit? That was probably Gawker, too.

Some might say having every account on your network compromised is exactly the kind of unwanted publicity attention that Gawker was founded on.

Personally, I'm more interested in how we can learn from this hack. Where did Gawker go wrong, and how can we avoid making those mistakes on our projects?

  1. Gawker saved passwords. You should never, ever store user passwords. If you do, you're storing passwords incorrectly. Always store the salted hash of the password -- never the password itself! It's so easy, even members of Mensa er .. can't .. figure it out.

  2. Gawker used encryption incorrectly. The odd choice of archaic DES encryption meant that the passwords they saved were all truncated to 8 characters. No matter how long your password actually was, you only had to enter the first 8 characters for it to work. So much for choosing a secure pass phrase. Encryption is only as effective as the person using it. I'm not smart enough to use encryption, either, as you can see in Why Isn't My Encryption.. Encrypting?

  3. Gawker asked users to create a username and password on their site. The FAQ they posted about the breach has two interesting clarifications:

    2) What if I logged in using Facebook Connect? Was my password compromised?
    No. We never stored passwords of users who logged in using Facebook Connect.

    3) What if I linked my Twitter account with my Gawker Media account? Was my Twitter password compromised?
    No. We never stored Twitter passwords from users who linked their Twitter accounts with their Gawker Media account.

    That's right, people who used their internet driver's license to authenticate on these sites had no security problems at all! Does the need to post a comment on Gizmodo really justify polluting the world with yet another username and password? It's only the poor users who decided to entrust Gawker with a unique username and 'secure' password who got compromised.

(Beyond that, "don't be a jerk" is good advice to follow in business as well as your personal life. I find that you generally get back what you give. When your corporate mission is to succeed by exploiting every quasi-legal trick in the book, surely you can't be surprised when you get the same treatment in return.)

But honestly, as much as we can point and laugh at Gawker and blame them for this debacle, there is absolutely nothing unique or surprising about any of this. Regular readers of my blog are probably bored out of their minds by now because I just trotted out a whole bunch of blog posts I wrote 3 years ago. Again.

Here's the dirty truth about website passwords: the internet is full of websites exactly like the Gawker network. Let's say you have good old traditional username and passwords on 50 different websites. That's 50 different programmers who all have different ideas of how your password should be stored. I hope for your sake you used a different (and extremely secure) password on every single one of those websites. Because statistically speaking, you're screwed.

In other words, the more web sites you visit, the more networks you touch and trust with a username and password combination -- the greater the odds that at least one of those networks will be compromised exactly like Gawker was, and give up your credentials for the world to see. At that point, unless you picked a strong, unique password on every single site you've ever visited, the situation gets ugly.

The bad news is that most users don't pick strong passwords. This has been proven time and time again, and the Gawker data is no different. Even worse, most users re-use these bad passwords across multiple websites. That's how this ugly Twitter worm suddenly appeared on the back of a bunch of compromised Gawker accounts.

Xkcd-excerpt

Now do you understand why I've been so aggressive about promoting the concept of the internet driver's license? That is, logging on to a web site using a set of third party credentials from a company you can actually trust to not be utterly incompetent at security? Sure, we're centralizing risk here to, say, Google, or Facebook -- but I trust Google a heck of a lot more than I trust J. Random Website, and this really is no different in practice than having password recovery emails sent to your GMail account.

I'm not here to criticize Gawker. On the contrary, I'd like to thank them for illustrating in broad, bold relief the dirty truth about website passwords: we're all better off without them. If you'd like to see a future web free of Gawker style password compromises -- stop trusting every random internet site with a unique username and password! Demand that they allow you to use your internet driver's license -- that is, your existing Twitter, Facebook, Google, or OpenID credentials -- to log into their website.

Posted by Jeff Atwood    72 Comments

My Holiday in Beautiful Panau

November 28, 2010

There is a high correlation between "programmer" and "gamer". One of the first Area 51 sites we launched, based on community demand, was gaming.stackexchange.com. Despite my fundamental skepticism about gaming as a Q&A topic -- as expressed on episode 87 of Herding Code -- I have to admit it has far exceeded my expectations.

But then maybe I shouldn't be so surprised. I've talked about the relationship between gamer and programmer before:

I used to recommend games on this very blog that I particularly enjoyed and felt were worthy of everyone's attention. I don't do this a lot any more, now that my blogging schedule has slipped to one post a week, if I'm lucky. (If you're wondering why, it's because running your own business is crazy stupid amounts of work when you turn it up to eleven.) Here are a few games I've recommended in the past:

I haven't had a ton of time to play games, other than the inevitable Rock Band 3, but I've been consumed by another game I had no idea would become so addictive -- Just Cause 2.

Just-cause-2

It's what you might call an open world sandbox game, in the vein of the Grand Theft Autos. But I could never get into the GTA games, even after trying GTA 3 and its sequels Vice City and San Andreas. They just left me cold, somehow.

Where GTA and its ilk often felt a tad too much like work for my tastes, Just Cause 2 is almost the opposite -- it is non-stop, full blown open world pandemonium from start to finish. One of the game's explicit goals is that you advance the plot by blowing stuff up. No, seriously. I'm not kidding. You have an entire 1000+ square kilometer island paradise at your disposal, filled with cities and military bases, spanning the range from snowy mountains to deserts to idyllic beaches -- all just waiting for you to turn them into "chaos points" … by any means necessary.

Just-cause-2-helicopter

Of course, you get around by hijacking whatever vehicles happen by, be they boats, airplanes, jumbo jets, cars, tanks, trucks, buses, monster trucks, motorcycles, scooters, tractors or anything in between. Even on foot it is fun to navigate the island of Panau, because the developers gave us an impossibly powerful personal zipline that you can fire at any object in the game to propel yourself toward it. Combine that with the magical parachute you can deploy anywhere, anytime, and they make for some fascinating diversions (parasailing anyone?). You can also use the zipline to attach any two objects together. Think about that for a second. Have you ever wondered what happens when you zipline a moving vehicle to a tree? Or a pedestrian? Or another vehicle? Hmm. As a result, simply going walkabout on the island is more fun than I ever would have imagined.

Between the 49 plot missions, 9 stronghold takeovers, 104 unique vehicles, the optional boat/plane/car/parachute race missions,the opportunities for insane stunt points, the umpteen zillion upgrade crates and faction objects to collect, and the 360+ locations in the 1000+ square kilometers of Panau -- there's always something interesting happening around every corner. And whatever it is, it's probably beautiful and blows up real good.

Just-cause-2-explosion

In short, Just Cause 2 is deliriously, stupidly, absurdly entertaining. I can't even remember the last game I completed where I felt compelled to go back after finishing the main storyline to discover even more areas I missed during my initial playthrough and get (most of) the in-game achievements. Whatever amount of time you have to play, Just Cause 2 will happily fill it with totally unscripted, awesome open world pandemonium.

Don't take my word for it; even the notoriously acidic game reviewer Yahtzee had almost nothing negative to say about Just Cause 2, which is his version of a positive review. And Metacritic gives Just Cause 2 a solid 84. Not that it can't be improved, of course; after such a sublime sandbox experience, I'm desperately curious to see what they'll add for Just Cause 3.

Luckily for you, the game has been out long enough that it can be picked up for a song on PS3, Xbox, or PC. Steam has Just Cause 2 on sale right now in an 8 player pack for $60, and Amazon has all versions in stock for under $30. Beware, though, as the PC version does require a pretty solid video card along with Windows Vista or newer -- but the upside is that I have mine cranked up to 2048x1152 with almost all the options on, and it rarely dips below 60 fps.

I spent my holidays on the beautiful island of Panau, and I don't regret a second of it. If you're looking for a vacation spot, I heartily recommend the open world sandbox of Panau. But while you're visiting, do be mindful of any errant gunfire, vehicles, and explosions.

Posted by Jeff Atwood    22 Comments

Your Internet Driver's License

November 24, 2010

Back in summer 2008 when we were building Stack Overflow, I chose OpenID logins for reasons documented in Does The World Really Need Yet Another Username and Password:

I realize that OpenID is far from an ideal solution. But right now, the one-login-per-website problem is so bad that I am willing to accept these tradeoffs for a partial worse is better solution. There's absolutely no way I'd put my banking credentials behind an OpenID. But there are also dozens of sites that I don't need anything remotely approaching banking-grade security for, and I use these sites far more often than my bank. The collective pain of remembering all these logins -- and the way my email inbox becomes a de-facto collecting point and security gateway for all of them -- is substantial.

It always pained me greatly that every rinky-dink website on the entire internet demanded that I create a special username and password just for them. Yes, if you're an alpha geek, then you probably use a combination of special software and USB key from your utility belt to generate secure usernames and passwords for the dozens of websites you frequent. But for the vast, silent majority of normals, who know nothing of security but desire convenience above all, this means one thing: using the same username and password over and over. And it's probably a simple password, too.

This is the status quo of identity on the internet. It is deeply and fundamentally broken.

But it doesn't have to be this way. If you open your wallet (or purse, or man-purse, or whatever), I bet you'll find a variety of credentials you use to prove your identity wherever you go.

Wallet-contents

The average wallet contains a few different forms of identity with varying strengths:

  • Strong: California driver's license, student ID
  • Moderate: credit cards, health insurance card, video rental membership, gym card
  • Weak: Albertson's Preferred Card, Best Buy Rewards Zone Card, Coffee loyalty card

(and sometimes even, uh, cards for free lapdances, apparently)

In the real world, we don't regularly hold two dozen forms of identity like we expect people to on the web. Not only would you be carrying around the freaking Constanza wallet at that point, it would be insane. In the real world, we somehow manage to get by with about two or three strong forms of identity, complemented by a few other weaker forms to taste.

I'm proposing that our web wallets begin to mimic our physical wallets. Whenever a website needs to know who I am, they should ask to see my Internet Driver's License.

Bigfoot-drivers-license

Now, I don't literally mean a driver's license. I'm using this term figuratively to mean online credentials that I can re-use in more than one place on the internet. If all I want to do is leave a comment on a blog -- like, say, this one -- then one of the weaker forms of identity will surely do. If I'm starting a new bank account, or setting up a profile on a dating website, then maybe a stronger credential from my virtual wallet is necessary.

The core concept that users need to get used to is logging in to a website by showing a third party credential to validate their identity. This idea isn't nearly as crazy as it seemed in 2008. How many websites can you log into by showing your Facebook, Google, or Twitter credentials now? Lots!

Disqus-login

The whole online identity situation may seem as impossible as peace in the Middle East at this point. But when faced with a problem that appears intractable, is your solution to throw your hands up, mindlessly embrace the status quo, and wearily sigh "whaddaya gonna do?"

Some people do that. It's their right. Personally, I prefer to be the change I want to see. So for us, on Stack Overflow and the Stack Exchange network, that means aggressively promoting the concept of the Internet Driver's License. Including educating users as necessary.

For example, consider this ATM machine. To use it, do I need to sign up for an account at Shanghai Peking Development Bank? No. I can use any form of trusted third-party credentials the machine supports.

Atm-machine

Similarly, to log into any Stack Exchange site, including Stack Overflow, present any OpenID or OAuth 2.0 compliant identity provider as your Internet Driver's License.

Atm-machine-stackoverflow

When we founded Stack Overflow, we set out with the explicit mission to make the internet better. Adding yet another meaningless username and password to the fabric of the web does not make it better. What does make the internet better is continued pursuit of better, simpler, re-usable forms of third party online identity. That's why I urge you to join me in supporting OpenID, OAuth 2.0, and any other promising implementations of the Internet Driver's License.

Posted by Jeff Atwood    84 Comments

Breaking the Web's Cookie Jar

November 13, 2010

The Firefox add-in Firesheep caused quite an uproar a few weeks ago, and justifiably so. Here's how it works:

  • Connect to a public, unencrypted WiFi network. In other words, a WiFi network that doesn't require a password before you can connect to it.
  • Install Firefox and the Firesheep add-in.
  • Wait. Maybe have a latte while you're waiting.
  • Click on the user / website icons that appear over time in Firesheep to instantly log in as that user on that website.

Crazy! This guy who wrote Firesheep must be a world-class hacker, right?

Well, no. The work to package this up in a point-and-click way that is (sort of) accessible to power users is laudable, but what Firesheep actually does is far from magical. It's more of an art project and PR stunt than an actual hack of any kind. Still, I was oddly excited to see Firesheep get so much PR, because it highlights a fundamental issue with the architecture of the web.

The web is kind of a primitive medium. The only way websites know who you are is through tiny, uniquely identifiying strings your browser sends to the webserver on each and every click:

GET / HTTP/1.1
Host: diy.stackexchange.com
Connection: keep-alive
User-Agent: Chrome/7.0.517.44
Accept-Language: en-US,en;q=0.8
Cookie: diyuser=t=ZlQOG4kege&s=8VO9gjG7tU12s
If-Modified-Since: Tue, 09 Nov 2010 04:41:12 GMT

These are the typical sort of HTTP headers your browser sends to a website on every click. See that little cookie in bright red? To a website, that's your fingerprint, DNA, and social security number all rolled into one. Some part of the cookie contains a unique user ID that tells the website you are you.

And guess what? That cookie is always broadcast in plain text every single time you click a link on any website. Right out in the open where anyone -- well, technically, anyone who happens to be on the same network as you and is in a position to view your network packets -- can just grab it out of the ether and immediately impersonate you on any website you are a member of.

Broken-cookie

Now that you know how cookies work (and I'm not saying it's rocket surgery or anything), you also know that what Firesheep does is relatively straightforward:

  1. Listen to all HTTP traffic.
  2. Wait for HTTP headers from a known website.
  3. Isolate the part of the cookie header that identifies the user.
  4. Launch a new browser session with that cookie. Bam! As far as the target webserver is concerned, you are that user!

All Firesheep has to do, really, is listen. That's pretty much all there is to this "hack". Scary, right? Well, then you should be positively quaking in your boots, because this is the way the entire internet has worked since 1994, when cookies were invented.

So why wasn't this a problem in, say, 2003? Three reasons:

  1. Commodity public wireless internet connections were not exactly common until a few years ago.
  2. Average people have moved beyond mostly anonymous browsing and transferred significant parts of their identity online (aka the Facebook effect).
  3. The tools required to listen in on a wireless network are slightly … less primitive now.

Firesheep came along at the exact inflection point of these three trends. And mind you, it is still not a sure thing -- Firesheep requires a particular set of wireless network chipsets that support promiscuous mode in the lower level WinPcap library that Firesheep relies on. But we can bet that the floodgates have been opened, and future tools similar to this one will become increasingly a one-click affair.

The other reason this wasn't a problem in 2003 is because any website that truly needed security switched to encrypted HTTP -- aka Secure HTTP -- long ago. HTTPS was invented in 1994, at the same time as the browser cookie. This was not a coincidence. The creators of the cookie knew from day one they needed a way to protect them from prying eyes. Even way, way back in the dark, primitive ages of 2003, any banking website or identity website worth a damn wouldn't even consider using plain vanilla HTTP. They'd be laughed off the internet!

The outpouring of concern over Firesheep is justified, because, well, the web's cookie jar has always been kind of broken -- and we ought to do something about it. But what?

Yes, you can naively argue that every website should encrypt all their traffic all the time, but to me that's a "boil the sea" solution. I'd rather see a better, more secure identity protocol than ye olde HTTP cookies. I don't actually care if anyone sees the rest of my public activity on Stack Overflow; it's hardly a secret. But gee, I sure do care if they somehow sniff out my cookie and start running around doing stuff as me! Encrypting everything just to protect that one lousy cookie header seems like a whole lot of overkill to me.

I'm not holding my breath for that to happen any time soon, though. So here's what you can do to protect yourself, right now, today:

  1. We should be very careful how we browse on unencrypted wireless networks. This is the great gift of Firesheep to all of us. If nothing else, we should be thanking the author for this simple, stark warning. It's an unavoidable fact of life: if you must go wireless, seek out encrypted wireless networks. If you have no other choices except unencrypted wireless networks, browse anonymously -- quite possible if all you plan to do is casually surf the web and read a few articles -- and only log in to websites that support https. Anything else risks identity theft.
  2. Get in the habit of accessing your web mail through HTTPS. Email is the de-facto skeleton key to your online identity. When your email is compromised, all is lost. If your webmail provider does not support secure http, they are idiots. Drop them like a hot potato and immediately switch to one that does. Heck, the smart webmail providers already switched to https by default!
  3. Lobby the websites you use to offer HTTPS browsing. I think we're clearly past the point where only banks and finance sites should be expected to use secure HTTP. As more people shift more of their identities online, it makes sense to protect those identities by moving HTTPS from the domain of a massive bank vault door to just plain locking the door. SSL isn't as expensive as it used to be, in every dimension of the phrase, so this is not an unreasonable thing to ask your favorite website for.

This is very broad advice, and there are a whole host of technical caveats to the above. But it's a starting point toward evangelizing the risks and responsible use of open wireless networks. Firesheep may indeed have broken the web's cookie jar. But it was kind of an old, beat up, cracked cookie jar in the first place. I hope the powers that be will use Firesheep as incentive to build a better online identity solution than creaky old HTTP cookies.

Posted by Jeff Atwood    65 Comments

The Keyboard Cult

October 22, 2010

As a guy who spends most of his day typing words on a screen, it's hard for me to take touch computing seriously. I love my iPhone 4, and smartphones are the ultimate utility belt item, but attempting to compose any kind of text on the thing is absolutely crippling. It is a reasonable compromise for a device that fits in your pocket … but that's all.

The minute I switch back to my regular keyboard, I go from being Usain Bolt to the Flash.

Touchscreen-vs-keyboard

Touchscreens are great for passively browsing, as Scott Adams noted:

Another interesting phenomenon of the iPhone and iPad era is that we are being transformed from producers of content into consumers. With my BlackBerry, I probably created as much data as I consumed. It was easy to thumb-type long explanations, directions, and even jokes and observations. With my iPhone, I try to avoid creating any message that are over one sentence long. But I use the iPhone browser to consume information a hundred times more than I did with the BlackBerry. I wonder if this will change people over time, in some subtle way that isn't predictable. What happens when people become trained to think of information and entertainment as something they receive and not something they create?

Because we run an entire network of websites devoted to learning by typing words on a page, it's difficult for me to get past this.

But I'm not here to decry the evils of touchscreen typing. It has its place in the pantheon of computing. I'm here to sing the praises of the humble keyboard. The device that, when combined with the internet, turns every human being into a highly efficient global printing press.

My love affair with the keyboard goes way back:

Maybe I'm biased. As I recently remarked on programmers.stackexchange.com, I can't take slow typists seriously as programmers. When was the last time you saw a hunt-and-peck pianist?

I've been monogamous with the Microsoft Natural Keyboard 4000 for a long time. But in this supposedly happy marriage, I was accidentally neglecting one of the most crucial aspects of the keyboard experience.

The vast majority of keyboards included with white box systems or sold at office supply stores are rubber dome or membrane keyboards.

Membrane-remote-control

They are inexpensive, mass produced, relatively low quality devices that are inconsistent and degrade the user experience. Most users don't know this, or simply don't care. The appeal of cheap rubber dome or membrane keyboards is that they're usually available in a variety of styles, are included "free" with a new system, and they may sport additional features like media controls or wireless connectivity. But these cheap keyboards typically don't provide users with any tactile feedback, the keys feel mushy and may not all actuate at the same point, and the entire keyboard assemblies themselves tend to flex and move around when typed on. Not fun.

All this time, I've been typing on keyboards with least-common-denominator rubber dome innards. I was peripherally aware of higher quality mechanical keyboards, but I never appreciated them until I located this absolutely epic mechanical keyboard guide thread. It's also the source of an entire forum of people at geekhack.org who are mechanical keyboard enthusiasts. These kinds of communities and obsessions, writ so large and with such obvious passion, fascinate me. They are the inspiration for what we are trying to do with Stack Overflow and the Stack Exchange network.

If you don't have time to read that epic guide (but you should!), allow me to summarize:

  1. Almost all computer and laptop keyboards today use cheap, low quality switches -- rubber dome, membrane, scissor, or foam element.
  2. Mechanical switches are considered superior in every way by keyboard enthusiasts.
  3. Because the general public largely doesn't care about keyboard feel or durability, and because mechanical switches are more expensive, mechanical switch keyboards are quite rare these days.

Mechanical switches look, well, mechanical. They're spiritually the same as those old-school arcade buttons we used to mash on in the 1980s. You push down on the key, and the switch physically actuates.

Buckling-spring-switch

Yes, we are rapidly approaching the threshold of esoterica here. Mechanical keyboards were already becoming rare even before the internet, so I'd wager many people now reading this can't possibly know the difference between a typical cheap membrane keyboard and a fancy mechanical model because they've never had the opportunity to try one!

We should rectify that.

If you want to dip your fingers into the world of mechanical switch keyboards, start by asking yourself a few questions:

  • Are you willing to spend $70 to $300 for a keyboard?
  • How noisy do you want your typing to be?
  • Do you want a tactile "snap" when the key is depressed?
  • How much force do you type with -- do you have a light or heavy touch?
  • How much key travel do you want?
Next, there are further subtleties to consider, like how the keys are printed:

  • Pad Printed -- the standard cheap stuff. Little more than stickers. Keycaps will wear off fast.
  • Laser Etched -- permanent, but leaves tiny surface scars on the keys due to the characters being literally burned into the keys. May also be a tiny bit blurry.
  • Dye Sublimated -- dye set into plastic; expensive but nearly optimal.
  • Injection Molded -- two keys in different colors are physically bonded together. Very expensive but considered as close to perfect as you can get. Notably, NeXT keyboards used this method.

And what about the shape of the keycaps? Cylindrical? Spherical? Flat? And if you're an avid keyboard gamer, you might want to consider n-key rollover, too. I warned you this rabbit hole was deep.

Let's start looking at a few likely candidates. The one you may already know is Das Keyboard.

Das-keyboard

Das is a good, reliable brand of mechanical keyboards. They have two primary models. Each is available in the "blank keycaps" versions if you are the sort of ninja typist who doesn't need to look at the keyboard -- you type by chanelling the Force.

The "silent" mechanical switch distinction is an important one: mechanical switches can be loud. How loud? The DAS website actually sells honest-to-god earplugs as a keyboard accessory. I'm sure it's slightly tongue in cheek. Maybe. But consider yourself warned, and choose the silent model if you aren't a fan of the clickety-clack typing.

If you want the most old-school IBM-esque experience possible, and a true classic buckling spring keyboard, then Unicomp is your huckleberry. The common models are the Customizer 104/105 and SpaceSaver 104/105.

Unicomp-spacesaver

Next up is Elite Keyboards, but I can only recommend the (slightly expensive) Topre Realforce model due to the cheap pad keycap printing used on their other models.

Topre-realforce

Finally, Deck Keyboards -- I remember writing about these guys years ago. They have a full sized keyboard now with a lot of attention to detail: The Deck Legend.

Deck-legend-keyboard

It is also the only keyboard in its class that is backlit, if that's your bag.

Of course, none of these premium fancypants mechanical switch keyboards are really necessary. The most important aspect of writing isn't the keyboard you use, but the simple act of getting out there and writing as much as you can. But if, like me, you accidentally fall in love with the keyboard and everything it represents -- then I think you owe it to yourself to find out what a great keyboard is supposed to feel like.

Posted by Jeff Atwood    143 Comments

Because Everyone Needs a Router

September 25, 2010

Do you remember when a router used to be an exotic bit of network kit?

Those days are long gone. A router is one of those salt-of-the-earth items now; anyone who pays for an internet connection needs a router, for:

  1. NAT and basic hardware firewall protection from internet evildoers
  2. A wired network hub to connect local desktop PCs
  3. A wireless hub to connect laptops, phones, consoles, etcetera

Let me put it this way: my mom – and my wife's mom – both own routers. If that isn't the definition of mainstream, I don't know what is.

Since my livelihood revolves around being on the internet, and because I'm a bit of a tweaker, I have a fancy-ish router. But it is of late 2007 vintage:

Although the DGL-4500 is a nice router, and it has served me well with no real complaints, the last major firmware update for it was a year and a half ago. There have been some desultory minor updates since then, but clearly the vendor has, shall we say, moved on to focusing on newer models.

The router is (literally!) the central component in my overall internet experience, and I was growing increasingly uncomfortable with the status quo. Frankly, the prospect of three year old hardware with year old firmware gives me the heebie-jeebies.

So, I asked the pros at Super User, even going so far as to set up a Recommend Me a Router chat room. (We disallow product recommendation questions as they become uselessly out of date so quickly, but this is a perfect topic for a chat room.) I got some fantastic advice from my fellow Super Users via chat, though much of it was of the far too sane "if it ain't broke don't fix it" variety. Well, that's just not how I work. To be fair, the router market is not exactly a hotbed of excitement at the moment; it is both saturated and heavily commoditized, particularly now that the dust has settled from the whole 802.11 A/B/G/N debacle. There just isn't much going on.

But in the process of doing my router research, I discovered something important, and maybe even revolutionary in its own quiet little way. The best router models all run open source firmware!

That's right, the truly great routers are available in "awesome" edition. (There may be other open source router firmwares out there, but these are the two I saw most frequently.) I learned that these open source firmwares can turn a boring Clark Kent router into Superman. And they are always kept updated by the community, in perpetuity.

In my weaker moments, I toyed with the idea of building a silent mini x86 PC that could run a routing optimized distribution of Linux, but the reality is that current commodity routers have more than enough memory and embedded CPU power – not to mention the necessary wireless and gigabit ethernet hub bits already built in. Dedicating a whole x86 PC to routing is power inefficient, overly complex, and awkward.

Yes, today's router marketplace is commoditized and standardized and boring – but there are still a few clear hardware standouts. I turned to the experts at SmallNetBuilder for their in-depth technical reviews, and found two consensus recommendations:

Update: Though these models are still fine choices, particularly if you can find a great deal on them, I have newer recommendations in Because Everyone (Still) Needs a Router.

Buffalo Nfiniti Wireless-N High Power Router ($80)

Buffalo_wzr-hp-g300nh

NETGEAR WNDR3700 RangeMax Dual Band Wireless-N ($150)

Netgear-wndr3700-router

Both of these models got glowing reviews from the networking experts at SmallNetBuilder, and both are 100% compatible with the all-important open source dd-wrt firmware. You can't go wrong with either, but I chose the less expensive Buffalo Nfiniti router. Why?

  1. It's almost half the price, man!
  2. The "high power" part is verifiably and benchmarkably true, and I have some wireless range problems at my home.
  3. I do most of my heavy network lifting through wired gigabit ethernet, so I can't think of any reason I'd need the higher theoretical wireless throughput of the Netgear model.
  4. Although the Netgear has a 680 Mhz embedded CPU and 128mb RAM, the Buffalo's 400 MHz embedded CPU and 64mb of RAM is not exactly chopped liver, either; it's plenty for dd-wrt to work with. I'd almost go so far as to say the Netgear is a bit overkill… if you're into that sort of thing.

I received my Buffalo Nfiniti and immediately installed dd-wrt on it, which was very simple and accomplished through the existing web UI on the router. (Buffalo has a history of shipping rebranded dd-wrt distributions in their routers, so the out-of-box firmware is a kissing cousin.)

After rebooting, I was in love. The (more) modern gigabit hardware, CPU, and chipset was noticably snappier everywhere, even just dinking around in the admin web pages. And dd-wrt scratches every geek itch I have – putting that newer hardware to great use. Just check out the detailed stats I can get, including that pesky wireless signal strength problem. The top number is the Xbox 360 outside, the bottom number is my iPhone from about 10 feet away.

Dd-wrt-wireless-client-info

Worried your router is running low on embedded CPU grunt, or that 64 megabytes of memory is insufficient? Never fear; dd-wrt has you covered. Just check out the detailed, real time memory and cpu load stats.

Dd-wrt-memory-cpu-stats

Trying to figure out how much WAN/LAN/Wireless bandwidth you're using? How does a real time SVG graph, right from the router admin pages, grab you?

Dd-wrt-bandwidth-graph

It's just great all around. And I haven't even covered the proverbial laundry list of features that dd-wrt offers above and beyond most stock firmware! Suffice it to say that this is one of those times when the "let's support everything" penchant of open source projects works in our favor. Don't worry, it's all (mostly) disabled by default. Those features and tweaks can all safely be ignored; just know that they're available to you when and if you need them.

This is boring old plain vanilla commodity router hardware, but when combined with an open source firmware, it is a massive improvement over my three year old, proprietary high(ish) end router. The magic router formula these days is a combination of commodity hardware and open-source firmware. I'm so enamored of this one-two punch combo, in fact, I might even say it represents the future. Not just of the everyday workhorse routers we all need to access the internet – but the future of all commodity hardware.

Routers; we all need 'em, and they are crucial to our internet experience. Pick whichever router you like – as long as it's compatible with one of the open source firmware packages! Thanks to a wide variety of mature commodity hardware choices, plus infinitely and perpetually updated open source router firmware, I'm happy to report that now everyone can have a great router.

Posted by Jeff Atwood    78 Comments

YouTube vs. Fair Use

September 17, 2010

In YouTube: The Big Copyright Lie, I described my love-hate relationship with YouTube, at least as it existed in way back in the dark ages of 2007.

Now think back through all the videos you've watched on YouTube. How many of them contained any original content?

It's perhaps the ultimate case of cognitive dissonance: by YouTube's own rules [which prohibit copyrighted content], YouTube cannot exist. And yet it does.

How do we reconcile YouTube's official hard-line position on copyright with the reality that 90% of the content on their site is clearly copyrighted and clearly used without permission? It seems YouTube has an awfully convenient "don't ask, don't tell" policy-- they make no effort to verify that the uploaded content is either original content or fair use. The copyrighted content stays up until the copyright owner complains. Then, and only then, is it removed.

Today's lesson, then, is be careful what you ask for.

At the time, I just assumed that YouTube would never be able to resolve this problem through technology. The idea that you could somehow fingerprint every user-created uploaded video against every piece of copyrighted video ever created was so laughable to me that I wrote it off as impossible.

A few days ago I uploaded a small clip from the movie Better Off Dead to YouTube, in order to use it in the Go That Way, Really Fast blog entry. This is quintessential fair use: a tiny excerpt of the movie, presented in the context of a larger blog entry. So far, so good.

But then I uploaded a small clip from a different movie that I'm planning to use in another, future blog entry. Within an hour of uploading it, I received this email:

Dear {username},

Your video, {title}, may have content that is owned or licensed by {company}.

No action is required on your part; however, if you are interested in learning how this affects your video, please visit the Content ID Matches section of your account for more information.

Sincerely,
- The YouTube Team

This 90 second clip is from a recent movie. Not a hugely popular movie, mind you, but a movie you've probably heard of. This email both fascinated and horrified me. How did they match a random, weirdly cropped (thanks, Windows Movie Maker) clip from the middle of a non-blockbuster movie within an hour of me uploading it? This had to be some kind of automated process that checks uploaded user content against every piece of copyrighted content ever created (or the top n subset thereof), exactly the kind that I thought was impossible.

Uh oh.

I began to do some research. I quickly found Fun with YouTube's Audio Content ID System, which doesn't cover video, but it's definitely related:

I was caught by surprise one day when I received an automated email from YouTube informing me that my video had a music rights issue and it was removed from the site. I didn't really care.

Then a car commercial parody I made (arguably one of my better videos) was taken down because I used an unlicensed song. That pissed me off. I couldn't easily go back and re-edit the video to remove the song, as the source media had long since been archived in a shoebox somewhere. And I couldn't simply re-upload the video, as it got identified and taken down every time. I needed to find a way to outsmart the fingerprinter. I was angry and I had a lot of free time. Not a good combination.

I racked my brain trying to think of every possible audio manipulation that might get by the fingerprinter. I came up with an almost-scientific method for testing each modification, and I got to work.

Further research led me to this brief TED talk, How YouTube Thinks About Copyright.

We compare each upload against all the reference files in our database. This heat map is going to show you how the brain of this system works.

Here we can see the reference file being compared to the user generated content. The system compares every moment of one to the other to see if there's a match. This means we can identify a match even if the copy uses just a portion of the original file, plays it in slow motion, and has degraded audio or video.

The scale and speed of this system is truly breathtaking -- we're not just talking about a few videos, we're talking about over 100 years of video every day between new uploads and the legacy scans we regularly do across all of the content on the site. And when we compare those 100 years of video, we're comparing it against millions of reference files in our database. It'd be like 36,000 people staring at 36,000 monitors each and every day without as much as a coffee break.

I have to admit that I'm astounded by the scope, scale, and sheer effectiveness of YouTube's new copyright detection system that I thought was impossible! Seriously, watch the TED talk. It's not long. The more I researched YouTube's video identification tool, the more I realized that resistance is futile. It's so good that the only way to defeat it is by degrading your audio and video so much that you have effectively ruined it. And when it comes to copyright violations, if you can achieve mutually assured destruction, then you have won. Absolutely and unconditionally.

This is an outcome so incredible I am still having trouble believing it. But I have the automatically blocked uploads to prove it.

Now, I am in no way proposing that copyright is something we should be trying to defeat or work around. I suppose I was just used to the laissez faire status quo on YouTube, and the idea of a video copyright detection system this effective was completely beyond the pale. My hat is off to the engineers at Google who came up with this system. They aren't the bad guys here; they offer some rather sane alternatives when copyright matches are found:

If Content ID identifies a match between a user upload and material in the reference library, it applies the usage policy designated by the content owner. The usage policy tells the system what to do with the video. Matches can be to only the audio portion of an upload, the video portion only, or both.

There are three usage policies -- Block, Track or Monetize. If a rights owner specifies a Block policy, the video will not be viewable on YouTube. If the rights owner specifies a Track policy, the video will continue to be made available on YouTube and the rights owner will receive information about the video, such as how many views it receives. For a Monetize policy, the video will continue to be available on YouTube and ads will appear in conjunction with the video. The policies can be region-specific, so a content owner can allow a particular piece of material in one country and block the material in another.

The particular content provider whose copyright I matched chose the draconian block policy. That's certainly not Google's fault, but I guess you could say I'm Feeling Unlucky.

Although the 90 second clip I uploaded is clearly copyrighted content -- I would never dispute that -- my intent is not to facilitate illegal use, but to "quote" the movie scene as part of a larger blog entry. YouTube does provide recourse for uploaders; they make it easy to file a dispute once the content is flagged as copyrighted. So I dutifully filled out the dispute form, indicating that I felt I had a reasonable claim of fair use.

Youtube-fair-use-dispute

Unfortunately, my fair use claim was denied without explanation by the copyright holder.

Let's consider the four guidelines for fair use I outlined in my original 2007 blog entry:

  1. Is the use transformative?
  2. Is the source material intended for the public good?
  3. How much was taken?
  4. What's the market effect?

While we're clear on 3 and 4, items 1 and 2 are hazy in a mashup. This would definitely be transformative, and I like to think that I'm writing for the erudition of myself and others, not merely to entertain people. I uploaded with the intent of the video being viewed through a blog entry, with YouTube as the content host only. But it was still 90 seconds of the movie viewable on YouTube by anyone, context free.

So I'm torn.

On one hand, this is an insanely impressive technological coup. The idea that YouTube can (with the assistance of the copyright holders) really validate every minute of uploaded video against every minute of every major copyrighted work is unfathomable to me. When YouTube promised to do this to placate copyright owners, I was sure they were delaying for time. But much to my fair-use-loving dismay, they've actually gone and built the damn thing -- and it works.

Just, maybe, it works a little too well. I'm still looking for video sharing services that offer some kind of fair use protection.

Posted by Jeff Atwood    78 Comments

Revisiting Solid State Hard Drives

September 15, 2010

It's been almost a year since I covered The State of Solid State Hard Drives. Not a heck of a lot has changed, but the topic is still worth revisiting, because if you care at all about how your computer performs, solid state hard drives remain a life changing experience. Here's why:

  1. A solid state hard drive is easily the best and most obvious performance upgrade you can make on any computer for a given amount of money. Unless your computer is absolute crap to start with.
  2. The practical minimum solid state hard drive size, 128 GB, has modestly declined in price -- from about $350 to about $250.

(yes, you can get by with 64 GB, but at least with my Windows installs I find that I have to think about disk space with 64 GB, whereas with 128 GB I don't have to worry -- ever. Don't make me think, man!)

The rest of the components inside your PC are downright boring in comparison. CPUs? All stupid fast at any price, with more cores and gigahertz than you'll ever need unless you're one of those freaks who does nothing but raytracing and video transcoding all day long. Memory? Dirt cheap, and average users won't need more than 2 gigabytes of the stuff in practical use, which at the current going rate for DDR3 is less than 50 bucks.

Thanks to the neverending march of Moore's Law, PCs are becoming speedy at any price these days. It's difficult for me to muster any enthusiasm for the latest Intel CPU updates when I spend almost zero real world time waiting for the CPU to do anything on my computer. I guess it's true: absolute power corrupts absolutely.

Absolute-power-corrupts-absolutely

But hard drives, now, there's where you can pay a bit more and see a groundbreaking, generational leap in performance worthy of that investment -- as long as you skip over the old and busted spinning rust hard drives, and choose a newfangled solid state hard drive.

The current king of the hill seems to be the Crucial RealSSD C300.

RealSSD C300

Pretty sexy, right? Oh, who am I kidding, it's a boring slab of aluminum and silicon. But like all truly sexy things, what turns me on is the part I can't see -- the sexy, sexy, sexy performance inside this baby.

Tr-drivebench

See those bars dragging down the bottom of this graph? All spinning rust. Heck, even the SSD I recommended last year is only middle of the pack here. AnandTech concurs -- the Crucial RealSSD C300 is top dog, at least for now.

(Be careful, though, that your operating system supports the SSD TRIM command, otherwise you'll suffer severe performance degradation over time with almost any SSD. Operating systems earlier than Windows 7 and the latest, greatest Linux kernel should beware -- and, shockingly, OSX still doesn't support TRIM!)

Where it gets trickier, though, is when you need more than 128 GB of storage, or when you are limited to one 2.5" hard drive -- like in a laptop. In that case, ideally you'd like something that has the speed of a solid state hard drive, but the capacity (and economical price per gigabyte) of a traditional magnetic platter hard drive. You might say … a hybrid hard drive, the kind I was dreaming about back in 2006

After all this analysis, it's clear to me that traditional hard drives and flash memory are quite complimentary; they're strong in different areas. But flash drives are the future. They will definitely replace hard drives in almost all low end and low power devices-- and future high performance hard drives will need to have a substantial chunk of flash memory on board to stay competitive.

I had always been disappointed that hybrid hard drives, drives that combine both flash memory and traditional magnetic platters, never came to fruition. It was either traditional or SSD and nothing in between. It seemed like such an obvious "best of both worlds" scenario to me. But I recently discovered that decent hybrid drives do finally exist -- though in a small and mostly unheralded way.

Seagate's Momentus XT takes a totally respectable 2.5", 7200 RPM drive with a 32 megabyte buffer and combines it with 4 gigabytes of flash memory. The result is the exactly what I had always hoped:

Seagate's Momentus XT should become the standard hard drive in any notebook shipped. The biggest problem I have with using any brand new machine, regardless of how fast it is, is that it never feels fast because it usually has a HDD and not an SSD. While the Momentus XT isn't quite as fast as an SSD, it's a significant improvement over the mechanical drives found in notebooks today.

In many cases the Momentus XT performs like a VelociRaptor, but in a lower power, quieter package. The impact of adding just a small amount of SLC NAND is tremendous. The potential for hybrid drives continues to be huge; what Seagate has shown here is that with a minimal amount of NAND you can achieve some tremendous performance gains.

And the best part? 500 gigabytes of near-SSD performance for $130! Or, if that's too spendy, how about 320 gigabytes of near-SSD performance for $99?

I've ordered a few of these drives to upgrade my laptops and home theater PC. Sure, I'll invest in a SSD for my beastly desktop, but I can't justify $300 to put a SSD in a laptop I spent all of $800 on, or a home theater PC that set me back a mere $500. But a hundred bucks for near-SSD performance and decent capacity? Sign me up. And hard drive vendors: although I love SSDs to death, please keep these improved hybrid drives coming, too, please!

Posted by Jeff Atwood    65 Comments

Go That Way, Really Fast

September 12, 2010

When it comes to running Stack Overflow, the company, I take all my business advice from one person, and one person alone: Curtis Armstrong.

Curtis Armstrong

More specifically, Curtis Armstrong as Charles De Mar from the 1985 absurdist teen comedy classic, Better Off Dead. When asked for advice on how to ski down a particularly treacherous mountain, he replied:

Go that way, really fast. If something gets in your way... turn.

Go that way, really fast. If something gets in your way … turn.

(I recommend watching the video clip, or, better yet, the entire movie. It's brilliant in ways I can't possibly explain here.)

In the five months since we announced our funding, we have …

… and honestly, I'm a little worried we're still not going fast enough.

There are any number of Stack Overflow engine clones out there already, and I say more power to 'em. I'm proud to have something worth copying. If we do nothing else except help lead the world away from the ancient, creaky, horribly broken bulletin board model of phpBB and vBulletin -- attempting to get information out of those things is like panning for gold in a neverending river of sewage -- then that is more than I could have ever hoped for.

It is our stated goal as a company to live in harmony with the web, by only doing things that we believe make the internet better, at least in some small way. No, seriously. It's in writing and everything, I swear! We're not here to subvert or own anyone or anything. We just love community, and we love getting great answers to our questions. So if something gets in our way while doing that, well, we're not gonna fight you. We'll just turn. And keep going forward, really fast. Which is why those clones better move quick if they want to keep up with us.

While I like to think that having Charles De Mar as a business advisor is unique to our company, the idea that speed is important is hardly original to us. For example, certain Google projects also appear to understand Boyd's Law of Iteration.

Boyd decided that the primary determinant to winning dogfights was not observing, orienting, planning, or acting better. The primary determinant to winning dogfights was observing, orienting, planning, and acting faster. In other words, how quickly one could iterate. Speed of iteration, Boyd suggested, beats quality of iteration.

Speed of iteration -- the Google Chrome project has it.

1.0December 11, 2008
2.0May 24, 2009
3.0October 12, 2009
4.0January 25, 2010
5.0May 25, 2010
6.0September 2, 2010

Chrome was a completely respectable browser in V1 and V2. The entire project has moved forward so fast that it now is, at least in my humble opinion, the best browser on the planet. Google went from nothing, no web browser at all, to best-of-breed in under two years. Meanwhile, Internet Explorer took longer than the entire development period of Chrome to go from version 7 to version 8. And by the time Internet Explorer 9 ships -- even though it's actually looking like Microsoft's best, most competent technical upgrade of the browser yet -- it will be completely outclassed at launch by both Firefox and Chrome.

The Google Android project is another example. Android doesn't have to be better than the iPhone (and it most definitely isn't; it's been mediocre at best until recent versions). They just need to be faster at improving. Google is pushing out Froyos and Gingerbreads and Honeycombs with incredible, breakneck speed. Yes, Apple has indisputably better taste -- and an impeccably controlled experience. But at their current rate of progress, they'll be playing second or third fiddle to Google in the mobile space inside a few years. It's inevitable.

So, until further notice, we'll be following the same strategy as the Android and Chrome teams. We're going to go that way, really fast. And if something gets in our way, we'll turn.

Posted by Jeff Atwood    40 Comments

Vampires (Programmers) versus Werewolves (Sysadmins)

August 27, 2010

Kyle Brandt, a system administrator, asks Should Developers have Access to Production?

A question that comes up again and again in web development companies is:

"Should the developers have access to the production environment, and if they do, to what extent?"

My view on this is that as a whole they should have limited access to production. A little disclaimer before I attempt to justify this view is that this standpoint is in no way based on the perceived quality or attitude of the developers -- so please don't take it this way.

This is a tricky one for me to answer, because, well, I'm a developer. More specifically, I'm one of the developers Kyle is referring to. How do I know that? Because Kyle works for our company, Stack Overflow Internet Services Incorporated©®™. And Kyle is a great system administrator. How do I know that? Two reasons:

  1. He's one of the top Server Fault users.
  2. He had the audacity to write about this issue on the Server Fault blog.

From my perspective, the whole point of the company is to talk about what we're doing. Getting things done is important, of course, but we have to stop occasionally to write up what we're doing, how we're doing it, and why we're even doing it in the first place -- including all our doubts and misgivings and concerns. If we don't, we're cheating ourselves, and you guys, out of something much deeper. Yes, writing about what we're doing and explaining it to the community helps us focus. It lets our peers give us feedback. But most importantly of all, it lets anyone have the opportunity to learn from our many, many mistakes … and who knows, perhaps even the occasional success.

That's basically the entire philosophy behind our Stack Exchange Q&A network, too. Let's all talk about this stuff in public, so that we can teach each other how to get better at whatever the heck it is we love to do.

(Sometimes I get the feeling this idea makes my co-founder nervous, which I continually struggle to understand. If we don't walk the walk, why are we even doing this? But I digress.)

The saga of System Administrators versus Programmers is not a new one; I don't think I've ever worked at any company where these two factions weren't continually battling with each other in some form. It's truly an epic struggle, but to understand it, you have to appreciate that both System Administrators and Programmers have different, and perhaps complementary, supernatural powers.

Programmers are like vampires. They're frequently up all night, paler than death itself, and generally afraid of being exposed to daylight. Oh yes, and they tend think of themselves (or at least their code) as immortal.

Bela-lugosi-dracula

System Administrators are like werewolves. They may look outwardly ordinary, but are incredibly strong, mostly invulnerable to stuff that would kill regular people -- and prone to strange transformations during a moon "outage".

Wolfman

Let me be very clear that just as Kyle respects programmers, I have a deep respect for system administrators:

Although there is certainly some crossover, we believe that the programming community and the IT/sysadmin community are different beasts. Just because you're a hotshot programmer doesn't mean you have mastered networking and server configuration. And I've met a few sysadmins who could script circles around my code. That's why Server Fault gets its own domain, user profiles, and reputation system.

Different "beasts" indeed.

Anyway, if you're looking for a one size fits all answer to the question of how much access programmers should have to production environments, I'm sorry, I can't give you one. Every company is different, every team is different. I know, it's a sucky answer, but it depends.

However, as anyone who has watched the latest season of True Blood (or, God help us all, the Twilight Eclipse movie) can attest, there are ways for vampires and werewolves to work together. In a healthy team, everyone feels their abilities are being used and not squandered.

On our team, we're all fair-to-middling sysadmins. But there are a million things to do, and having a professional sysadmin means we can focus on the programming while the networking, hardware, and operational stuff gets a whole lot more TLC and far better (read: non-hacky) processes put in place. We're happy to refocus our efforts on what we're expert at, and let Kyle put his skills to work in areas that he's expert at. Now, that said, we don't want to cede full access to the production servers -- but there's a happy middle ground where our access becomes infrequent and minor over time, except in the hopefully rare event of an all hands on deck emergency.

The art of managing vampires and werewolves, I think, is to ensure that they spend their time not fighting amongst themselves, but instead, using those supernatural powers together to achieve a common goal they could not otherwise. In my experience, when programmers and system administrators fight, it's because they're bored. You haven't given them a sufficiently daunting task, one that requires the full combined use of their unique skills to achieve.

Remember, it's not vampires versus werewolves. It's vampires and werewolves.

Posted by Jeff Atwood    56 Comments

What's On Your Utility Belt?

August 13, 2010

Like any self-respecting geek, I'm mostly an indoor enthusiast.

But on those unfortunate occasions when I am compelled -- for reasons entirely beyond my control -- to leave the house, I do so fully armed with my crucial utility belt items. Yes, you heard me, I transform from the geeky Bruce Wayne to the gosh-darned Batman!

Batman-utility-belt

At least, that's how I like to think of it.

I've been talking about this every-day carry stuff for quite a while now. The 2010 edition of my personal utility belt is mostly subtle tweaks, but I daresay it's the best one yet.

Atwood-keychain-2010

The art of every-day carry must go on. What you see here is the contents of my pocket:

  1. Patriot 32 GB USB flash drive

    Now you can have a whole freakin' hard drive worth of files in your pocket. Just in case, you know, you have an emergency need to upload a virus to an alien mothership, or something. Beware the many cheap, slow USB flash drives out there; this one is a real gem. It's inexpensive, and per my measurements, about as fast as they get. This is important because the larger the flash drive, the more important speed becomes. Hard to believe I've gone from carrying a 512 megabyte flash drive in 2005 to a 32 gigabyte flashdrive in 2010.

  2. Leatherman Squirt P4

    Ounce for ounce, nothing beats the utility of the Leatherman Squirt. This time I opted for the plier (P) version instead of the scissors (S), and after seeing how much more generally useful the pliers are, I am now a little ashamed to admit I ever carried the wussy scissors version. Pliers all the way, baby. And yes, that is a Pulp Fiction joke you see on it.

  3. Fenix mini AAA LED flashlight model LD01R2

    Since I've been carrying them in 2005, the average LED flashlight has gone from bright, to very bright, to amazingly bright, to ridiculously blinding laser-like bright. It's scary how bright these fancy milled aluminum AAA LED flashlights get now. What I like about this one is that it lets you trade off stupid-brightness for something practical, like greater runtime: you can twist the top to switch levels: 9 Lumens for 11 hours, 28 Lumens for 3.5 hours, or 85 Lumens for 1 hour.

  4. Small Nite Ize s-biner

    These little nite-ize carabiners are awesome for quick attachment and detachment of your EDC items, but I'll warn you: resist the urge to put everything you carry on a carabiner, because if you do, the weight and "jangliness" goes up a lot -- and this way lies madness. Consider how many items you actually remove from your keychain regularly. For me, the only item I frequently removed to work with was the Squirt, so that's the only one I put on a carabiner.

Rest assured, everything here is carefully selected with the appropriate levels of monomaniacal attention to detail. For this weight and size, I don't think you can do better. (And don't think I've forgotten about optimizing my wallet, either. Oh no. Quite the contrary.)

However, I have to add a special category this year for the other must-have EDC utility belt item: the smartphone. What self-respecting superhero would leave the house these days without their smartphone? I'm not religious about it, but I use and rather like the iPhone 4, and I'm continually amazed how many things it does that I used to carry separate items for:

  • cell phone (obviously)
  • "Nintendo DSwhatever" for portable gaming
  • GPS
  • point and shoot digital camera
  • near-desktop quality mobile web browser and email client
  • mp3 player with speakers
  • audio and hi-def video recorder
  • DVD player
  • ebook reader
  • watch, alarm
  • emergency flashlight (via front facing LED flash control)
  • scanner
  • level and ruler

Smartphones really are the ultimate gadget. The list of functions is already enormous, and I'm sure I'm leaving out a few other things that you can do with a modern smartphone.

In a pinch, I could conceivably drop the AAA LED flashlight and the USB flash drive from my EDC kit and substitute the smartphone. Not exactly, mind you, but it's getting closer every year. At this rate, Apple could introduce a flip-out blade on the iPhone 7 and reduce my entire EDC kit to one item.

Anyway, that's what's on my utility belt in 2010. What's on yours?

Posted by Jeff Atwood    82 Comments

Groundhog Day, or, the Problem with A/B Testing

July 20, 2010

On a recent airplane flight, I happened to catch the movie Groundhog Day. Again.

Groundhog-day-movie-bill-murray

If you aren't familiar with this classic film, the premise is simple: Bill Murray, somehow, gets stuck reliving the same day over and over.

It's been at least 5 years since I've seen Groundhog Day. I don't know if it's my advanced age, or what, but it really struck me on this particular viewing: this is no comedy. There's a veneer of broad comedy, yes, but lurking just under that veneer is a deep, dark existential conundrum.

It might be amusing to relive the same day a few times, maybe even a few dozen times. But an entire year of the same day -- an entire decade of the same day -- everything happening in precisely, exactly the same way? My back of the envelope calculation easily ran to a decade. But I was wrong. The director, Harold Ramis thinks it was actually 30 or 40 years.

I think the 10-year estimate is too short. It takes at least 10 years to get good at anything, and alloting for the down time and misguided years [Phil] spent, it had to be more like 30 or 40 years [spent reliving the same day].

We only see bits and pieces of the full experience in the movie, but this time my mind began filling in the gaps. Repeating the same day for decades plays to our secret collective fear that our lives are irrelevant and ultimately pointless. None of our actions -- even suicide, in endless grisly permutations -- ever change anything. What's the point? Why bother? How many of us are trapped in here, and how can we escape?

This is some dark, scary stuff when you really think about it.

You want a prediction about the weather, you're asking the wrong Phil.

I'll give you a winter prediction.
It's gonna be cold,
it's gonna be gray,
and it's gonna last you for the rest of your life.

Comedy, my ass. I wanted to cry.

But there is a way out: redemption through repetition. If you have to watch Groundhog Day a few times to appreciate it, you're not alone. Indeed, that seems to be the whole point. Just ask Roger Ebert:

"Groundhog Day" is a film that finds its note and purpose so precisely that its genius may not be immediately noticeable. It unfolds so inevitably, is so entertaining, so apparently effortless, that you have to stand back and slap yourself before you see how good it really is.

Certainly I underrated it in my original review; I enjoyed it so easily that I was seduced into cheerful moderation. But there are a few films, and this is one of them, that burrow into our memories and become reference points. When you find yourself needing the phrase This is like "Groundhog Day" to explain how you feel, a movie has accomplished something.

There's something delightfully Ouroboros about the epiphanies and layered revelations in repeated viewings of a movie that is itself about (nearly) endless repetition.

Which, naturally, brings me to A/B testing. That's what Phil spends most of those thirty years doing. He spends it pursuing a woman, technically, but it's how he does it that is interesting:

Rita: This whole day has just been one long setup.

Phil: It hasn't.

Rita: And I hate fudge!

Phil: [making a mental list] No white chocolate. No fudge.

Rita: What are you doing? Are you making some kind of list? Did you call my friends and ask what I like and what I don't like? Is this what love is for you?

Phil: This is real. This is love.

Rita: Stop saying that! You must be crazy.

Phil doesn't just go on one date with Rita, he goes on thousands of dates. During each date, he makes note of what she likes and responds to, and drops everything she doesn't. At the end he arrives at -- quite literally -- the perfect date. Everything that happens is the most ideal, most desirable version of all possible outcomes on that date on that particular day. Such are the luxuries afforded to a man repeating the same day forever.

Groundhog-day-phil-rita

This is the purest form of A/B testing imaginable. Given two choices, pick the one that "wins", and keep repeating this ad infinitum until you arrive at the ultimate, most scientifically desirable choice. Your marketing weasels would probably collapse in an ecstatic, religious fervor if they could achieve anything even remotely close to the level of perfect A/B testing depicted in Groundhog Day.

But at the end of this perfect date, something impossible happens: Rita rejects Phil.

Phil wasn't making these choices because he honestly believed in them. He was making these choices because he wanted a specific outcome -- winning over Rita -- and the experimental data told him which path he should take. Although the date was technically perfect, it didn't ring true to Rita, and that made all the difference.

That's the problem with A/B testing. It's empty. It has no feeling, no empathy, and at worst, it's dishonest. As my friend Nathan Bowers said:

A/B testing is like sandpaper. You can use it to smooth out details, but you can't actually create anything with it.

The next time you reach for A/B testing tools, remember what happened to Phil. You can achieve a shallow local maximum with A/B testing -- but you'll never win hearts and minds. If you, or anyone on your team, is still having trouble figuring that out, well, the solution is simple.

Just watch Groundhog Day again.

Posted by Jeff Atwood    75 Comments

Whatever Happened to Voice Recognition?

June 21, 2010

Remember that Scene in Star Trek IV where Scotty tried to use a Mac Plus?

Star-trek-4-apple-mac-plus

Using a mouse or keyboard to control a computer? Don't be silly. In the future, clearly there's only one way computers will be controlled: by speaking to them.

There's only one teeny-tiny problem with this magical future world of computers we control with our voices.

Voice-recognition-accuracy-rate-over-time

It doesn't work.

Despite ridiculous, order of magnitude increases in computing power over the last decade, we can't figure out how to get speech recognition accuracy above 80% -- when the baseline human voice transcription accuracy rate is anywhere from 96% to 98%!

In 2001 recognition accuracy topped out at 80%, far short of HAL-like levels of comprehension. Adding data or computing power made no difference. Researchers at Carnegie Mellon University checked again in 2006 and found the situation unchanged. With human discrimination as high as 98%, the unclosed gap left little basis for conversation. But sticking to a few topics, like numbers, helped. Saying “one” into the phone works about as well as pressing a button, approaching 100% accuracy. But loosen the vocabulary constraint and recognition begins to drift, turning to vertigo in the wide-open vastness of linguistic space.

As Robert Fortner explained in Rest in Peas: The Unrecognized Death of Speech Recognition, after all these years, we're desperately far away from any sort of universal speech recognition that's useful or practical.

Now, we do have to clarify that we're talking about universal recognition: saying anything to a computer, and having it reliably convert that into a valid, accurate text representation. When you constrain the voice input to a more limited vocabulary -- say, just numbers, or only the names that happen to be in your telephone's address book -- it's not unreasonable to expect a high level of accuracy. I tend to think of this as "voice control" rather than "voice recognition".

Still, I think we're avoiding the real question: is voice control, even hypothetically perfect voice control, more effective than the lower tech alternatives? In my experience, speech is one of the least effective, inefficient forms of communicating with other human beings. By that, I mean ...

  • typical spoken communication tends to be off-the-cuff and ad-hoc. Unless you're extremely disciplined, on average you will be unclear, rambling, and excessively verbose.
  • people tend to hear about half of what you say at any given time. If you're lucky.
  • spoken communication puts a highly disproportionate burden on the listener. Compare the time it takes to process a voicemail versus the time it takes to read an email.

I am by no means against talking with my fellow human beings. I have a very deep respect for those rare few who are great communicators in the challenging medium of conversational speech. Though we've all been trained literally from birth how to use our voices to communicate, voice communication remains filled with pitfalls and misunderstandings. Even in the best of conditions.

So why in the world -- outside of a disability -- would I want to extend the creaky, rickety old bridge of voice communication to controlling my computer? Isn't there a better way?

Robert's post contains some examples in the comments from voice control enthusiasts:

in addition to extremely accurate voice dictation, there are those really cool commands, like being able to say something like "search Google for Balloon Boy" or something like that and having it automatically open up your browser and enter the search term -- something like this is accomplished many times faster than a human could do it. Or, being able to total up a column of numbers in Microsoft Excel by saying simply "total this column" and seeing the results in a blink of an eye, literally.

That's funny, because I just fired up the Google app on my iPhone, said "balloon boy" into it, and got .. a search for "blue boy". I am not making this up. As for the Excel example, total which column? Let's assume you've dealt with the tricky problem of selecting what column you're talking about with only your voice. (I'm sorry, was it D5? B5?) Wouldn't it be many times faster to click the toolbar icon with your mouse, or press the keyboard command equivalent, to sum the column -- rather than methodically and tediously saying the words "sum this column" out loud?

I'm also trying to imagine a room full of people controlling their computers or phones using their voices. It's difficult enough to get work done in today's chatty work environments without the added burden of a floor full of people saying "zoom ... enhance" to their computers all day long. Wouldn't we all end up hoarse and deaf?

Let's look at another practical example -- YouTube's automatic speech recognition feature. I clicked through to the first UC Berkeley video with this feature, clicked the CC (closed caption) icon, and immediately got .. this.

Uc-berkeley-physics-lecture

"Light exerts force on matter". But according to Google's automatic speech recognition, it's "like the search for some matter". Unsurprisingly, it does not get better from there. You'd be way more confused than educated if you had to learn this lecture from the automatic transcription.

Back when Joel Spolsky and I had a podcast together, a helpful listener suggested using speech recognition to get a basic podcast transcript going. Everything I knew about voice recognition told me this wouldn't help, but harm. What's worse: transcribing everything by hand, from scratch -- or correcting every third or fourth word in an auto-generated machine transcript? Maybe it's just me, but the friction of the huge error rate inherent in the machine transcript seems far more intimidating than a blank slate human transcription. The humans may not be particularly efficient, but they all add value along the way -- collective human judgment can editorially improve the transcript, by removing all the duplication, repetition, and "ums" of a literal, by-the-book transcription.

In 2004, Mike Bliss composed a poem about voice recognition. He then read it to voice recognition software on his PC, and rewrote it as recognized.

a poem by Mike Bliss

like a baby, it listens
it can't discriminate
it tries to understand
it reflects what it thinks you say
it gets it wrong... sometimes
sometimes it gets it right.
One day it will grow up,
like a baby, it has potential
will it go to work?
will it turn to crime?
you look at it indulgently.
you can't help loving it, can you?
a poem by like myth

like a baby, it nuisance
it can't discriminate
it tries to oven
it reflects lot it things you say
it gets it run sometimes
sometimes it gets it right
won't day it will grow bop
Ninth a baby, it has provincial
will it both to look?
will it the two crime?
you move at it inevitably
you can't help loving it, cannot you?

The real punchline here is that Mike re-ran the experiment in 2008, and after 5 minutes of voice training, the voice recognition got all but 2 words of the original poem correct!

I suspect that's still not good enough in the face of the existing simpler alternatives. Remember handwriting recognition? It was all the rage in the era of the Apple Newton.

Doonesbury-newton

It wasn't as bad as Doonesbury made it out to be. I learned Palm's Graffiti handwriting recognition language and got fairly proficient with it. More than ten years later, you'd expect to see massively improved handwriting recognition of some sort in today's iPads and iPhones and iOthers, right? Well, maybe, if by "massively improved" you mean "nonexistent".

While it still surely has its niche uses, I personally don't miss handwriting recognition. Not even a little. And I can't help wondering if voice recognition will go the same way.

Posted by Jeff Atwood    122 Comments

The Vast and Endless Sea

June 1, 2010

After we created Stack Overflow, some people were convinced we had built a marginally better mousetrap for asking and answering questions. The inevitable speculation began: can we use your engine to build a Q&A site about {topic}? Our answer was Stack Exchange. Pay us $129 a month (and up), and you too can create a hosted Q&A community on our engine – for whatever topic you like!

Well, I have a confession to make: my heart was never in Stack Exchange. It was a parallel effort in a parallel universe only tangentially related to my own. There's a whole host of reasons why, but if I had to summarize it in a sentence, I'd say that money is poisonous to communities. That $129/month doesn't sound like much – and it isn't – but the commercial nature of the enterprise permeated and distorted everything from the get-go.

(fortunately, the model is changing with Stack Exchange 2.0, but that's a topic for another blog post.)

Yes, Stack Overflow Internet Services Incorporated©®™ is technically a business, even a venture capital backed business now – but I didn't co-found it because I wanted to make money. I co-founded it because I wanted to build something cool that made the internet better. Yes, selfishly for myself, of course, but also in conjunction with all of my fellow programmers, because I know none of us is as dumb as all of us.

Nobody is participating in Stack Overflow to make money. We're participating in Stack Overflow because …

  • We love programming
  • We want to leave breadcrumb trails for other programmers to follow so they can avoid making the same dumb mistakes we did
  • Teaching peers is one of the best ways to develop mastery
  • We can follow our own interests wherever they lead
  • We want to collectively build something great for the community with our tiny slices of effort

I don't care how much you pay me, you'll never be able to recreate the incredibly satisfying feeling I get when demonstrating mastery within my community of peers. That's what we do on Stack Overflow: have fun, while making the internet one infinitesimally tiny bit better every day.

So is it any wonder that some claim Stack Overflow is more satisfying than their real jobs? Not to me.

If this all seems like a bunch of communist hippie bullcrap to you, I understand. It's hard to explain. But there is quite a bit of science documenting these strange motivations. Let's start with Dan Pink's 2009 TED talk.

Dan's talk centers on the candle problem. Given the following three items …

  1. A candle
  2. A box of thumbtacks
  3. A book of matches

… how can you attach the candle to the wall?

It's not a very interesting problem on its own – that is, until you try to incentivize teams to solve it:

Now I want to tell you about an experiment using the candle problem by a scientist from Princeton named Sam Glucksberg. Here's what he did.

To the first group, he said, "I'm going to time you to establish norms, averages for how long it typically takes someone to solve this sort of problem."

To the second group, he said, "If you're in the top 25 percent of the fastest times you get five dollars. If you're the fastest of everyone we're testing here today you get 20 dollars." (This was many years ago. Adjusted for inflation, it's a decent sum of money for a few minutes of work.)

Question: How much faster did this group solve the problem?

Answer: It took them, on average, three and a half minutes longer. Three and a half minutes longer. Now this makes no sense, right? I mean, I'm an American. I believe in free markets. That's not how it's supposed to work. If you want people to perform better, you reward them. Give them bonuses, commissions, their own reality show. Incentivize them. That's how business works. But that's not happening here. You've got a monetary incentive designed to sharpen thinking and accelerate creativity – and it does just the opposite. It dulls thinking and blocks creativity.

It turns out that traditional carrot-and-stick incentives are only useful for repetitive, mechanical tasks. The minute you have to do anything even slightly complex that requires even a little problem solving without a clear solution or rules – those incentives not only don't work, they make things worse!

Pink eventually wrote a book about this, Drive: The Surprising Truth About What Motivates Us.

Drive by Daniel H. Pink

There's no need to read the book; this clever ten minute whiteboard animation will walk you through the main points. If you view only one video today, view this one.

The concept of intrinsic motivation may not be a new one, but I find that very few companies are brave enough to actually implement them.

I've tried mightily to live up to the ideals that Stack Overflow was founded on when building out my team. I don't care when you come to work or what your schedule is. I don't care where in the world you live (provided you have a great internet connection). I don't care how you do the work. I'm not going to micromanage you and assign you a queue of task items. There's no need.

If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.
Antoine de Saint-Exupéry

Because I know you yearn for the vast and endless sea, just like we do.

Posted by Jeff Atwood    85 Comments

On Working Remotely

May 6, 2010

When I first chose my own adventure, I didn't know what working remotely from home was going to be like. I had never done it before. As programmers go, I'm fairly social. Which still means I'm a borderline sociopath by normal standards. All the same, I was worried that I'd go stir-crazy with no division between my work life and my home life.

Well, I haven't gone stir-crazy yet. I think. But in building Stack Overflow, I have learned a few things about what it means to work remotely -- at least when it comes to programming. Our current team encompasses 5 people, distributed all over the USA, along with the team in NYC.

Usa-stack-overflow-team-map

My first mistake was attempting to program alone. I had weekly calls with my business partner, Joel Spolsky, which were quite productive in terms of figuring out what it was we were trying to do together -- but he wasn't writing code. I was coding alone. Really alone. One guy working all by yourself alone. This didn't work at all for me. I was unmoored, directionless, suffering from analysis paralysis, and barely able to get motivated enough to write even a few lines of code. I rapidly realized that I'd made a huge mistake in not having a coding buddy to work with.

That situation rectified itself soon enough, as I was fortunate enough to find one of my favorite old coding buddies was available. Even though Jarrod was in North Carolina and I was in California, the shared source code was the mutual glue that stuck us together, motivated us, and kept us moving forward. To be fair, we also had the considerable advantage of prior history, because we had worked together at a previous job. But the minimum bar to work remotely is to find someone who loves code as much as you do. It's … enough. Anything else on top of that -- old friendships, new friendships, a good working relationship -- is icing that makes working together all the sweeter. I eventually expanded the team in the same way by adding another old coding buddy, Geoff, who lives in Oregon. And again by adding Kevin, who I didn't know, but had built amazing stuff for us without even being asked to, from Texas. And again by adding Robert, in Florida, who I also didn't know, but spent so much time on every single part of our sites that I felt he had been running alongside our team the whole way, there all along.

The reason remote development worked for us, in retrospect, wasn't just shared love of code. I picked developers who I knew -- I had incontrovertible proof -- were amazing programmers. I'm not saying they're perfect, far from it, merely that they were top programmers by any metric you'd care to measure. That's why they were able to work remotely. Newbie programmers, or competent programmers who are phoning it in, are absolutely not going to have the moxie necessary to get things done remotely -- at least, not without a pointy haired manager, or grumpy old team lead, breathing down their neck. Don't even think about working remotely with anyone who doesn't freakin' bleed ones and zeros, and has a proven track record of getting things done.

While Joel certainly had a lot of high level input into what Stack Overflow eventually became, I only talked to him once a week, at best (these calls were the genesis of our weekly podcast series). I had a strong, clear vision of what I wanted Stack Overflow to be, and how I wanted it to work. Whenever there was a question about functionality or implementation, my team was able to rally around me and collectively make decisions we liked, and that I personally felt were in tune with this vision. And if you know me at all, you know I'm not shy about saying no, either. We were able to build exactly what we wanted, exactly how we wanted.

Bottom line, we were on a mission from God. And we still are.

So, there are a few basic ground rules for remote development, at least as I've seen it work:

  • The minimum remote team size is two. Always have a buddy, even if your buddy is on another continent halfway across the world.
  • Only grizzled veterans who absolutely love to code need apply for remote development positions. Mentoring of newbies or casual programmers simply doesn't work at all remotely.
  • To be effective, remote teams need full autonomy and a leader (PM, if you will) who has a strong vision and the power to fully execute on that vision.

This is all well and good when you have a remote team size of three, as we did for the bulk of Stack Overflow development. And all in the same country. Now we need to grow the company, and I'd like to grow it in distributed fashion, by hiring other amazing developers from around the world, many of whom I have met through Stack Overflow itself.

But how do you scale remote development? Joel had some deep seated concerns about this, so I tapped one of my heroes, Miguel de Icaza -- who I'm proud to note is on our all-star board of advisors -- and he was generous enough to give us some personal advice based on his experience running the Mono project, which has dozens of developers distributed all over the world.

Time-zone-differences

At the risk of summarizing mercilessly (and perhaps too much), I'll boil down Miguel's advice the best I can. There are three tools you'll need in place if you plan to grow a large-ish and still functional remote team:

  1. Real time chat

    When your team member lives in Brazil, you can't exactly walk by his desk to ask him a quick question, or bug him about something in his recent checkin. Nope. You need a way to casually ping your fellow remote team members and get a response back quickly. This should be low friction and available to all remote developers at all times. IM, IRC, some web based tool, laser beams, smoke signals, carrier pigeon, two tin cans and a string: whatever. As long as everyone really uses it.

    We're currently experimenting with Campfire, but whatever floats your boat and you can get your team to consistently use, will work. Chat is the most essential and omnipresent form of communication you have when working remotely, so you need to make absolutely sure it's functioning before going any further.

  2. Persistent mailing list

    Sure, your remote team may know the details of their project, but what about all the other work going on? How do they find out about that stuff or even know it exists in the first place? You need a virtual bulletin board: a place for announcements, weekly team reports, and meeting summaries. This is where a classic old-school mailing list comes in handy.

    We're using Google Groups and although it's old school in spades, it works plenty well for this. You can get the emails as they arrive, or view the archived list via the web interface. One word of caution, however. Every time you see something arrive in your inbox from the mailing list you better believe, in your heart of hearts, that it contains useful information. The minute the mailing list becomes just another "whenever I have time to read that stuff", noise engine, or distraction from work … you've let someone cry wolf too much, and ruined it. So be very careful. Noisy, argumentative, or useless things posted to the mailing list should be punishable by death. Or noogies.

  3. Voice and video chat

    As much as I love ASCII, sometimes faceless ASCII characters just aren't enough to capture the full intentions and feelings of the human being behind them. When you find yourself sending kilobytes of ASCII back and forth, and still are unsatisfied that you're communicating, you should instill a reflexive habit of "going voice" on your team.

    Never underestimate the power of actually talking to another human being. I know, I know, the whole reason we got into this programming thing was to avoid talking to other people, but bear with me here. You can't be face to face on a remote team without flying 6 plus hours, and who the heck has that kind of time? I've got work I need to get done! Well, the next best thing to hopping on a plane is to fire up Skype and have a little voice chat. Easy peasy. All that human nuance which is totally lost in faceless ASCII characters (yes, even with our old pal *<:-)) will come roaring back if you regularly schedule voice chats. I recommend at least once a week at an absolute minimum; they don't have to be long meetings, but it sure helps in understanding the human being behind all those awesome checkins.

Nobody hates meetings and process claptrap more than I do, but there is a certain amount of process you'll need to keep a bunch of loosely connected remote teams and developers in sync.

  1. Monday team status reports

    Every Monday, as in somebody's-got-a-case-of-the, each team should produce a brief, summarized rundown of:

    • What we did last week
    • What we're planning to do this week
    • Anything that is blocking us or we are concerned about

    This doesn't have to be (and in fact shouldn't be) a long report. The briefer the better, but do try to capture all the useful highlights. Mail this to the mailing list every Monday like clockwork. Now, how many "teams" you have is up to you; I don't think this needs to be done at the individual developer level, but you could.

  2. Meeting minutes

    Any time you conduct what you would consider to be a "meeting" with someone else, take minutes! That is, write down what happened in bullet point form, so those remote team members who couldn't be there can benefit from -- or at least hear about -- whatever happened.

    Again, this doesn't have to be long, and if you find taking meeting minutes onerous then you're probably doing it wrong. A simple bulleted list of sentences should suffice. We don't need to know every little detail, just the big picture stuff: who was there? What topics were discussed? What decisions were made? What are the next steps?

Both of the above should, of course, be mailed out to the mailing list as they are completed so everyone can be notified. You do have a mailing list, right? Of course you do!

If this seems like a lot of jibba-jabba, well, that's because remote development is hard. It takes discipline to make it all work, certainly more discipline than piling a bunch of programmers into the same cubicle farm. But when you imagine what this kind of intellectual work -- not just programming, but anything where you're working in mostly thought-stuff -- will be like in ten, twenty, even thirty years … don't you think it will look a lot like what happens every day right now on Stack Overflow? That is, a programmer in Brazil helping a programmer in New Jersey solve a problem?

If I have learned anything from Stack Overflow it is that the world of programming is truly global. I am honored to meet these brilliant programmers from every corner of the world, even if only in a small way through a website. Nothing is more exciting for me than the prospect of adding international members to the Stack Overflow team. The development of Stack Overflow should be reflective of what Stack Overflow is: an international effort of like-minded -- and dare I say totally awesome -- programmers. I wish I could hire each and every one of you. OK, maybe I'm a little biased. But to me, that's how awesome the Stack Overflow community is.

I believe remote development represents the future of work. If we have to spend a little time figuring out how this stuff works, and maybe even make some mistakes along the way, it's worth it. As far as I'm concerned, the future is now. Why wait?

Posted by Jeff Atwood    78 Comments

What's Wrong With CSS

April 30, 2010

We're currently in the midst of a CSS Zen Garden type excerise on our family of Q&A websites, which I affectionately refer to as "the Trilogy":

(In case you were wondering, yes, meta is the Star Wars Holiday Special.)

These sites all run the same core engine, but the logo, domain, and CSS "skin" that lies over the HTML skeleton is different in each case:

serverfault.com screenshot superuser.com screenshot
meta.stackoverflow screenshot stackoverflow.com screenshot

They are not terribly different looking, it's true, but we also want them to be recognizable as a family of sites.

We're working with two amazing designers, Jin Yang and Nathan Bowers, who are helping us whip the CSS and HTML into shape so they can produce a set of about 10 different Zen Garden designs. As new sites in our network get democracied into being, these designs will be used as a palette for the community to choose from. (And, later, the community will decide on a domain name and logo as well.)

Anyway, I bring this up not because my pokemans, let me show you them, but because I have to personally maintain four different CSS files. And that number is only going to get larger. Much larger. That scares me a little.

Most of all, what I've learned from this exercise in site theming is that CSS is kind of painful. I fully support CSS as a (mostly) functional user interface Model-View-Controller. But even if you have extreme HTML hygiene and Austrian levels of discipline, CSS has some serious limitations in practice.

Things in particular that bite us a lot:

  • Vertical alignment is a giant, hacky PITA. (Tables work great for this though!)
  • Lack of variables so we have to repeat colors all over the place.
  • Lack of nesting so we have to repeat huge blocks of CSS all over the place.

In short, CSS violates the living crap out of the DRY principle. You are constantly and unavoidably repeating yourself.

That's why I'm so intrigued by two Ruby gems that attempt to directly address the deficiencies of CSS.

1. Less CSS

/* CSS */

#header {
  -moz-border-radius: 5;
  -webkit-border-radius: 5;
  border-radius: 5;
}

#footer {
  -moz-border-radius: 10;
  -webkit-border-radius: 10;
  border-radius: 10;
}
// LessCSS

.rounded_corners (@radius: 5px) {
  -moz-border-radius: @radius;
  -webkit-border-radius: @radius;
  border-radius: @radius;
}

#header {
  .rounded_corners;
}

#footer {
  .rounded_corners(10px);
}

2. SASS

/* CSS */

.content_navigation {
  border-color: #3bbfce;
  color: #2aaebd;
}

.border {
  padding: 8px;
  margin: 8px;
  border-color: #3bbfce;
}
// Sass

!blue = #3bbfce
!margin = 16px

.content_navigation
  border-color = !blue
  color = !blue - #111

.border
  padding = !margin / 2
  margin = !margin / 2
  border-color = !blue

As you can see, in both cases we're transmogrifying CSS into a bit more of a programming language, rather than the static set of layout rules it currently exists as. Behind the scenes, we're generating plain vanilla CSS using these little dynamic languages. This could be done at project build time, or even dynamically on every page load if you have a good caching strategy.

I'm not sure how many of these improvements CSS3 will bring, never mind when the bulk of browsers in the world will support it. But I definitely feel that the core changes identified in both Less CSS and SASS address very real pain points in practical CSS use. It's worth checking them out to understand why they exist, what they bring to the table, and how you could possibly adopt some of these strategies in your own CSS and your favorite programming language.

Posted by Jeff Atwood    89 Comments

So You'd Like to Send Some Email (Through Code)

April 21, 2010

I have what I would charitably describe as a hate-hate relationship with email. I desperately try to avoid sending email, not just for myself, but also in the code I write.

Despite my misgivings, email is the cockroach of communication mediums: you just can't kill it. Email is the one method of online contact that almost everyone -- at least for that subset of "everyone" which includes people who can bear to touch a computer at all -- is guaranteed to have, and use. Yes, you can make a fairly compelling case that email is for old stupid people, but let's table that discussion for now.

So, reluctantly, we come to the issue of sending email through code. It's easy! Let's send some email through oh, I don't know, let's say ... Ruby, courtesy of some sample code I found while browsing the Ruby tag on Stack Overflow.

require 'net/smtp'

def send_email(to, subject = "", body = "")
    from = "my@email.com"
    body= "From: #{from}\r\nTo: #{to}\r\nSubject: #{subject}\r\n\r\n#{body}\r\n"

    Net::SMTP.start('192.168.10.213', 25, '192.168.0.218') do |smtp|
        smtp.send_message body, from, to
    end
end

send_email "foo@example.com", "title", "body goes here"

There's a bug in this code, though. Do you see it?

Just because you send an email doesn't mean it will arrive. Not by a long shot. Bear in mind this is email we're talking about. It was never designed to survive a bitter onslaught of criminals and spam, not to mention the explosive, exponential growth it has seen over the last twenty years. Email is a well that has been truly and thoroughly poisoned -- the digital equivalent of a superfund cleanup site. The ecosystem around email is a dank miasma of half-implemented, incompletely supported anti-spam hacks and workarounds.

Which means the odds of that random email your code just sent getting to its specific destination is .. spotty. At best.

If you want email your code sends to actually arrive in someone's AOL mailbox, to the dulcet tones of "You've Got Mail!", there are a few things you must do first. And most of them are only peripherally related to writing code.

1. Make sure the computer sending the email has a Reverse PTR record

What's a reverse PTR record? It's something your ISP has to configure for you -- a way of verifying that the email you send from a particular IP address actually belongs to the domain it is purportedly from.

Not every IP address has a corresponding PTR record. In fact, if you took a random sampling of addresses your firewall blocked because they were up to no good, you'd probably find most have no PTR record - a dig -x gets you no information. That's also apt to be true for mail spammers, or their PTR doesn't match up: if you do a dig -x on their IP you get a result, but if you look up that result you might not get the same IP you started with.

That's why PTR records have become important. Originally, PTR records were just intended as a convenience, and perhaps as a way to be neat and complete. There still are no requirements that you have a PTR record or that it be accurate, but because of the abuse of the internet by spammers, certain conventions have grown up. For example, you may not be able to send email to some sites if you don't have a valid PTR record, or if your pointer is "generic".

How do you get a PTR record? You might think that this is done by your domain registrar - after all, they point your domain to an IP address. Or you might think whoever handles your DNS would do this. But the PTR record isn't up to them, it's up to the ISP that "owns" the IP block it came from. They are the ones who need to create the PTR record.

A reverse PTR record is critical. How critical? Don't even bother reading any further until you've verified that your ISP has correctly configured the reverse PTR record for the server that will be sending email. It is absolutely the most common check done by mail servers these days. Fail the reverse PTR check, and I guarantee that a huge percentage of the emails you send will end up in the great bit bucket in the sky -- and not in the email inboxes you intended.

2. Configure DomainKeys Identified Mail in your DNS and code

What's DomainKeys Identified Mail? With DKIM, you "sign" every email you send with your private key, a key only you could possibly know. And this can be verified by attempting to decrypt the email using the public key stored in your public DNS records. It's really quite clever!

The first thing you need to do is generate some public-private key pairs (one for every domain you want to send email from) via OpenSSL. I used a win32 version I found. Issue these commands to produce the keys in the below files:

$ openssl genrsa -out rsa.private 1024
$ openssl rsa -in rsa.private -out rsa.public -pubout -outform PEM

These public and private keys are just big ol' Base64 encoded strings, so plop them in your code as configuration string resources that you can retrieve later.

Next, add some DNS records. You'll need two new TXT records.

  1. _domainkey.example.com
    "o=~\; r=contact@example.com"
  2. selector._domainkey.example.com
    "k=rsa\; p={public-key-base64-string-here}"

The first TXT DNS record is the global DomainKeys policy and contact email.

The second TXT DNS record is the public base64 key you generated earlier, as one giant unbroken string. Note that the "selector" part of this record can be anything you want; it's basically just a disambiguating string.

Almost done. One last thing -- we need to sign our emails before sending them. In any rational world this would be handled by an email library of some kind. We use Mailbee.NET which makes this fairly painless:

smtp.Message = dk.Sign(smtp.Message, 
               null, AppSettings.Email.DomainKeyPrivate, false, "selector");

3. Set up a SenderID record in your DNS

To be honest, SenderID is a bit of a "nice to have" compared to the above two. But if you've gone this far, you might as well go the distance. SenderID, while a little antiquated and kind of.. Microsoft/Hotmail centric.. doesn't take much additional effort.

SenderID isn't complicated. It's another TXT DNS record at the root of, say, example.com, which contains a specially formatted string documenting all the allowed IP addresses that mail can be expected to come from. Here's an example:

"v=spf1 a mx ip4:10.0.0.1 ip4:10.0.0.2 ~all"

You can use the Sender ID SPF Record Wizard to generate one of these for each domain you send email from.

That sucked. How do I know all this junk is working?

I agree, it sucked. Email sucks; what did you expect? I used two methods to verify that all the above was working:

  1. Test emails sent to a GMail account.

    Use the "show original" menu on the arriving email to see the raw message content as seen by the email server. You want to verify that the headers definitely contain the following:

    Received-SPF: pass
    Authentication-Results: ... spf=pass ... dkim=pass
    

    If you see that, then the Reverse PTR and DKIM signing you set up is working. Google provides excellent diagnostic feedback in their email server headers, so if something isn't working, you can usually discover enough of a hint there to figure out why.

  2. Test emails sent to the Port25 email verifier

    Port25 offers a really nifty public service -- you can send email to check-auth@verifier.port25.com and it will reply to the from: address with an extensive diagnostic! Here's an example summary result from a test email I just sent to it:

    SPF check:          pass
    DomainKeys check:   fail
    DKIM check:         pass
    Sender-ID check:    pass
    SpamAssassin check: ham
    

    You want to pass SPF, DKIM, and Sender-ID. Don't worry about the DomainKeys failure, as I believe it is spurious -- DKIM is the "newer" version of that same protocol.

Yes, the above three steps are quite a bit of work just to send a lousy email. But I don't send email lightly. By the time I've reached the point where I am forced to write code to send out email, I really, really want those damn emails to arrive. By any means necessary.

And for those who are the unfortunate recipients of these emails: my condolences.

Posted by Jeff Atwood    66 Comments

Three Monitors For Every User

April 4, 2010

As far as I'm concerned, you can never be too rich, too thin, or have too much screen space. By "screen", I mean not just large monitors, but multiple large monitors. I've been evangelizing multiple monitors since the dark days of Windows Millennium Edition:

If you're a long time reader you're probably sick of hearing about this stuff by now, but something rather wonderful has happened since I last wrote about it:

If you're only using one monitor, you are cheating yourself out of potential productivity. Two monitors is a no-brainer. It's so fundamental that I included it as a part of the Programmer's Bill of Rights.

But you can do better.

As good as two monitors is, three monitors is even better. With three monitors, there's a "center" to focus on. And 50% more display area. While there's certainly a point of diminishing returns for additional monitors, I think three is the sweet spot. Even Edward Tufte, in the class I recently attended, explicitly mentioned multiple monitors. I don't care how large a single display can be; you can never have enough desktop space.

Normally, to achieve three monitors, you have to either:

  1. Buy an exotic video card that has more than 2 monitor connections.
  2. Install a second video card.

Fortunately, that is no longer true. I was excited to learn that the latest ATI video cards have gone from two to three video outputs. Which means you can now achieve triple monitors with a single video card upgrade! They call this "eyefinity", but it's really just shorthand for "raising the standard from two display outputs to three".

But, there is a (small) catch. The PC ecosystem is in the middle of shifting display output standards. For evidence of this, you need look no further than the back panel of one of these newfangled triple display capable ATI video cards:

Radeon-eyefinity-video-card-outputs

It contains:

  • two DVI outputs
  • one HDMI output
  • one DisplayPort output

I suspect part of this odd connector layout is due to space restrictions (DVI is awfully chunky), but I've always understood DisplayPort to be the new, improved DVI connector for computer monitors, and HDMI to be the new, improved s-video/component connector for televisions. Of course these worlds are blurring, as modern high-definition TVs make surprisingly effective computer monitors, too.

Anyway, since all my monitors have only DVI inputs, I wasn't sure what to do with the other output. So I asked on Super User. The helpful answers led me to discover that, as I suspected, the third output has to be DisplayPort. So to connect my third monitor, I needed to convert DisplayPort to DVI, and there are two ways:

  1. a passive, analog DisplayPort to DVI conversion cable for ~$30 that supports up to 1920x1200
  2. an active, digital DisplayPort to DVI converter for $110 that supports all resolutions

I ended up going with the active converter, which has mixed reviews, but it's worked well for me over the last few weeks.

Accell-ultraav-displayport-to-dvi-adapter

Note that this adapter requires USB power, and given the spotty results others have had with it, some theorize that it needs quite a bit of juice to work reliably. I plugged it into my system's nearby rear USB ports which do tend to deliver more power (they're closer to the power supply, and have short cable paths). Now, I have gotten the occasional very momentary black screen with it, but nothing severe enough to be a problem or frequent enough to become a pattern. If you have DisplayPort compatible monitors, of course, this whole conversion conundrum is a complete non-issue. But DisplayPort is fairly new, and even my new-ish LCD monitors don't support it yet.

The cool thing about this upgrade, besides feeding my video card addiction, is that I was able to simplify my hardware configuration. That's always good. I went from two video cards to one, which means less power consumption, simpler system configuration, and fewer overall driver oddities. Basically, it makes triple monitors -- dare I say it -- almost a mainstream desktop configuration. How could I not be excited about that?

I was also hoping that Nvidia would follow ATI's lead here and make three display outputs the standard for all their new video cards, too, but sadly that's not the case. It turns out their new GTX 480 fails in other ways, in that it's basically the Pentium 4 of video cards -- generating ridiculous amounts of heat for very little performance gain. Based on those two facts, I am comfortable endorsing ATI wholeheartedly at this point. But, do be careful, because not all ATI cards support triple display outputs (aka "eyefinity"). These are the ones that I know do:

Unless you're a gamer, there's no reason to care about anything other than the least expensive model here, which will handily crush any 2D or 3D desktop GUI acceleration needs you might have. As an addict, of course I bought the high end model and it absolutely did not disappoint -- more than doubling my framerates in the excellent game Battlefield: Bad Company 2 over the GTX 280 I had before.

I'm excited that a triple monitor setup is now, thanks to ATI, so easily attainable for desktop users -- as long as you're aware of the DisplayPort caveat I discussed above. I'd encourage anyone who is even remotely interested in the (many) productivity benefits of a triple monitor setup to seriously consider an ATI video card upgrade.

Posted by Jeff Atwood    98 Comments

Usability On The Cheap and Easy

March 31, 2010

Writing code? That's the easy part. Getting your application in the hands of users, and creating applications that people actually want to use -- now that's the hard stuff.

I've been a long time fan of Krug's book Don't Make Me Think. Not just because it's a quick, easy read (and it is!) -- but because it's the most concise and most approachable book I've ever found to teach the fundamental importance of usability. As far as I'm concerned, if you want to help us make the software industry a saner place, the first step is getting Don't Make Me Think in the hands of as many of your coworkers as you can. If you don't have people that care about usability on your project, your project is doomed.

Beyond getting people over the hurdle of at least paging through the Krug book, and perhaps begrudgingly conceding that this usability stuff matters, the next challenge is figuring out how to integrate usability testing into your project. It's easy to say "Usability is Important!", but you have to walk the walk, too. I touched on some low friction ways to get started in Low-Fi Usability Testing. That rough outline is now available in handy, more complete book form -- Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems.

Rocket-surgery-made-easy

Don't worry, Krug's book is just as usable as his advice. It's yet another quick, easy read. Take it from the man himself:

  • Usability testing is one of the best things people can do to improve Web sites (or almost anything they’re creating that people have to interact with).
  • Since most organizations can’t afford to hire someone to do testing for them on a regular basis, everyone should learn to do it themselves. And …
  • I could probably write a pretty good book explaining how to do it.

If you're wondering what the beginner's "how do I boil water?" recipe for software project usability is, stop reading this post and get a copy of Rocket Surgery Made Easy. Now.

One of the holy grails of usability testing is eyetracking -- measuring where people's eyes look as they use software and web pages. Yes, there are clever JavaScript tools that can measure where users move their pointers, but that's only a small part of the story. Where the eye wanders, the pointer may not, and vice-versa. But, who has the time and equipment necessary to conduct an actual eyetracking study? Almost nobody.

That's where Eyetracking Web Usability comes in.

Eyetracking-web-usability

Eyetracking Web Usability is chock full of incredibly detailed eyetracking data for dozens of websites. Even though you (probably) can't afford to do real eyetracking, you can certainly use this book as a reference. There is enough variety in UI and data that you can map the results, observations, and explanations found here to what your project is doing.

This particular book is rather eyetracking specific, but it's just the latest entry in a whole series on usability, and I recommend them all highly. These books are a fount of worthwhile data for anyone who works on software and cares about usability, from one of the most preeminent usability experts on the web.

Usability isn't really cheap or easy. It's an endless war, with innumerable battlegrounds, stretching all the way back to the dawn of computing. But these books, at least, are cheap and easy in the sense that they give you some basic training in fighting the good (usability) fight. That's the best I can do, and it's all I'd ask from anyone else I work with.

Posted by Jeff Atwood    32 Comments

The Opposite of Fitts' Law

March 24, 2010

If you've ever wrangled a user interface, you've probably heard of Fitts' Law. It's pretty simple -- the larger an item is, and the closer it is to your cursor, the easier it is to click on. Kevin Hale put together a great visual summary of Fitts' Law, so rather than over-explain it, I'll refer you there.

The short version of Fitts' law, to save you all that tedious reading, is this:

  • Put commonly accessed UI elements on the edges of the screen. Because the cursor automatically stops at the edges, they will be easier to click on.
  • Make clickable areas as large as you can. Larger targets are easier to click on.

I know, it's very simple, almost too simple, but humor me by following along with some thought exercises. Imagine yourself trying to click on ...

  • a 1 x 1 target at a random location
  • a 5 x 5 target at a random location
  • a 50 x 50 target at a random location
  • a 5 x 5 target in the corner of your screen
  • a 1 x 100 target at the bottom of your screen

Fitts' Law is mostly common sense, and enjoys enough currency with UI designers that they're likely to know about it even if they don't follow it as religiously as they should. Unfortunately, I've found that designers are much less likely to consider the opposite of Fitts' Law, which is arguably just as important.

If we should make UI elements we want users to click on large, and ideally place them at corners or edges for maximum clickability -- what should we do with UI elements we don't want users to click on? Like, say, the "delete all my work" button?

Alan Cooper, in About Face 3, calls this the ejector seat lever.

In the cockpit of every jet fighter is a brightly painted lever that, when pulled, fires a small rocket engine underneath the pilot's seat, blowing the pilot, still in his seat, out of the aircraft to parachute safely to earth. Ejector seat levers can only be used once, and their consequences are significant and irreversible.

Applications must have ejector seat levers so that users can—occasionally—move persistent objects in the interface, or dramatically (sometimes irreversibly) alter the function or behavior of the application. The one thing that must never happen is accidental deployment of the ejector seat.

Unintended-ejection-seat-lever-consequences

The interface design must assure that a user can never inadvertently fire the ejector seat when all he wants to do is make some minor adjustment to the program.

I can think of a half-dozen applications I regularly use where the ejector seat button is inexplicably placed right next to the cabin lights button. Let's take a look at our old friend GMail, for example:

Gmail-send-vs-save-now

I can tell what you're thinking. Did he click Send or Save Now? Well, to tell you the truth, in all the excitement of composing that angry email, I kind of lost track myself. Good thing we can easily undo a sent mail! Oh wait, we totally can't. Consider my seat, or at least that particular rash email, ejected.

It's even worse when I'm archiving emails.

Gmail-archive-vs-report-spam

While there were at least 10 pixels between the buttons in the previous example, here there are all of ... three. Every few days I accidentally click Report Spam when I really meant to click Archive. Now, to Google's credit, they do offer a simple, obvious undo path for these accidental clicks. But I can't help wondering why it is, exactly, that these two buttons with such radically different functionality just have to be right next to each other.

Undo is powerful stuff, but wouldn't it be better still if I wasn't pulling the darn ejector seat lever all the time? Wouldn't it make more sense to put that risky ejector seat lever in a different location, and make it smaller? Consider the WordPress post editor.

Wordpress-update-vs-trash

Here, the common Update operation is large and obviously a button -- it's easy to see and easy to click on. The less common Move to Trash operation is smaller, presented as a vanilla hyperlink, and placed well away from Update.

The next time you're constructing a user interface, you should absolutely follow Fitts' law. It just makes sense. But don't forget to follow the opposite of Fitts' law, too -- uncommon or dangerous UI items should be difficult to click on!

Posted by Jeff Atwood    122 Comments

Compiled or Bust?

March 19, 2010

While I may have mixed emotions toward LINQ to SQL, we've had great success with it on Stack Overflow. That's why I was surprised to read the following:

If you are building an ASP.NET web application that's going to get thousands of hits per hour, the execution overhead of Linq queries is going to consume too much CPU and make your site slow. There’s a runtime cost associated with each and every Linq Query you write. The queries are parsed and converted to a nice SQL Statement on every hit. It’s not done at compile time because there’s no way to figure out what you might be sending as the parameters in the queries during runtime.

So, if you have common Linq to Sql statements like the following one ..

var query = from widget in dc.Widgets
            where widget.ID == id && widget.PageID == pageId
            select widget;
var widget = query.SingleOrDefault();

.. throughout your growing web application, you are soon going to have scalability nightmares.

J.D. Conley goes further:

So I dug into the call graph a bit and found out the code causing by far the most damage was the creation of the LINQ query object for every call! The actual round trip to the database paled in comparison.

I must admit, these results seem ... unbelievable. Querying the database is so slow (relative to typical code execution) that if you have to ask how long it will take, you can't afford it. I have a very hard time accepting the idea that dynamically parsing a Linq query would take longer than round-tripping to the database. Pretend I'm from Missouri: show me. Because I am unconvinced.

All of this is very curious, because Stack Overflow uses naive, uncompiled Linq queries on every page, and we are a top 1,000 website on the public internet by most accounts these days. We get a considerable amount of traffic; the last time I checked it was about 1.5 million pageviews per day. We go to great pains to make sure everything is as fast as we can. We're not as fast as we'd like to be yet, but I think we're doing a reasonable job so far. The journey is still very much underway -- we realize that overnight success takes years.

Anyway, Stack Overflow has dozens to hundreds of plain vanilla uncompiled Linq to SQL queries on every page. What we don't have is "scalability nightmares". CPU usage has been one of our least relevant constraints over the last two years as the site has grown. We've also heard from other development teams, multiple times, that Linq to SQL is "slow". But we've never been able to reproduce this even when armed with a profiler.

Quite the mystery.

Now, it's absolutely true that Linq to SQL has the performance peculiarity both posters are describing. We know that's true because Rico tells us so, and Rico ... well, Rico's the man.

In short the problem is that the basic Linq construction (we don’t really have to reach for a complex query to illustrate) results in repeated evaluations of the query if you ran the query more than once.

Each execution builds the expression tree, and then builds the required SQL. In many cases all that will be different from one invocation to another is a single integer filtering parameter. Furthermore, any databinding code that we must emit via lightweight reflection will have to be jitted each time the query runs. Implicit caching of these objects seems problematic because we could never know what good policy is for such a cache -- only the user has the necessary knowledge.

It's fascinating stuff; you should read the whole series.

What's unfortunate about Linq in this scenario is that you're intentionally sacrificing something that any old and busted SQL database gives you for free. When you send a garden variety parameterized SQL query through to a traditional SQL database, it's hashed, then matched against existing cached query plans. The computational cost of pre-processing a given query is only paid the first time the database sees the new query. All subsequent runs of that same query use the cached query plan and skip the query evaluation. Not so in Linq to SQL. As Rico said, every single run of the Linq query is fully parsed every time it happens.

Now, there is a way to compile your Linq queries, but I personally find the syntax kind of ... ugly and contorted. You tell me:

Func<Northwinds, IQueryable<Orders>, int> q =
     CompiledQuery.Compile<Northwinds, int, IQueryable<Orders>>
                ((Northwinds nw, int orderid) => 
                            from o in nw.Orders 
                            where o.OrderId == orderid 
                            select o );

Northwinds nw = new Northwinds(conn);

foreach (Orders o in q(nw, orderid))
{
}

Anyway, that's neither here nor there; we can confirm the performance penalty of failing to compile our queries ourselves. We recently wrote a one time conversion job against a simple 3 column table containing about 500,000 records. The meat of it looked like this:

db.PostTags.Where(t => t.PostId == this.Id).ToList();

Then we compared it with the SQL variant; note that this is also being auto-cast down to the handy PostTag object as well, so the only difference is whether or not the query itself is SQL.

db.ExecuteQuery(
   "select * from PostTags where PostId={0}", this.Id).ToList();

On an Intel Core 2 Quad running at 2.83 GHz, the former took 422 seconds while the latter took 275 seconds.

The penalty for failing to compile this query, across 500k iterations, was 147 seconds. Wow! That's 1.5 times slower! Man, only a BASIC programmer would be dumb enough to skip compiling all their Linq queries. But wait a second, no, wait 147 seconds. Let's do the math, even though I suck at it. Each uncompiled run of the query took less than one third of a millisecond longer.

At first I was worried that every Stack Overflow page was 1.5 times slower than it should be. But then I realized it's probably more realistic to make sure that any page we generate isn't doing 500 freakin' thousand queries! Have we found ourselves in the sad tragedy of micro-optimization theater ... again? I think we might have. Now I'm just depressed.

While it's arguably correct to say that every compiled Linq query (or for that matter, any compiled anything) will be faster, your decisions should be a bit more nuanced than compiled or bust. How much benefit you get out of compilation depends how many times you're doing it. Rico would be the first to point this out, and in fact he already has:

Testing 1 batches of 5000 selects 

uncompiled 543.48 selects/sec compiled 925.75 selects/sec

Testing 5000 batches of 1 selects

uncompiled 546.03 selects/sec compiled 461.89 selects/sec

Have I mentioned that Rico is the man? Do you see the inversion here? Either you're doing 1 batch of 5000 queries, or 5000 batches of 1 query. One is dramatically faster when compiled; the other is actually a big honking net negative if you consider the developer time spent converting all those beautifully, wonderfully simple Linq queries to the contorted syntax necessary for compilation. Not to mention the implied code maintenance.

I'm a big fan of compiled languages. Even Facebook will tell you that PHP is about as half as fast as it should be on a good day with a tailwind. But compilation alone is not the entire performance story. Not even close. If you're compiling something -- whether it's PHP, a regular expression, or a Linq query, don't expect a silver bullet, or you may end up disappointed.

Posted by Jeff Atwood    71 Comments