The Teeth
While in college, I was lucky enough to attend a conference called the Future of Web Apps (FOWA) in Miami in February, 2008. The conference featured a wide range of speakers, but a single slide from Cal Henderson’s talk still sticks with me to this day. He was between Flickr and Slack, and was giving a talk summarizing his recent book, Building Scalable Web Sites.
During the talk, Cal covered things from Continuous Delivery to automation, all of which were relatively new concepts at the time.
What stuck with me was a single slide in which he talks about “The Teeth.” I find myself referencing that one slide many times throughout each year.
This blog post is an homage to that single slide in his talk at FOWA 2008 Miami.
Customer Value over Time
When building software products, whether they are web apps, desktop applications, mobile apps, or server appliances, they only survive if they deliver value to your customer. “Value” is MBA for benefit minus cost. Does the customer receive more benefit than the cost? Does your $7 todo app give the customer $10 worth of extra productivity?
The web significantly changed the process of building software. Rather than swapping out floppies in sequence or listening to the purr of a CD-ROM drive, it was just there. What once took months to put something into the customers hands, it now took seconds. It was there and it was new. All the time, every day.
So this is the world that we play in as software developers. Time is the most precious asset. If we don’t move fast enough, our competitors will get there first. Speed is a competitive advantage. The towering question of software is “How quickly can we deliver customer value?”
A Tale of Two Processes
Developing software is a tug-of-war between two ideas: “Do we incrementally change the product in place?” or “Do we blow the thing up and rewrite it?”
Blowing the whole thing up and rewriting it in place is listed as one of the things you should never do, according to Joel Spolsky. The problem with a rewrite is that you’re often subjected to “second 90%” of the work. A rewrite is not re-integrated into the original system until it nears completion. This integration work is difficult to estimate and often where the dragons be. Large rewrites can be dangerous.
In the other school of thought lies incremental change. Set a vision and iterate toward it. Ship as often as possible. By shipping as often as possible, you constantly pay the price of integration. Should company politics get in the way or the project run late, you at least delivered some value to the customer. Delivering value quickly to the customer is one of the important pieces of Extreme Programming and things like it.
So we have these two approaches toward delivering value to the customer, and we start to notice some shapes emerge…
The Important Thing About Teeth
After laying out this argument in his original talk, Cal draws some eyes above the line he’s created, creating a face out of the previously staid graph. “The problem with teeth is that the larger they are, the more likely they are to break the skin.”
“So when developing software,” Cal continues, “Keep your teeth small.”
Breaking the skin can be taking down production, shipping regressions, or something else equally catastrophic that requires a sudden unship.
I love this imagery. (Here I am, writing a blog post 10 years after the fact.) It gives teams great short-hand for sequencing projects:
“How will we avoid the Teeth?”
Every Smile is Unique
There are many failures modes to a project with large teeth. Let’s enumerate a few of them:
- The project may never ship
- The project gets canceled
- The project becomes a Death March
- The project becomes unsustainable due to parallel development
- The project may ship and then need to be rolled back
- The project may ship but be so foreign when compared to the original that it doesn’t get used
Let’s dive in.
1. The project may never ship
The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time. —Tom Cargill, Bell Labs
Projects with long times between development and delivery are often subject to the “second 90%” of project estimation. Once you’re done with the first 90% of development, you have the second 90% of integration.
That second 90% was not accounted for in estimates. The second 90% is not budgeted for. When the second 90% shows up, there are some difficult discussions as to whether the project should have been undertaken in the first place.
Although many developers practice parts of Continuous Delivery, we often find ways of sneaking around that. Yes, we deploy our code daily but we find ways to delay integration. Feature flags are a great tool in many cases, but are often a way of delaying integration.
In the worst case, the second 90% is so long that the project never ships and the company goes under like Netscape.
2. The project gets canceled
Because of the second 90%, the project will sometimes get outright canceled. The engineers working on the project have been scheduled to roll off onto a higher priority project. Them’s the breaks, and the team just spent the last few months building something that never makes it to customers.
Talk about a morale killer.
3. The project becomes a Death March
As the toothen project enters the second 90% of development, it can be tough to keep the team together. Signs of burnout start to appear, and what seemed like a project that would be shipping soon now has no end in sight.
This point is a good time to take a nice vacation, because there’s still plenty of work left to do.
Managers and the powers that be will sometimes insist that progress be made at all costs. We are so close to the finish line, just a little bit more! And just like that, your team now has two problems: a late project and a Death March.
4. The project becomes unsustainable due to parallel development
Legacy systems are funny. No one likes them. The word legacy indicates outdated, crufty, frustrating.
But that legacy software exists for a reason, otherwise you’d just delete it. There’s a good chance that that legacy software is paying your paycheck.
When attempting a Rewrite, this becomes a problem. There’s a good chance the existing legacy system has customers and has bugs. Someone has to shovel the coal into the furnace. As you’re developing the New System, maintaining feature parity becomes a Sisyphean task. You carefully port features to the new system, only to see some new behavior be added to or corrected in the old system.
How does this look to the customer? It looks like you’re moving at half-speed since your team’s time is split between two projects.
5. The project may ship and then need to be rolled back
The Rewrite requires a lot of overhead. Through parallel development tracks, forward-ported bug fixes, and long timelines, it can be difficult to make sure that there are no regressions in the eyes of the user. Did we remember everything?
Every few Rewrites, the team forgets something big. That one feature that was deprioritized for release turned out to be important to a lot of customers. So important that you need to perform a sudden rollback. Before rolling the Rewrite out again, you need to implement that thing-that-you-now-know-is-important.
When you ship a Rewrite, wait a few weeks before celebrating.
6. The project may ship but be so foreign when compared to the original that it doesn’t get used
The early development of Facebook was fascinating. The world was just switching gears from boxed, major-versioned software to iterative products. This crossroads led to some interesting product releases.
Once every 6-12 months, you’d log on to Facebook and everything would be totally different. What was once a right-hand nav was now on the left. The Wall as a large <textarea> would be gone. There is now a thing called a News Feed and it’s the only thing you need in your life.
The user base would collectively revolt. “Join my Group to send a message to Facebook that we DON’T want the News Feed.” Or, “Like this page to tell Facebook that the new design SUCKS!” Large releases colliding with the web made for unhappy users.
Over time, Facebook developed a more graceful way of rolling out changes. “Opt in to our new News Feed to get the latest design!” a polite banner would read. We the users were once again in control. We decided when we wanted to make the trip to CompUSA to pick up that box and drop the CD-ROM into our 4x drive.
These days, the process of rolling out features at Facebook is so sophisticated it likely cannot be comprehended by a single person. One thing is for sure: You hardly notice. The teeth are imperceptibly small.
When developing a Rewrite, you need to take painstaking care that you don’t give your users a shock. Opt-ins and meticulous testing and measurement is more added overhead to delivering value to customers. The extra testing is not always needed, and can be often replaced with slow and methodical progress toward a vision.
Conclusion
When we write software we should always ask, “How do we avoid the Teeth?”
In my career, I try to be aware of the things I remain inflexible on and keep that number small. This process of truly shipping value continuously is one of the few things where I am rigid. In my experience, this continuous delivery makes for healthier teams and happier PMs.
The continuous route is not without its challenges. It requires a strong vision up front, and requires the team to hold that uncomfortable feeling of “I know where we’re going, but I don’t know how we’ll get there.” It requires a high level of trust among everyone involved.
As you work with your team, see if this idea of the Teeth sticks. Hopefully you find it as useful as I have in my day-to-day.
Isn’t it time you scheduled an appointment with your dentist?
Special thanks to Robin Rendle, Alex Navasardyan, Alid Castano, Chris Lesinski, and Allan Grinshtein for providing feedback on early drafts of this post.
Extra special thanks to Robin Rendle and Allan Grinshtein for typography help for this post.
One last thanks to Cal Henderson for giving that talk over 10 years ago.
The lovely illustration was done by Ash Jin.