RIP Fred Brooks

Fred Brooks passed away this month. I first wrote about him on this blog in April 2007, right after I had finished reading his seminal book on software engineering, The Mythical Man-Month. I was about 2 years into my career as a software engineer at that point. How time flies!

I thought it might be interesting to pull out my dog-eared copy of the book and reflect on a few of my favorite concepts and how they've borne out in my career 15 years on.

The Mythical Man-Month

First off is the titular concept from the book, in which Brooks's Law is coined.

Adding manpower to a late software project makes it later.

I will say that in my career, I've only seen one project where bodies were hurled at a late project to hasten its completion. Interestingly, that was also the latest project I've ever seen in my career. 

Thankfully, my experience has been that Brooks's Law has permeated amongst the managers I've worked with—if not in name, they at least intuitively understood that they couldn't just add more people to a project to get it done faster.

The Second-System Effect

I've seen this play out several times in my career. The idea that we'll replace the hacky system we developed before we knew better with one where we take our time and "get it right" this time.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used "next time." Sooner or later the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.

It's easy to forget that just because we have a lot of ideas for improvements that not all of them are worth doing.

No Silver Bullet

I would dare say that most software engineers who have heard of Fred Brooks, found their way to him via his essay No Silver Bullet.

There is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.

Not only are there no silver bullets now in view, the very nature of software makes it unlikely that there will be any.

Personally, the message of No Silver Bullet is so ingrained in me as a software engineer, that I don't consciously think about where it came from very often, although it colors my world view constantly.

Every time I see a headline on Hacker News about Framework X vs. Framework Y, or Language A vs. Language B, I understand intuitively how little they can possibly impact the success of a software project.

When I wrote my post Stack Chasers vs. Evergreens in 2018, I don't think I was even aware that I was channeling No Silver Bullet that I had read over a decade earlier. It was in my bones at this point that technological solutions for improving the art of software engineering are inherently, severely limited.

Brooks's treatment of essential complexity vs. accidental complexity had such an impact on me, that I reference it when approaching challenges in my own life that have nothing to do with software engineering or my career at all.

The Mythical Man-Month is holding up amazingly well in 2022. I can't think of any other nearly 50-year-old book written about computer software that is still so relevant and insightful.

Thank you, Fred! R.I.P. And get a copy of The Mythical Man-Month if you haven't already read it. 

What Is a Staff Engineer?

What's next for someone who has been at the Senior Engineer level for several years or more? It used to be that you had to go into people management to level up. But nowadays, more and more companies are offering a level above Senior called Staff, that is not inherently a people management role. So what is Staff? 

There's an interesting post over on the GitLab blog where the author talked to several Staff Engineers in the company and identified the common threads in their roles:

• Proactive problem identification
• Cultural stewardship
• Knowledge sharing
• Technical empowerment
• Broader impact beyond the team

I would summarize this list as doing technical work that crosses multiple team boundaries. A Staff Engineer would be doing whatever it takes to make cross-team initiatives that need an expert software engineer successful.

It's hard not to come across Will Larson's work when searching for definitions of Staff Engineer. He literally wrote a book called Staff Engineer that is all about this very topic. Here is Will's answer to the question, "What do Staff Engineers actually do?"…

The most straightforward answer is that Staff engineers keep doing much of what made them successful as Senior engineers: building relationships, writing software, coordinating projects. However, that's a misleading answer. Staff engineers do those same tasks, but whereas previously they were the core of their work, now they're auxiliary tasks. Their daily schedule varies a bit by archetype, but there's a shared foundation across all archetypes: setting and editing technical direction, providing sponsorship and mentorship, injecting engineering context into organizational decisions, exploration, and what Tanya Reilly calls being glue.

Will talks more about archetypes of the Staff Engineer in another article which is what introduced me to his work (Hacker News discussion here). He's identified four different kinds of Staff Engineers that are commonly found in the wild:

• The Tech Lead guides the approach and execution of a particular team. They partner closely with a single manager, but sometimes they partner with two or three managers within a focused area. Some companies also have a Tech Lead Manager role, which is similar to the Tech Lead archetype but exists on the engineering manager ladder and includes people management responsibilities.
• The Architect is responsible for the direction, quality, and approach within a critical area. They combine in-depth knowledge of technical constraints, user needs, and organization level leadership.
• The Solver digs deep into arbitrarily complex problems and finds an appropriate path forward. Some focus on a given area for long periods. Others bounce from hotspot to hotspot as guided by organizational leadership.
• The Right Hand extends an executive's attention, borrowing their scope and authority to operate particularly complex organizations. They provide additional leadership bandwidth to leaders of large-scale organizations.

Read Will's article for deep dives into all the archetypes, complete with examples of what their schedules are like and interviews with actual Staff Engineers who he sees as fitting these different archetypes. It's very enlightening, and I can't really imagine a better way of getting insight into the real-world nuts and bolts of that title.

My takeaway is that a Senior Engineer sort of maxes out as an expert software engineer. The Staff Engineer is about taking engineering expertise and expanding the scope of one's influence, which could mean overseeing a group of people, facilitating communication between groups of people, or solving problems involving multiple codebases. And maybe this goes without saying, but you're not likely to see the Staff Engineer title existing outside of large technical organizations, where the scope of engineering activity requires this sort of role.

Remote Hiring and the Paradox of Choice

Oh, how I love remote work. And oh, boy, I've written a lot about the topic on this blog. We all know the reasons why remote work is amazing; there's no need to rehash at this point.

But I haven't written much about the flipside, particularly the process of being hired for a remote job. If you consider how much more awesome working from your home is compared to a typical open plan nightmare, then a similar multiplier exists in the other direction for how much more frustrating a job search is for a remote job compared to a local in-person job.

Just as the abundance of remote jobs is a dream from a job seeker's perspective, the abundance of the talent pool from a hiring company's perspective leads to some annoying behavior.

There are clear comparisons to be made to the shift to online dating from "traditional" dating. As the pool of eligible participants increases, the value of each individual participant drops. And just as the modern phenomenon of "ghosting" is common in the world of online dating, it's also common in the world of remote job hunting.

It's way more common in a remote job search to go through multiple rounds of interviews with a company and then never hear from them again, for example. Or for recruiters to hound you on LinkedIn until you agree to a phone call, gush about the perfect fit for a role, and then never contact you again about that next meeting with the hiring manager. Follow-up emails go into a black hole. If you get feedback at all, you find out you were rejected for some seemingly minor and arbitrary shortcoming. That's just how it is when the potential talent pool for a job opening expands across a country or across the globe.

Just as I mentioned earlier a multiplier for how much better a remote job is to a traditional job, you will experience general flakiness from hiring companies at a similar multiplier. There's just something about the abstraction and lack of physical proximity between the participants that amplifies this behavior.

And believe me, it's not just employers who are making this difficult, there's no end to the shenanigans that occur on the candidate's side. Both sides are enjoying the benefits while cursing the drawbacks.

So let's file this post under "Advice for a Younger Me." I write this post in the same spirit as posts I wrote years ago like The Many-Worlds Interpretation of Developer Interviews and People Hire Clones of Themselves. The hiring process for software engineers seems so bizarre and arbitrary a lot of the time that it helps to remember that there are reasons for the weirdness that have nothing to do with you, and it's like this for everybody.

It's a brave new world out there. Long live remote work. :)

Swamped to Sustainable

I've written a lot about the concept of sustainability on this blog, and it's something I'm constantly thinking about with regards to software development.

That's why I enjoyed this recent post by Greg Kogan about being "swamped" all the time:

I used to think being swamped was a good sign. I’m doing stuff! I’m making progress! I’m important! I have an excuse to make others wait! …

Now, I’m impressed by people who are not swamped. They prioritize ruthlessly to separate what’s most important from everything else, think deeply about those most-important things, execute them well to make a big impact, do that consistently, and get others around them to do the same. Damn, that’s impressive!

Being swamped isn’t a badge of honor, it’s something to work on.

There were good comments in the discussion of the post on Hacker News, including a link to a manifesto of sorts I wasn't previously familiar with called Sustainable Development.

Sustainable Development is a set of principles for software teams to use to operate in healthy and productive ways, and for the long term.

Software teams practicing Sustainable Development follow guidelines that benefit them in three areas:

Physical: They work in ways that allows them to maintain good physical health.

Emotional: They work in an environment that supports their emotional health.

Cognitive: They work in ways that encourage creativity and support the intellectual nature of software engineering.

I could see this being used as a framing device for discussions in a sprint retrospective.

Development teams wanting to adopt Sustainable Development simply write down the practices they wish to embrace, and then commit to following them. Each practice should benefit the team in at least one of the three areas of sustainability (physical, emotional, or cognitive).

It's often hard getting people to speak productively in retros, and I feel like a good place to start is asking the question to the group: What did we do in this last sprint that doesn't seem sustainable? What can we do to increase the sustainability of our sprints? The suggested sample practices could give the group some ideas on action items to work toward greater sustainability.

I hope every team can agree that swamped is not sustainable.

Documentation Is a Competitive Advantage

People often say that one of the reasons Stripe is so successful is that their documentation is world-class.

Most companies, open source projects, and individual developers find this to be a Herculean challenge. I'd love to know if Stripe has any secrets to keeping their Docs updated. Though, my gut says that they likely just put in the work. Their Docs are incredibly important to their success.

- cdevroe, Hacker News commenter

Thanks to Stripe and a few others, developers' expectations have changed. Now they expect thorough, navigable documentation presented in a clean, error-free UI. If you deliver anything less, few developers will even bother with your API.

- ReadMe Blog

It really is amazing how much of a difference clear, up-to-date documentation makes to set one product apart from its competitors. 

But I like to think about the impact documentation has on one's career as a software engineer. I've written extensively on this blog about the importance of written communication in the context of remote work.

There's so much value one can contribute as a software engineer that goes beyond writing code. 

Some examples:

- You're working on a ticket where the acceptance criteria are ambiguous in some way. You ask the product owner for guidance on the parts that aren't clear. Instead of just moving on with your work on the ticket, go back to the ticket and either update the acceptance criteria to what you discussed with the product owner, or leave a comment on that ticket that documents what you discussed and how it affects the requirements.

- You ran into some error when running the system that you've never seen before. You ask in the team chat for help. A teammate has seen this error before and pings you with the steps on how to get past the error. Before continuing on with your work, write a bug ticket in the team's backlog with as much info as you can while the problem and solution are still fresh in your mind, and post a link to the bug ticket in your team chat.

- You discover while following a process documented in the team's wiki that some of the steps are out-of-date or not relevant anymore. Take some notes on what you found lacking in the wiki page, and then come back and edit the page yourself to make it clearer.

Most engineers don't do these things, and as such, they're great ways to stand out for engineers that make a point to do them. The work of an engineer involves so much learning—valuable information that is often lost when the feature is added or the bug is fixed, and we move on without documenting what we learned along the way, artifacts that won't be captured in the code alone.

Just as products can stand out with great documentation, so can engineers.

New People Write the Best Documentation

Have you ever heard that the best way to learn a subject is to teach it?

I've had the experience several times when I've joined a new team and started ramping up, that their documentation is out of date and skips over important information.

We know there's always that awkward transitional phase when a new engineer joins a team that's existed for a while, because they're not immediately ready to take on "real work", even if they're an experienced engineer in general. 

This is a perfect opportunity to improve the documentation! In fact, I think a great first assignment for an engineer new to a team is to update the onboarding documentation. They'll see exactly what's missing as they try to follow along, in a way that veterans on the team never will.

When experienced folks are tasked with writing documentation, they don't need the documentation, so they will tend to hand-wave and make leaps in explaining things because they're not truly starting from scratch like a new person is.

Documentation written with a fresh perspective is so helpful, and of course, living documents are best. Establish a virtuous cycle where each new person leaves the docs a little better than they found them.

Let developers do "off the board" work

When sprints are well planned and sustainably paced, it's natural to have developers who have finished all of their committed backlog items with time to spare. In occasions like this, I've seen teams where the Scrum Master—or some other authority figure—reflexively looks for additional items in the backlog to "pull into" the sprint to keep the developers busy. I think this is a mistake, and I'll explain why.

For Agile teams that are organizing their work in sprints, it's traditional to have some sort of visual board or central place that anyone on the team or anyone interested in the team's work can go to see which items are being worked on and the progress of the items. Some people get uncomfortable with the idea of work happening off the board, where it's not officially recognized as work that counts. They want to see each developer with at least one assigned item on the board that is in progress right up until the end of the sprint. That's how we know that we're getting maximum velocity and value from the team, right?

But I think it's important to normalize the idea that people need time to work on things that are "off the board" for a couple of reasons:

1. People always have work to get done that is not directly related to a team goal.
2. Continuous sprinting is not sustainable.

Here are a few legitimate things to work on that don't belong on a sprint board:

• Training
• Preparing for a presentation
• Proof of concept / demo of an intriguing tool, framework, technology
• Updating documentation that's been bugging you
• Spikes into performance improvements
• Cleaning out your inbox
• For companies that do some sort of periodic "goal setting" for each employee, let people work on goals from their list
• In general…things that don't involve QA

I think that last point is important, because people will often talk about downtime being useful for addressing technical debt. But that only works for these end-of-sprint downtime scenarios as long as you're not adding to the regression testing burden of any QA folks that are still testing development work completed during the current sprint. A big technical-debt-reducing refactor is not something we want to check in casually without QA time reserved for making sure it hasn't blown up anything (unless your codebase has truly superb automated regression tests in place).

It is incumbent on the developers to identify their own priorities that they can pull from when they have extra time. Don't ask for more backlog items if you have work in mind that's off the board. For Scrum Masters, the go-to move should not be to pull in more backlog items without asking if the developer has any off-the-board work that can keep them busy.

Apart from recognizing the reality that developers have work to do that isn't directly related to a product backlog, it's important to reflect on the idea of sustainable pace and what "sprinting" really means anyway.

What does the word sprint mean outside the software world? It's a short, high intensity run. In the real world, you don't finish a sprint and then instantly start sprinting again. You have to catch your breath and take a pause before you can sprint again. Otherwise you'll collapse. It's important that software teams do this too—take a breather before the next sprint begins.

Smooth, predictable iterations necessitate a sustainable pace and regular buffers. It's okay if we have time each sprint that is not maximally stuffed with points.