Hacking and Refactoring:

In 2001, there was a history-making conference of software-engineering thinkers in Snowbird, Colorado. The product of that meeting was a remarkable document called the Agile Manifesto, a call to overturn many of the assumptions of traditional software development. I was invited to be at Snowbird, but couldn't make it.

Ever since, though, I've been sensing a growing convergence between agile programming and the open-source movement. I've seen agile concepts and terminology being adopted rapidly and enthusiastically by my colleagues in open-source-land—especially ideas like refactoring, unit testing, and design from stories and personas. From the other side, key agile-movement figures like Kent Beck and Martin Fowler have expressed strong interest in open source both in published works and to me personally. Fowler has gone so far as to include open source on his list of agile-movement schools.

I agree that we belong on that list. But I also agree with Fowler's description of of open source as a style, rather than a process. I think his reservations as to whether open source can be described as just another agile school are well-founded. There is something more complicated and interesting going on here. and I realized when I read Fowler's description of open source that at some point I was going to have to do some hard thinking and writing in an effort to sort it all out.

While doing research for my forthcoming book, The Art of Unix Programming, I read one particular passage in Fowler's Refactoring that finally brought it all home. He writes:

One argument is that refactoring can be an alternative to up-front design. In this scenario, you don't do any design at all. You just code the first approach that comes into your head, get it working, and then refactor it into shape. Actually, this approach can work. I've seen people do this and come out with a very well-defined piece of software. Those who support Extreme Programming often are portrayed as advocating this approach.

I read this, and had one of those moments where everything comes together in your head with a great ringing crash and the world assumes a new shape—a moment not unlike the one I had in late 1996 when I got the central insight that turned into The Cathedral and the Bazaar. In the remainder of this essay I'm going to try to articulate what I now think I understand about open source, agile programming, how they are related, and why the connection should be interesting even to programmers with no stake in either movement.

Now I need to set a little background here, because I'm going to need to have to talk about several different categories which are contingently but not necessarily related.

First, there is Unix programmer. Unix is the operating system with the longest living tradition of programming and design. It has an unusually strong and mature technical culture around it, a culture which originated or popularized many of the core ideas and tools of modern software design. The Art of Unix Programming is a concerted attempt to capture the craft wisdom of this culture, one to which I have successfully enlisted quite a few of its founding elders.

Second, there is hacker. This is a very complex term, but more than anything else, it describes an attitude—an intentional stance that relates hackers to programming and other disciplines in a particular way. I have described the hacker stance and its cultural correlates in detail in How To Become A Hacker.

Third, there is open-source programmer. Open source is a programming style with strong roots in the Unix tradition and the hacker culture. I wrote the modern manifesto for it in 1997, The Cathedral and the Bazaar, building on earlier thinking by Richard Stallman and others.

These three categories are historically closely related. It is significant that a single person (accidentally, me) wrote touchstone documents for the second and third and is attempting a summum bonum of the first. That personal coincidence reflects a larger social reality that in 2003 these categories are becoming increasingly merged — essentially, the hacker community has become the core of the open-source community, which is rapidly re-assimilating the parts of the Unix culture that got away from the hackers during the ten bad years after the AT&T divestiture in 1984.

But the relationship is not logically entailed; we can imagine a hacker culture speaking a common tongue other than Unix and C (in the far past its common tongue was Lisp), and we can imagine an explicit ideology of open source developing within a cultural and technical context other than Unix (as indeed nearly happened several different times).

With this scene-setting done, I can explain that my first take on Fowler's statement was to think "Dude, you've just described hacking!"

I mean something specific and powerful by this. Throwing together a prototype and refactoring it into shape is a rather precise description of the normal working practice of hackers since that culture began to self-define in the 1960s. Not a complete one, but it captures the most salient feature of how hackers relate to code. The open-source community has inherited and elaborated this practice, building on similar tendencies within the Unix tradition.

The way Fowler writes about design-by-refactoring has two huge implications for the relationship between open source and agile programming:

First, Fowler writes as though he didn't know he was describing hacking. In the passage, he appears unaware that design by repeated refactoring is not just a recent practice semi-accidentally stumbled on by a handful of agile programmers, but one which hundreds of thousands of hackers have accumulated experience with for over three decades and have in their bones. There is a substantial folklore, an entire craft practice, around this!

Second, in that passage Fowler described the practice of hacking better than hackers themselves have done. Now, admittedly, the hacker culture has simply not had that many theoreticians, and if you list the ones that are strongly focused on development methodology you lose Richard Stallman and are left with, basically, myself and maybe Larry Wall (author of Perl and occasional funny and illuminating ruminations on the art of hacking). But the fact that we don't have a lot of theoreticians is itself an important datum; we have always tended to develop our most important wisdoms as unconscious and unarticulated craft practice.

These two observations imply an enormous mutual potential, a gap across which an arc of enlightenment may be beginning to blaze. It implies two things:

First, people who are excited by agile-programming ideas can look to open source and the Unix tradition and the hackers for the lessons of experience. We've been doing a lot of the stuff the agile movement is talking about for a long time. Doing it in a clumsy, unconscious, learned-by-osmosis way, but doing it nevertheless. I believe that we have learned things that you agile guys need to know to give your methodologies groundedness. Things like (as Fowler himself observes) how to manage communication and hierarchy issues in distributed teams.

Second, open-source hackers can learn from agile programmers how to wake up. The terminology and conceptual framework of agile programming sharpens and articulates our instincts. Learning to speak the language of open source, peer review, many eyeballs, and rapid iterations gave us a tremendous unifying boost in the late 1990s; I think becoming similarly conscious about agile-movement ideas like refactoring, unit testing, and story-centered design could be just as important for us in the new century.

I've already given an example of what the agile movement has to teach the hackers, in pointing out that repeated redesign by refactoring is a precise description of hacking. Another thing we can stand to learn from agile-movement folks is how to behave so that we can actually develop requirements and deliver on them when the customer isn't, ultimately, ourselves.

For the flip side, consider Fowler's anecdote on page 68-69, which ends "Even if you know exactly what is going on in your system, measure performance, don't speculate. You'll learn something, and nine times out of ten it won't be that you were right." The Unix guy in me wants to respond "Well, duh!". In my tribe, profiling before you speculate is DNA; we have a strong tradition of this that goes back to the 1970s. From the point of view of any old Unix hand, the fact that Fowler thought he had to write this down is a sign of severe naivete in either Fowler or his readership or both.

In reading Refactoring, I several times had the experience of thinking "What!?! That's obvious!" closely followed by "But Fowler explains it better than Unix traditions do..." This may be because he relies less on the very rich shared explanatory context that Unix provides.

How deep do the similarities run? Let's take a look at what the Agile Manifesto says:

Individuals and interactions over processes and tools. Yeah, that sounds like us, all right. Open-source developers will toss out a process that isn't working in a nanosecond, and frequently do, and take gleeful delight in doing so. In fact, the reaction against heavyweight process has a key part of our self-identification as hackers for at least the last quarter century, if not longer.

Working software over comprehensive documentation. That's us, too. In fact, the radical hacker position is that source code of a working system is its documentation. We, more than any other culture of software engineering, emphasize program source code as human-to-human communication that is expected to bind together communities of cooperation and understanding distributed through time and space. In this, too, we build on and amplify Unix tradition.

Customer collaboration over contract negotiation. In the open-source world, the line between "developer" and "customer" blurs and often disappears. Non-technical end users are represented by developers who are proxies for their interests—as when, for example, companies that run large websites second developers to work on Apache Software Foundation projects.

Responding to change over following a plan. Absolutely. Our whole development style encourages this. It's fairly unusual for any of our projects to have any plan more elaborate than "fix the current bugs and chase the next shiny thing we see".

With these as main points, it's hardly surprising that so many of the Principles behind the Agile Manifesto read like Unix-tradition and hacker gospel. "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Well, yeah—we pioneered this. Or "Simplicity—the art of maximizing the amount of work not done—is essential." That's Unix-tradition holy writ, there. Or "The best architectures, requirements, and designs emerge from self-organizing teams."

This is stone-obvious stuff to any hacker, and exactly the sort of subversive thinking that most panics managers attached to big plans, big budgets, big up-front design, and big rigid command-and-control structures. Which may, in fact, be a key part of its appeal to hackers and agile developers—because at least one thing that points agile-movement and open-source people in the same direction is a drive to take control of our art back from the suits and get out from under big dumb management.

The most important difference I see between the hackers and the agile-movement crowd is this: the hackers are the people who never surrendered to big dumb management — they either bailed out of the system or forted up in academia or industrial R&D labs or technical-specialty areas where pointy-haired bosses weren't permitted to do as much damage. The agile crowd, on the other hand, seems to be composed largely of people who were swallowed into the belly of the beast (waterfall-model projects, Windows, the entire conventional corporate-development hell so vividly described in David Yourdon's books) and have been smart enough not just to claw their way out but to formulate an ideology to justify not getting sucked back in.

Both groups are in revolt against the same set of organizational assumptions. And both are winning because those assumptions are obsolete, yesterday's adaptations to a world of expensive machines and expensive communications. But software development doesn't need big concentrations of capital and resources anymore, and doesn't need the control structures and hierarchies and secrecy and elaborate rituals that go with managing big capital concentrations either. In fact, in a world of rapid change, these things are nothing but a drag. Thus agile techniques. Thus, open source. Converging paths to the same destination, which is not just software that doesn't suck but a software-development process that doesn't suck.

When I think about how the tribal wisdom of the hackers and the sharp cut-the-bullshit insights of the agile movement seem to be coming together, my mind keeps circling back to Phil Greenspun's brief but trenchant essay Redefining Professionalism for Software Engineers. Greenspun proposes, provocatively but I think correctly, that the shift towards open-source development is a key part of the transformation of software engineering into a mature profession, with the dedication to excellence and ethos of service that accompanies professionalism. I have elsewhere suggested that we are seeing a close historical analog of the transition from alchemy to chemistry. Secrets leak out, but skill sustains; the necessity to stop relying on craft secrecy is one of the crises that occupational groups normally face as they attain professional standing.

I'm beginning to think that from the wreckage of the software industry big dumb management made, I can see the outline of a mature, humane discipline of software engineering emerging — and that it will be in large part a fusion of the responsiveness and customer focus of the agile movement with the wisdom and groundedness of the Unix tradition, expressed in open source.