Right back at ya, Captain:

Last Saturday morning in San Diego I had breakfast with Steven den Beste, the redoubtable captain of U.S.S. Clueless. One of the side-effects of that meeting was a long critique of open-source development. Herewith my response.

Steve and I agree on the scaling problem that has pushed software development efforts to the ragged edge of what is sustainable even by corporations with lots of money. Software project sizes are roughly doubling every eighteen months, and for reasons Steve alluded to the expected bug count per thousand lines is actually rising.

My assertion is that software development has reached a scale at which (a) even large corporations can often no longer afford to field enough developers to be effective at today's project scales, and (b) traditional methods of software quality assurance (ranging from formal methods to internal walkthroughs) are no longer effective. The only development organizations that seem to thrive on today's complexity regime are open-source teams.

Note that I am not claiming that open source is a silver bullet for the software-complexity problem. There are no silver bullets, no permanent solutions. What I am claiming is that at the leading edge of large-scale software, closed-source development doesn't work any more. The future belongs to open source plus whatever other practices and technologies we learn to use with it to develop at ever-higher scales of complexity.

Steve's analysis of the open-source phenomenon is very intelligent, but doesn't quite understand either the mode of organization, the associated technology, or the factional politics within the movement. Diagnostic of the slight disconnect is when he writes "For [the zealots], the only true "Open Source" is governed by the strong form of the GPL, and all other forms and licenses are harmful dilution of the concept." In fact, the people he's talking about reject the term "open source" entirely and insist on the ideologically-loaded term "free software".

A more serious error is when he writes "It is plausible that an OSS project would require each participant to sign an NDA before being given access to the source." It is not plausible. The licenses and community values of the open-source community would not permit this. His two bullet points characterizing open source are missing its most important characteristic: the entire practice is designed to facilitate scrutiny by people with no institutional or contract relationship to the core development team. The astringent effect of peer review by people who have nothing to lose by reporting bugs is precisely the point of the whole game.

Steve doesn't undertand the importance or the power of this effect. This slightly skews his whole essay; much of it is talking past what open-source people do, rather than addressing us. He's also unaware of a lot of the real-world evidence for the success of the method. Some of the things he thinks are technologically or economically impossible are actually being done, routinely.

He's correct when he says that most contributors are self-selected and self-motivated. He overestimates the cost of training newbies, though. They self-train; normally, the first time a core developer hears from a newbie is typically when the newbie sends a patch -- self-evidence that the newbie has already acquired a critical level of knowledge about the software. The "sink or swim" method turns out to work, and work well.

It's incorrect to imply, as he does, that open-source development is unsustainable because the people doing it are flaky amateurs. Steve hasn't absorbed the implications of the Boston Consulting Group study that shows that about 40% of contributors to core projects are professionals getting paid for working on open source by patrons who need to use the results. In fact, what the open-source community is evolving into is something very like a huge machine for bringing newbies into apprenticeship contact with experienced developers and professionalizing both groups.

He also writes "OSS by its nature tends to be reactive rather than predictive. It doesn't look into the future, try to predict a problem which doesn't exist now but will exist then, and be ready with a solution. Rather, it tends to see problems that exist now and work on solutions for them." This is false -- or, at any rate, no more true than it is for closed-source development.

The open-source community built the Web and the Internet before it had acquired a name for itself and full consciousness of its own practices. Today, the cutting-edge work in operating systems languages, desktop user interfaces, relational databases and many other areas is being done either within the open-source community or in cooperation with it by academics. These prodigious efforts of imagination dwarf any "prediction" produced by closed-source software development in the last two decades.

Steve's "open source is reactive" claim strikes me as ironically funny, because I can remember when the standard knock on my crowd was that we're great at innovation but can't actually field product. How quickly they forget...

He's right enough about the difficulty of planning and high cost of face-to-face meetings, though. These are real problems. It's a testimony to the power of our practices that we manage to ship large volumes of high-quality software despite these obstacles.

What Steve called "player-killer" tactics have been tried -- there was a famous incident a few years back in which a TCP-wrappers distribution was Trojaned. The crack was detected and the community warned within hours. The black hats don't seem to bother trying this any more; our reaction time is too fast for that game to be very rewarding. The technical design of Linux helps here in ways that I won't go into here -- suffice it to say that it's intrinsically much harder to get a Trojan to do anything interesting than it is under Windows or other single-user operating systems.

So far, the supply of open-source developers seems to be pretty elastic -- we're not limited much by lacking bodies. Other factors loom much larger; patents, the DMCA, intrinsically hard technical problems. I don't understand why this is as well as I'd like to, but the facts are undeniable; the community is ten times the size my wildest high-end scenarios predicted a decade ago and seems to be growing faster as it gets larger.

Steve's whole argument that open-source can't win in embedded systems is very curious, since it predicts exactly the opposite of what is actually happening out there. Linux is taking over in embedded systems -- in fact, many observers would say it has already won that space. If Steve had worked in the field within the last three years he would probably know this.

Here are some data about the demand; the only non-general-purpose open-source software magazine in existence is the Linux Embedded Systems Journal. Open-source embedded developers like Monta Vista Software are bucking the recession by growing like crazy. The first cell-phone prototype running entirely open-source software just entered beta testing.

I was in California to meet Steve partly because Real Networks wanted me to be on stage when they announced the open-sourcing of their RTSP engine. Their CEO, Rob Glaser, was quite frank about the immediate business reasons: they needed to get ports to forty different Nokia cellphones and just couldn't figure out how to muster the resources for that short of inviting every stakeholder on the planet to hack the problem. Scaling bites. Hard.

In fact, some of the very characteristics that Steve thinks make embedded systems like cellphones safe for closed development seems to be the factors that are driving increased open-sourcing. The close tie to hardware actually decreases the value of secrecy, because it means the software is typically not easily re-usable by hardware competitors. Thus open sourcing is often a great way to recruit help from customer engineers without a real downside risk of plagiarism.

In fact, it's an open secret in the industry that the most important reason most closed-source embedded and driver software remains closed is not nerves about plagiarism but fear of patent audits on the source code. Graphics-card manufacturers, in particular, routinely swipe patented techniques from their competitors and bury them in binaries. (This is generally believed to be the reason nVidia's drivers aren't open.)

Another trend that's driving Linux and open-sourcing in embedded stuff is the shift from specialty embedded 8-bit processors to 32-bit chips with general-purpose architectures. Turns out the development costs for getting stuff to run on the 8-bit chips are sickeningly high and rising -- partly because the few wizards who can do good work on that hardware are expensive. The incremental cost for smarter hardware has dropped a lot; it's now cheaper to embed general-purpose chips running Linux because it means you have a larger, less expensive talent pool that can program them. Also, when your developers aren't fighting hardware limits as hard, you get better time to market (which, as Steve observes, is critical).

Steve is right about the comparative difficulty of applying open-source methods to vertical applications. But the difficulty is only comparative; it's happening anyway. The metalab archive carries a point-of-sale system for pizza parlors. I know of another case in which a Canadian auto dealership built specialized accounting software for their business and open-sourced it. The reasons? Same as usual; they wanted to lay off as much as possible of the development and maintainance cost on their competitors.

This is the same co-opetition logic that makes the Apache Software Foundation work -- it's just as powerful for vertical apps, though less obviously so. Each sponsoring company sees a higher payoff from having the software at a small fraction of the manpower cost for a complete in-house development. The method spreads risk in a way beneficial to all parties, too, because the ability of separate companies to sustain development tends to be uncorrelated -- unless they all sink, the project endures.

The way to solve the problem of not exposing your business logic to competitors is to separate your app into an open-source engine and a bunch of declarative business-rule schemas that you keep secret. Databases work this way, and websites (the web pages and CGIs are the schema). Many vertical apps can be partitioned this way too -- in fact, for things like tax-preparation software they almost have to be, because the complexity overhead of hacking executable code every time the rules change is too high.

Steve thinks the differences between Apache and Mozilla are bigger than they are. In fact, the core groups of both projects are full-time pros being funded by large users of the software.

So, let's address Steve's objections point by point:

For embedded software, OSS has the following problems:

• It can't be scheduled; timely delivery can't be relied on.

  Timely delivery can't be relied on for any software; see De Marco and Lister's excellent book Peopleware: Productive Projects and Teams on the delusion of deadlines, especially the empirical evidence that the "wake me up when it's done" strategy of not setting them actually gets your project done faster (also the implication of a recent Harvard Business School study of software project outcomes).

  Open source is at least not noticeably worse than closed-source on this axis. Arguably it's better, because the rapid release cycles allow users to pick up on project results as soon as they're good enough.

• Debugging requires access to custom hardware which usually can't easily be accessed across the net.

  There aren't good solutions to this problem yet, but the increasing use of "overpowered" 32-bit processors using standard busses is tending to reduce it in scope. The development tools and interface hardware used in embedded stuff are rapidly getting more generic and closer to what's used in general-purpose computers.

• Active participation even for junior people requires substantial amounts of project-specific knowledge which isn't easily acquired, especially remotely.

  This one puzzles me, because I think Steve ought to be right about it -- but I'm not hearing the kinds of noises that I'd hear if it were slowing down the move to Linux and open source significantly.

  At least part of the answer is that embedded-systems work is getting de-skilled in a particular sense -- more of it's being done by application specialists who are training up to the required level of programming, rather than programmers who have acquired expensive application-specific knowledge.

• A great deal of proprietary information is usually involved in the process, and if that's released the company can be seriously harmed.

  It's a question of tradeoffs. As RealNetworks found out when costing its Nokia contract, the choice is increasingly between giving up control of some of your proprietary IP and being too resource-bound to ship at all.

  There is no market for secrecy. There's a market for product. If you can't ship product, or your customers aren't confident that you can maintain it after shipping, all that proprietary IP amounts to is a millstone around your neck.

  There will be more stories like RTSP in the future. Count on it. In fact, the day will come when most of your contract partners simply won't accept the business risks of having someone else hold proprietary rights on the embedded software they use.

• It's nearly impossible to do embedded software without common impromptu face-to-face meetings with co-workers, either to ask questions or to brainstorm. Doing this electronically is sufficiently different as to not be practical.

  Yeah. They used to think that about operating systems, too. Obviously the Linux kernel is impossible, and therefore doesn't exist.

  (At which point Oolon Colluphid disappeared in a puff of logic.)

For vertical apps, the objections are:

• Security, security, security. You want me to trust my billing system to code written by anyone who happens to come along and volunteer to work on it, without any kind of check of credentials or checks on trustworthiness?

  One of the lessons the business world has been absorbing is that open-source projects are dramatically more secure than their closed-source competition -- anybody who compares the Bugtraq records on Apache vs. ISS defacements, or Linux vs. Windows remote exploits, will notice that real fast.

  It's not hard to understand why this is -- I've found that even corporate executives grok the theory pretty quickly. I won't do the whole argument here, but this article on Kerckhoff's Law holds the crucial clue. When you rely on the obscurity of source code for security, it means that the bad guys find the bugs faster than you can plug them -- there are more of them, and they have entropy on their side. Open source evens the odds for the good guys.

• Recruitment: for most of the kind of people involved in OSS, vertical apps are boring. (Unless they want to figure out how to steal from it.)

  This remains a problem. On the other hand, open source makes it easier to train domain specialists to be good enough programmers to get the job done. It's easier for physicists to learn to hack than it is for hackers to learn physics.

• It takes a lot of knowledge of the specific aspects of the problem to make a significant contribution, which means things like observing the actual process of guests checking in at the front desk of the hotel.

  This just reinforces the tendency for vertical-app developers to be obsessives about something else who learn to program, rather than obsessives about programming who learn something else.

  Professional programmers tend to bridle at this thought. Well, better learn to live with it. As software becomes more pervasive, the amount of it done by application-specialist "amateurs" is going to increase.

• The industry is full of horror stories of vertical apps which ran badly over budget and over schedule; the idea scares the hell out of business people. They're unlikely to be very enthused by the use of a process which by its nature *cannot* be reliably scheduled. (Remember that Mozilla ran two years long.)

  Schedules -- and the belief that deadlines make software happen faster -- are a delusion in the mind of management, one not supported by the actual evidence about project outcomes. This delusion is so entrenched that managers fail to interpret the 70% rate of project failures correctly. It's as if people were so determined to believe the Earth is flat that they ignore what their eyes tell them when ships sink over the horizon.

  No software larger than toy programs can be scheduled. Tactics aimed at doing so normally have the actual effect of increasing the time to market. `Aggressive' schedules effectively guarantee failure. The sooner we learn these objective truths, and that the illusion of control that schedules give is not worth the real costs, the sooner rates of outright project failure will dip below 70%.

  Go read Peopleware. Now.

For short life apps:

• Schedule is everything. If you're six months late, you're dead.

  See above. There are reasons open sourcing is less applicable to short-life applications, but this turns out not to be one of them.

• Secrecy is everything else. If you're on time but your competitor knows what you're doing a year ahead, he'll wipe you out.

  This argument has more force for short-life apps than for Steve's other categories, but remember that increasingly the alternative to open source is not being able to ship at all. Your competitor is in the same boat you are.

• How do you make money selling what anyone can get for free from any developer? If your product was developed out in the open, who exactly buys it afterwards?

  Steve has a stronger point here. It's one that people used to think applied to almost all software, but which turns out to be mainly a problem for short-life apps. Actually the distinguishing characteristic isn't expected lifetime per se, but something correlated with it -- whether the product needs continued downstream work (maintainance and upgrades) or not.

  Long-life, high-maintainance apps create niches for service businesses. That's the main way you make money in an open-source world. It's harder to make that work with a short-life app. Sometimes it's immpossible. Life is hard.

For long life apps:

• Will the participants be willing to work on what our marketing analysis says we need, or will they insist that they know what is required and try to add that instead? We don't need feature creep, or people trying to change the direction we're moving.

  In open-source projects, the function of "marketing analysis" tends to be taken be direct interaction with the user community. We find we do better work without a bunch of marketroids getting between us and our customers.

• There is major learning curve involved in making a reasonable contribution to these kinds of programs; you don't learn how a circuit board router works in a few days of study. In most cases you have to be conversant with the way that the package's customers do what they do, and most programmers don't know these things and can't easily learn them.

  See my previous remarks about application specialists and the democratization of programming. And every time you're tempted to say "But they couldn't possibly get away with that in application area X" remember that they once said that about all the areas where open source now dominates.

  It's just not smart to bet against the hackers. Not smart at all. We generally end up having the last laugh on the naysayers. As recently as 1990, "serious analysts" laughed at the idea of ubiquitous Internet. As late as 1996, they said Unix was dead. We showed them -- and there are more of us now, with better tools, than ever.

Steve is right that one of the most effective ways to head off bugs is to have a core group of professional engineers do a clean design. Where he's mistaken is in believing this truth has anything to tell us about open vs. closed development. Us open-source guys, it turns out, are really good at clean design.

This something to do with the fact that, as individuals, we tend to be exceptionally capable and self-motivated -- an elite selected by dedication to the art of programming. It has more to do with not having managers and marketroids pissing in the soup constantly, telling us what tools to use, imposing insane deadlines, demanding endless checklist features that don't actually benefit anyone.

But mostly it has to do with the ruthless, invaluable pressure of peer review -- the knowledge that every design decision we make will be examined by thousands of people who may well be smarter than we are, and if we fail the test our effort will be pitilessly discarded. In that kind of environment, you get good or you get gone.