Armed and Dangerous has moved to http://esr.ibiblio.org
This blog has moved. Please update your links to point to http://esr.ibiblio.org/
Monday, July 29, 2002
Right back at ya, Captain:
Last Saturday morning in San Diego I had breakfast with Steven den Beste, the redoubtable captain of U.S.S. Clueless. One of the side-effects of that meeting was a long critique of open-source development. Herewith my response.
Steve and I agree on the scaling problem that has pushed software development efforts to the ragged edge of what is sustainable even by corporations with lots of money. Software project sizes are roughly doubling every eighteen months, and for reasons Steve alluded to the expected bug count per thousand lines is actually rising.
My assertion is that software development has reached a scale at which (a) even large corporations can often no longer afford to field enough developers to be effective at today's project scales, and (b) traditional methods of software quality assurance (ranging from formal methods to internal walkthroughs) are no longer effective. The only development organizations that seem to thrive on today's complexity regime are open-source teams.
Note that I am not claiming that open source is a silver bullet for the software-complexity problem. There are no silver bullets, no permanent solutions. What I am claiming is that at the leading edge of large-scale software, closed-source development doesn't work any more. The future belongs to open source plus whatever other practices and technologies we learn to use with it to develop at ever-higher scales of complexity.
Steve's analysis of the open-source phenomenon is very intelligent, but doesn't quite understand either the mode of organization, the associated technology, or the factional politics within the movement. Diagnostic of the slight disconnect is when he writes "For [the zealots], the only true "Open Source" is governed by the strong form of the GPL, and all other forms and licenses are harmful dilution of the concept." In fact, the people he's talking about reject the term "open source" entirely and insist on the ideologically-loaded term "free software".
A more serious error is when he writes "It is plausible that an OSS project would require each participant to sign an NDA before being given access to the source." It is not plausible. The licenses and community values of the open-source community would not permit this. His two bullet points characterizing open source are missing its most important characteristic: the entire practice is designed to facilitate scrutiny by people with no institutional or contract relationship to the core development team. The astringent effect of peer review by people who have nothing to lose by reporting bugs is precisely the point of the whole game.
Steve doesn't undertand the importance or the power of this effect. This slightly skews his whole essay; much of it is talking past what open-source people do, rather than addressing us. He's also unaware of a lot of the real-world evidence for the success of the method. Some of the things he thinks are technologically or economically impossible are actually being done, routinely.
He's correct when he says that most contributors are self-selected and self-motivated. He overestimates the cost of training newbies, though. They self-train; normally, the first time a core developer hears from a newbie is typically when the newbie sends a patch -- self-evidence that the newbie has already acquired a critical level of knowledge about the software. The "sink or swim" method turns out to work, and work well.
It's incorrect to imply, as he does, that open-source development is unsustainable because the people doing it are flaky amateurs. Steve hasn't absorbed the implications of the Boston Consulting Group study that shows that about 40% of contributors to core projects are professionals getting paid for working on open source by patrons who need to use the results. In fact, what the open-source community is evolving into is something very like a huge machine for bringing newbies into apprenticeship contact with experienced developers and professionalizing both groups.
He also writes "OSS by its nature tends to be reactive rather than predictive. It doesn't look into the future, try to predict a problem which doesn't exist now but will exist then, and be ready with a solution. Rather, it tends to see problems that exist now and work on solutions for them." This is false -- or, at any rate, no more true than it is for closed-source development.
The open-source community built the Web and the Internet before it had acquired a name for itself and full consciousness of its own practices. Today, the cutting-edge work in operating systems languages, desktop user interfaces, relational databases and many other areas is being done either within the open-source community or in cooperation with it by academics. These prodigious efforts of imagination dwarf any "prediction" produced by closed-source software development in the last two decades.
Steve's "open source is reactive" claim strikes me as ironically funny, because I can remember when the standard knock on my crowd was that we're great at innovation but can't actually field product. How quickly they forget...
He's right enough about the difficulty of planning and high cost of face-to-face meetings, though. These are real problems. It's a testimony to the power of our practices that we manage to ship large volumes of high-quality software despite these obstacles.
What Steve called "player-killer" tactics have been tried -- there was a famous incident a few years back in which a TCP-wrappers distribution was Trojaned. The crack was detected and the community warned within hours. The black hats don't seem to bother trying this any more; our reaction time is too fast for that game to be very rewarding. The technical design of Linux helps here in ways that I won't go into here -- suffice it to say that it's intrinsically much harder to get a Trojan to do anything interesting than it is under Windows or other single-user operating systems.
So far, the supply of open-source developers seems to be pretty elastic -- we're not limited much by lacking bodies. Other factors loom much larger; patents, the DMCA, intrinsically hard technical problems. I don't understand why this is as well as I'd like to, but the facts are undeniable; the community is ten times the size my wildest high-end scenarios predicted a decade ago and seems to be growing faster as it gets larger.
Steve's whole argument that open-source can't win in embedded systems is very curious, since it predicts exactly the opposite of what is actually happening out there. Linux is taking over in embedded systems -- in fact, many observers would say it has already won that space. If Steve had worked in the field within the last three years he would probably know this.
Here are some data about the demand; the only non-general-purpose open-source software magazine in existence is the Linux Embedded Systems Journal. Open-source embedded developers like Monta Vista Software are bucking the recession by growing like crazy. The first cell-phone prototype running entirely open-source software just entered beta testing.
I was in California to meet Steve partly because Real Networks wanted me to be on stage when they announced the open-sourcing of their RTSP engine. Their CEO, Rob Glaser, was quite frank about the immediate business reasons: they needed to get ports to forty different Nokia cellphones and just couldn't figure out how to muster the resources for that short of inviting every stakeholder on the planet to hack the problem. Scaling bites. Hard.
In fact, some of the very characteristics that Steve thinks make embedded systems like cellphones safe for closed development seems to be the factors that are driving increased open-sourcing. The close tie to hardware actually decreases the value of secrecy, because it means the software is typically not easily re-usable by hardware competitors. Thus open sourcing is often a great way to recruit help from customer engineers without a real downside risk of plagiarism.
In fact, it's an open secret in the industry that the most important reason most closed-source embedded and driver software remains closed is not nerves about plagiarism but fear of patent audits on the source code. Graphics-card manufacturers, in particular, routinely swipe patented techniques from their competitors and bury them in binaries. (This is generally believed to be the reason nVidia's drivers aren't open.)
Another trend that's driving Linux and open-sourcing in embedded stuff is the shift from specialty embedded 8-bit processors to 32-bit chips with general-purpose architectures. Turns out the development costs for getting stuff to run on the 8-bit chips are sickeningly high and rising -- partly because the few wizards who can do good work on that hardware are expensive. The incremental cost for smarter hardware has dropped a lot; it's now cheaper to embed general-purpose chips running Linux because it means you have a larger, less expensive talent pool that can program them. Also, when your developers aren't fighting hardware limits as hard, you get better time to market (which, as Steve observes, is critical).
Steve is right about the comparative difficulty of applying open-source methods to vertical applications. But the difficulty is only comparative; it's happening anyway. The metalab archive carries a point-of-sale system for pizza parlors. I know of another case in which a Canadian auto dealership built specialized accounting software for their business and open-sourced it. The reasons? Same as usual; they wanted to lay off as much as possible of the development and maintainance cost on their competitors.
This is the same co-opetition logic that makes the Apache Software Foundation work -- it's just as powerful for vertical apps, though less obviously so. Each sponsoring company sees a higher payoff from having the software at a small fraction of the manpower cost for a complete in-house development. The method spreads risk in a way beneficial to all parties, too, because the ability of separate companies to sustain development tends to be uncorrelated -- unless they all sink, the project endures.
The way to solve the problem of not exposing your business logic to competitors is to separate your app into an open-source engine and a bunch of declarative business-rule schemas that you keep secret. Databases work this way, and websites (the web pages and CGIs are the schema). Many vertical apps can be partitioned this way too -- in fact, for things like tax-preparation software they almost have to be, because the complexity overhead of hacking executable code every time the rules change is too high.
Steve thinks the differences between Apache and Mozilla are bigger than they are. In fact, the core groups of both projects are full-time pros being funded by large users of the software.
So, let's address Steve's objections point by point:
For embedded software, OSS has the following problems:
For vertical apps, the objections are:
For short life apps:
For long life apps:
Steve is right that one of the most effective ways to head off bugs is to have a core group of professional engineers do a clean design. Where he's mistaken is in believing this truth has anything to tell us about open vs. closed development. Us open-source guys, it turns out, are really good at clean design.
This something to do with the fact that, as individuals, we tend to be exceptionally capable and self-motivated -- an elite selected by dedication to the art of programming. It has more to do with not having managers and marketroids pissing in the soup constantly, telling us what tools to use, imposing insane deadlines, demanding endless checklist features that don't actually benefit anyone.
But mostly it has to do with the ruthless, invaluable pressure of peer review -- the knowledge that every design decision we make will be examined by thousands of people who may well be smarter than we are, and if we fail the test our effort will be pitilessly discarded. In that kind of environment, you get good or you get gone.
posted by Eric at 11:48 PM