The Delusion of Expertise:

I learned something this weekend about the high cost of the subtle delusion that creative technical problem-solving is the preserve of a priesthood of experts, using powers and perceptions beyond the ken of ordinary human beings.

Terry Pratchett is the author of the Discworld series of satirical fantasies. He is — and I don't say this lightly, or without having given the matter thought and study — quite probably the most consistently excellent writer of intelligent humor in the last century in English. One has to go back as far as P.G. Wodehouse or Mark Twain to find an obvious equal in consistent quality, volume, and sly wisdom.

I've been a fan of Terry's since before his first Discworld novel; I'm one of the few people who remembers Strata, his 1981 first experiment with the disc-world concept. The man has been something like a long-term acquaintance of mine for ten years — one of those people you'd like to call a friend, and who you think would like to call you a friend, if the two of you ever arranged enough concentrated hang time to get that close. But we're both damn busy people, and live five thousand miles apart.

This weekend, Terry and I were both guests of honor at a hybrid SF convention and Linux conference called Penguicon held in Warren, Michigan. We finally got our hang time. Among other things, I taught Terry how to shoot pistols. He loves shooter games, but as a British resident his opportunities to play with real firearms are strictly limited. (I can report that Terry handled my .45 semi with remarkable competence and steadiness for a first-timer. I can also report that this surprised me not at all.)

During Terry's Guest-of-Honor speech, he revealed his past as (he thought) a failed hacker. It turns out that back in the 1970s Terry used to wire up elaborate computerized gadgets from Timex Sinclair computers. One of his projects used a primitive memory chip that had light-sensitive gates to build a sort of perceptron that could actually see the difference between a circle and a cross. His magnum opus was a weather station that would log readings of temperature and barometric pressure overnight and deliver weather reports through a voice synthesizer.

But the most astonishing part of the speech was the followup in which Terry told us that despite his keen interest and elaborate homebrewing, he didn't become a programmer or a hardware tech because he thought techies had to know mathematics, which he thought he had no talent for. He then revealed that he thought of his projects as a sort of bad imitation of programming, because his hardware and software designs were total lash-ups and he never really knew what he was doing.

I couldn't stand it. "And you think it was any different for us?" I called out. The audience laughed and Terry passed off the remark with a quip. But I was just boggled. Because I know that almost all really bright techies start out that way, as compulsive tinkerers who blundered around learning by experience before they acquired systematic knowledge. "Oh ye gods and little fishes", I thought to myself, "Terry is a hacker!"

Yes, I thought 'is' — even if Terry hasn't actually tinkered any computer software or hardware in a quarter-century. Being a hacker is expressed through skills and projects, but it's really a kind of attitude or mental stance that, once acquired, is never really lost. It's a kind of intense, omnivorous playfulness that tends to color everything a person does.

It burst upon me that Terry Pratchett has the hacker nature. Which, actually, explains something that has mildly puzzled me for years. Terry has a huge following in the hacker community — knowing his books is something close to basic cultural literacy for Internet geeks. One is actually hard-put to think of any other writer for whom this is as true. The question this has aways raised for me is: why Terry, rather than some hard-SF writer whose work explicitly celebrates the technologies we play with?

The answer now seems clear. Terry's hackerness has leaked into his writing somehow, modulating the quality of the humor. Behind the drollery, I and my peers worldwide have accurately scented a mind like our own.

I said some of this the following day, when I ran into Terry surrounded by about fifty eager fans in a hallway. The nature of the conference was such that about three-quarters of them were hackers, many faces I recognized. I brought up the topic again, emphasizing that the sort of playful improvisation he'd been describing was very normal for us, and that I thought it was kind of sad he'd been blocked by the belief that hackers need to know mathematics, because about all we ever use is some pieces of set theory, graph theory, combinatorics, and Boolean algebra. No calculus at all.

Terry then admitted that he had at one point independently re-invented Boolean algebra. I didn't find this surprising — I did that myself when I was about fifteen; I didn't mention this, though, because the moment was about Terry's mind and not mine. I think reinventing Boolean algebra is probably something a lot of bright proto-hackers do.

"Terry," I said, fully conscious of the peculiar authority I wield on this point as the custodian of the Jargon File, the how-to on How To Become A Hacker and several other related documents, "you are a hacker!"

The crowd agreed enthusiastically. Somebody handed Terry one of the "Geek" badge ribbons the convention had made for attendees who wanted to identify themselves as coming from the Linux/programming side. Much laughter ensued when it was discovered that the stickum on the ribbon had lost its virtue, and a nearby hacker had to ceremonially affix the thing to Terry's badge holder with a piece of duct tape.

Terry actually choked up a little while this was going on, and I don't think there was anyone there who didn't understand why. To the kind of teenager and young man he must have been — bright, curious, creative, proud of his own ability — it must have been very painful to conclude that he would never cut it as the techie he so obviously wanted to be. He ended up doing public-relations work for the British nuclear-power industry instead.

The whole sequence of events left me feeling delighted that I and my friends could deliver the affirmation Terry had deserved so long ago. But also — and here we come to the real point of this essay — I felt very angry at the system that had fed the young Terry such a huge load of cobblers about the nature of what programmers and hardware designers do.

I'm not referring to the obvious garbage about needing a brain-bending amount of mathematics. No; they fed Terry something much subtler and more crippling, a belief that real techies actually know what they're doing. The delusion of expertise.

The truth is that programmers only know what they're doing when the job is not very interesting. When you're breaking new ground in any technical field, exploration and improvisation is the nature of the game. Your designs are going to be lash-ups because you don't yet know any better and neither does anyone else. Systematization comes later, with the second system, during the re-write and the re-think. Einstein had it right; imagination is more valuable than knowledge, and people like Terry with a demonstrated ability to creatively wing it make far better hackers than analytically smart but unimaginative people who can only follow procedures.

The thought that Terry may have spent thirty years of working days grinding out press releases for the Central Electricity Generating Board because he didn't know this, rather than following his dreams into astronomy or programming or hardware design, bothers the crap out of me. If Terry was bright enough to invent Boolean algebra, he was bright enough to cut it in any of these fields. The educational system failed him by putting artificial requirements in his way and making him believe they were natural ones. It failed him even more fundamentally by teaching him a falsehood about the nature of expertise.

In doing this, it failed all of us. How many bright kids with first-class minds, I wonder, end up under-employed because of crap like this? How much creative potential are we losing?

OK, some might answer, so we got the Discworld fantasies instead...that ain't exactly chopped liver. The thing is, I'm not sure that was actually a trade-off. I'm enough of a writer myself to believe that you can't block a writing talent like Terry's merely by dropping him into a more demanding day job. It will come out.

On the other hand, one thing I am sure of is that you don't need intelligence or talents like Terry's just to do PR. One way or another, this man was going to do something with more lasting effects than soothing British farmers about radiation leaks. Inventing one of the funniest alternate worlds of the last hundred years during your free time is nice, and I devoutly hope he will get to keep doing it for decades to come — but in a society that valued and nurtured genius properly, I think Terry might have helped re-imagine the real world just as radically during his day job.

But he didn't. Tot it up to the cost of taking creativity too seriously, of undervaluing improvisation and play and imagination. And wonder how much else that error has cost us.