The programming assembly-line
I read Matthew Crawford’s book, “Shop Class as Soul Craft”1, nearly 2 years ago,
and in that time no other book has been on my mind more. There are things about
it that ring deeply true to me, and others I think are deeply
misunderstood. What’s worse, the topic is so raw and close to my heart that the
thought of writing about it makes me singe. Anyway, here goes.
The book is a philosophical meditation on the concept of work. What makes work
meaningful, satisfying and enjoyable? What makes it dull, boring and soul
crushing? He talks about how blue-collar work can be intellectually challenging
and satisfying, and how white-collar “knowledge work” can be repetitive and
stultifying. Whether or not you agree with his conclusions, his examination of
the underlying issues is stimulating, and is peppered with colorful and
entertaining episodes from his own life. He was the director of a Washington
think tank, but ended up as the proprietor of a bike repair shop specializing in
Japanese motor bikes. So he speaks from experience on both sides of the fence,
which already gives him more credibility than most authors on the topic. If you
want a shorter introduction to some of the core issues discussed in the book,
read these two2 essays3 by him.
The primary culprit for the dehumanization of work, according to Crawford, is
Taylorism. So we must look at that first, and then we will loop back to what all
this has to do with programmers and programming.
Taylorism
Taylorism is so deeply ingrained in every aspect of not just modern commerce but
life in general that it is almost impossible to identify it. Just like a fish
doesn’t even know that it is in water. If you haven’t read his “Principles of
Scientific Management”4, it is a short and easy read, and now available free on
Project Gutenberg. After reading it, you will realize that it is one of the
founding documents of modern life, as well as the progenitor of the “business
book” category.
Briefly, Taylorism has two central tenets:
- Measurement: associate metrics with all aspects of work.
- The separation of thinking and doing: An educated class of managers
measures and plans all the work, and is responsible for the overall process,
while a class of laborers carries out the implementation of those plans.
Taylor was convinced that the laborers he was studying were
“soldiering”—knowingly working slowly to maintain work and wages for all their
brethren.
Consider this dialogue that Taylor had with a worker in order to
convince him to move more pig iron:
“Schmidt, are you a high-priced man?”
“Vell, I don’t know vat you mean.”
“Oh, come now, you answer my questions. what I want to find out is whether you
are a high-priced man or one of these cheap fellows here. What I want to find
out is whether you want to earn $1.85 a day or whether you are satisfied with
$1.15, just the same as all those cheap fellows are getting.”
“Did I vant $1.85 a day? Vas dot a high-priced man? Vell, yes, I vas a
high-priced man.”
“Well, if you are a high-priced man, you will load that pig iron on that car
to-morrow for $1.85. Now do wake up and answer my question. Tell me whether you
are a high-priced man or not.”
“Vell—did I got $1.85 for loading dot pig iron on dot car to-morrow?”
“Yes, of course you do, and you get $1.85 for loading a pile like that every day
right through the year. That is what a high-priced man does, and you know it
just as well as I do.”
“Vell, den, I vas a high-priced man.”
“Now, hold on, hold on. You know just as well as I do that a high-priced man
has to do exactly as he’s told from morning till night. Well, if you are a
high-priced man, you will do exactly as this man tells you to-morrow, from
morning till night. When he tells you to pick up a pig and walk, you pick it up
and you walk, and when he tells you to sit down and rest, you sit down. You do
that right straight through the day. And what’s more, no back talk. Now a
high-priced man does just what he’s told to do, and no back talk. Do you
understand that? When this man tells you to walk, you walk; when he tells you to
sit down, you sit down, and you don’t talk back at him. Now you come on to work
here to-morrow morning and I’ll know before night whether you are really a
high-priced man or not.”
(emphasis mine)
The net effect of all this was that the worker could, under the appropriate
supervision, now load 47 tons of iron a day, up from his previous 12 1/2 tons a
day. He got a 60% pay raise, while the company moving the iron decreased its
labor per ton by about the same percentage (from 9.2 c/ton to 3.9 c/ton). But
another windfall for the company was the saving in time. Previously, moving 100
tons using 2 workers would have taken 4 days, whereas now it would take just a
little over 1 day.
The assembly line
The earliest well-known enterprise to go whole-hog Taylorist was Ford, with its
assembly line for manufacturing cars. The world never looked back.
The assembly line with narrowly specialized roles for each worker is a
consequence of the thinking/doing separation. The smarts are in the process, the
assembly line. The workers operating it are fungible. Of course, the workers did
not like this at all. Skilled craftsmen that built entire artifacts were
replaced with machines that required only semi-skilled operators churning out
one tiny part of the whole. The pace of their work was determined by the speed
of the assembly-line, which was set by managers. They had lost all autonomy and
control over their work. Their work had been de-skilled. The transfer of power
from skilled, knowledgeable craft workers to “the management” was sealed. Ford
made this palatable by paying his workers high wages.5
In his book, Dan Pink6 talks about the three factors that make work
meaningful: autonomy, mastery and purpose. The fungible non-distinguishable
worker manning the assembly line was stripped of all three.
The drive to specialize
The seeds of Taylorism go back to 1817, when David Ricardo outlined the
principle of comparitive advantage. He explained how two parties could gain by
making goods they were efficient at and then trading with each other, even if
each of them could make the traded good on its own. So even if the other party
is worse than you at making something, you are better off trading and focusing on
your specialty, because of the saved opportunity costs. (See the Wikipedia
article for a number of concrete examples7).
The undeniable math of comparitive advantage is the natural driver for
specialization. Taylor brought the concept down from the level of countries to
individual workers.
What does this have to do with writing software?
The field of software engineering was built on firmly Taylorist principles. It
had no other choice, because that was the default school of thought.
In 1968, NATO called a conference on the subject of “software engineering” (the
field did not have a name back then, and NATO’s pick of terminology
stuck)8. The industry was having a lot of trouble creating large and complex
software, and it was about time something was done about it.
Doug McIlroy expressed what became the rallying cry for the field:
“The software industry is not industrialized”!9 He said:
Software production today appears in the scale of industrialization somewhere
below the more backward construction industries. I think its proper place is
considerably higher, and would like to investigate the prospects for
mass-production techniques in software. In the phrase “mass production
techniques,” my emphasis is on “techniques” and not on mass production plain.
Of course mass production, in the sense of limitless replication of a prototype,
is trivial for software. But certain ideas from industrial technique I claim
are relevant. The idea of subassemblies carries over directly and is well
exploited. The idea of interchangeable parts corresponds roughly to our term
`modularity,’ and is fitfully respected. The idea of machine tools has an
analogue in assembly programs and compilers. Yet this fragile analogy is belied
when we seek for analogues of other tangible symbols of mass production. There
do not exist manufacturers of standard parts, much less catalogues of standard
parts. One may not order parts to individual specifications of size,
ruggedness, speed, capacity, precision or character set.
More than 40 years later that still remains the Holy Grail of software
engineering.
McIlroy envisioned a world where we would be constructing software by picking
components off a shelf, and snapping them together like Legos. The hard work
would be building the right blocks, and then it would be easy to snap them
together.
Later, Osterweil played a neat recursive trick and argued that the process of
making software could be expressed as software too10. The implication was that
it could be executed, just like software.
In some ways this vision has come true, in other ways, it hasn’t.
The biggest example of how the dream of software components has come to pass is
web applications. All the components are quite well-understood and, by now,
standard. The database. The object-relational mapping. The web server. The
Javascript frontend. The webapp developer can pick and choose from a wide
variety of choices for each of those. In fact, some grumble that the number of
choices adds complexity. All the same, unless you’re building something that
must scale to the planet (like Google, Amazon etc), it is ridiculously easy to
cobble together a web application that meets 80% of your vision within a day.
Post-modern programming11 starts with Googling, and ends with glue code.
In fact, this componentization has come far enough that many developers complain
that the plumbing of pre-existing parts has drained the joy and skill out of
their profession12. Does that sound familiar? The
craftsman-turned-assembly-line worker of the 20th century had exactly the same
lament!
John D. Cook argues that calling somebody “plumber programmer”13 is actually a
compliment, because it’s the plumbing and connecting and impedance matching that
is hard. But I suspect today that’s true because it’s only the plumbing that is
left to do.
How many times can you build a cookie-cutter CRUD app with a web frontend that
fetches and displays data from a SQL backend before you feel like all you are
doing is cranking a lever? If your Java IDE generates half your code for you,
and you auto-complete (Ctrl-Space!) most of the rest, do you feel like you are
operating machinery, and not really engaging in expression?14
The pyramid
In other ways, McIlroy’s vision is still a dream.
Massive development effort still goes into applications that are young and not
standardized. There are no components that will let you build your own web-scale
search engine quickly, for example.
Programmers are certainly not fungible assembly-line operators15.
As in all things in life, there is a hierarchy to these things. There is a small
number of elite wizard programmers who do write significant things from
scratch. Think of those who have implemented language compilers and runtimes,
database engines, search engines, operating system kernels and the
like. Unfortunately, such projects come along rarely, and the bar for
contribution is extremely high.
(By the way, if you are tired of plumbing together databases and PHP and really
want to experience the wild West of programming, go contribute patches to the
Linux kernel. No fancy languages or libraries that make you soft in
userspace. Just plain old C.)
Some problems, such as building webapps backed by databases, are well
understood, and their solutions have become templatized. Others, such as
indexing a billion documents, or storing a bazillion petabytes across a
gajillion machines, are at the edge of our knowledge, and their solutions are
still being actively explored.
If you think you are writing cookie-cutter code, it is because you are working
on a cookie-cutter problem.
Gratitude
But we must all realize that at the end of the day, we work on something that is
in economic demand (in other words, software engineers are well-paid),
intellectually challenging (at least, on most days) and in some small way
improves the world and helps people. Even all this self-examination and
criticism is a luxury that many do not have.
So, be grateful.
