A few months ago, I was driving through Los Angeles when the Waze app on my phone told me to take the Stadium Way exit off the 110 freeway. About five other cars peeled off with me, and we became a caravan, snaking through side streets and back onto the freeway a few miles later. I knew Waze had to be in charge of us, since Waze is the navigation app of choice in Los Angeles, and it was beyond coincidence that all these cars took the same wild maze run through streets only locals knew well.
What was Waze up to here, besides offering its users (or a subset of them) a way around a jam? Was it optimizing traffic by taking some cars off the highway and leaving others on? Running an experiment only some AI understood? There was no way to tell. I doubt anyone at Waze could say exactly what was going on either. Algorithms are like that. So are the large and constantly changing data sets informing algorithms most of us with mobile devices depend on every day.
In Re-engineering Humanity, Brett Frischmann and Evan Selinger have dug deeply into what's going on behind the "cheap bliss" in our fully connected world.
What they say is that we are all subjects of techno-social engineering. In other words, our algorithmic conveniences are re-making us, much as the technologies and techniques of agriculture re-makes farm animals. And, as with farming, there's an extraction business behind a lot of it.
They say "humanity's techno-social dilemma" is that "companies, institutions, and designers regularly treat us as programmable objects through personalized technologies that are attuned to our personal histories, present behavior and feelings, and predicted futures."
And we are not innocent of complicity in this. "We outsource memory, decision-making and even our interpersonal relations...we rely on the techno-social engineers' tools to train ourselves, and in doing so, let ourselves be trained."
There are obvious benefits to "delegating physical, cognitive, emotional and ethical labor to a third party", such as Waze, but there are downsides, which Brett and Evan number: 1) passivity, 2) decreased agency, 3) decreased responsibility, 4) increased ignorance, 5) detachment and 6) decreased independence. On the road to these diminished human states, we have "fetishised computers and idealized computation".
Doing both means "we work on problems best solved by computation", which in turn leads to "the imperialism of instrumental reason and the improper assumption that all problems are comprehensible in the language of computation and thus can be solved with the same set of social and technological tools".
They see today's faith in computational technology as the latest expression of belief in Frederick Taylor's theory of scientific management. More than a theory, Taylorism gave us norms for producing mass-market goods that have been with us for more than a century and have hardly gone away. In The Cluetrain Manifesto (which four of us wrote in 1999), Chris Locke explains:
Taylor's time-and-motion metrics sought to bring regularity and predictability to bear on the increasingly detailed division of labor. Under such a regimen, previously holistic craft expertise rapidly degraded into the mindless execution of single repetitive tasks, with each worker performing only one operation in the overall process. Because of its effect on workers' knowledge, de-skilling is a term strongly associated with mass production. And as skill disappeared, so did the unique voice of the craftsman. The organization was elegantly simple, if not terribly humane.
"At one level", write Brett and Evan, "Taylor's scientific management system is a type of data-dependent technology. Taylorism is one of the best early examples of data-driven innovation, a concept currently in vogue. Taylor's systems included the techniques for both gathering data and putting such data to use in managing people. Taylor's system thus encompassed the surveillance techniques employed by the 'efficiency experts'" of Taylor's time—and of ours.
Surveillance of people is now the norm for nearly every website and app that harvests personal data for use by machines. Privacy, as we've understood it in the physical world since the invention of the loincloth and the door latch, doesn't yet exist. Instead, all we have are the "privacy policies" of corporate entities participating in the data extraction marketplace, plus terms and conditions they compel us to sign, either of which they can change on a whim. Most of the time our only choice is to deny ourselves the convenience of these companies' services or live our lives offline.
Worse is that these are proffered on the Taylorist model, meaning mass-produced.
In "Contracts of Adhesion—Some Thoughts about Freedom of Contract" (Columbia Law Review, July 1943), Friedrich Kessler explained how these shitty non-agreements came to be:
The development of large scale enterprise with its mass production and mass distribution made a new type of contract inevitable—the standardized mass contract. A standardized contract, once its contents have been formulated by a business firm, is used in every bargain dealing with the same product or service. The individuality of the parties which so frequently gave color to the old type of contract has disappeared. The stereotyped contract of today reflects the impersonality of the market....Once the usefulness of these contracts was discovered and perfected in the transportation, insurance, and banking business, their use spread into all other fields of large scale enterprise.
Rather than obsolesce these kinds of contracts, digital technology and the internet allowed companies to automate them. The effects are not good for the human side of each contract. Brett and Evan explain, "Our current online contracting regime is a compelling example of how our legal rules coupled with a specific technological environment can lead us to behave like simple stimulus-response machines—perfectly rational, but also perfectly predictable and ultimately programmable."
In a chapter titled "On Extending Minds and Mind Control", Brett and Evan visit the ways humans are diminished as well as enlarged when programmed by their digital conveniences. To start getting what they mean, it helps to remember that every personal technology we operate—shoes, bikes, pens, books, cars, hammers, airplanes—extend our senses and enlarge our agency. It is no mistake that drivers speak in first-person possessive pronouns about the parts of the cars they operate: my tires, my engine, my fenders. To drive a car is to become one.
But what happens when our senses extend beyond the metal carapace we wear when we drive a car—outward through the unseen systems guiding our selves and every other car on the road? In this state we are not GPS satellites and Google data centers, but rather puppets at the ends of digital strings pulled by AI puppeteers.
We surely appreciate and rely on what they provide us, but we also yield agency in the process of blurring between our automotive selves and a vast system of dependencies, which even if they are doing good things for us, make us less than human—or, in Brett and Evans' words, "simple machines under the control and influence of those in control of technologies". Inevitably, they also say, "traffic engineers will assume the role of social planners." But the traffic engineers they're talking about are not the human kind working for highway departments, but machines run by companies making navigation apps, all of which have purposes beyond providing personal and civic goods.
The challenge Brett and Evan pose in their concluding chapter is "how to sustain the freedom to be off, to be free from techno-social engineering, to live and develop within undetermined techno-social environment". Toward these they set a buffet of possible approaches, three of which can be addressed by the agency of Linux Journal readers, many of whom are engineers by trade:
Then they give a nod to decentralization and go into possible reforms to legal systems.
In the book's last two pages, they also give a nod toward my own work: "Doc Searls and his colleagues at Customer Commons have been working for years on standardized terms for customers to use in managing their relationships with websites and other vendors", noting that the General Data Protection Regulation (GDPR) in Europe at least makes it possible "to imagine alternative contracting practices and more nuanced contractual relationships".
In the book's final paragraph, they say "Doc Searls' dream of customers systematically using contract and related tools to manage their relationships with vendors now seems feasible. It could be an important first step toward flipping the scientific-management-of-consumers script we've become so accustomed to."
The engineers among us have a hard job. But it should help to know how high the stakes are, and how we're already embedded so deeply in a re-engineered dystopia that we can't see how tragically ironic cheap bliss really is.
Doc Searls is a veteran journalist, author and part-time academic who spent more than two decades elsewhere on the Linux Journal masthead before becoming Editor in Chief when the magazine was reborn in January 2018. His two books are The Cluetrain Manifesto, which he co-wrote for Basic Books in 2000 and updated in 2010, and The Intention Economy: When Customers Take Charge, which he wrote for Harvard Business Review Press in 2012. On the academic front, Doc runs ProjectVRM, hosted at Harvard's Berkman Klein Center for Internet and Society, where he served as a fellow from 2006–2010. He was also a visiting scholar at NYU's graduate school of journalism from 2012–2014, and he has been a fellow at UC Santa Barbara's Center for Information Technology and Society since 2006, studying the internet as a form of infrastructure.
