Short Take: The Autocracy of Autonomous Systems

Autonomous systems are all around us. To large parts of the population, they are like the mythical creatures of yore of great power, great knowledge, and great control over us. Some in the wide world ask with growing consternation: “Are we still going to be in charge? Are we still the driver at the wheels, literally as well as figuratively?” Or are we slowly, unobtrusively ushering in the autocracy of autonomous systems. Such an autocracy portends a world where decisions are made by these systems with the ceding of control by us humans and lack of accountability of these systems, or by proxy, their developers. These deep dark fears have been voiced eloquently through the ages through many culture-defining books: Aldous Huxley’s “Brave New World” (1932), George Orwell’s “Nineteen eighty four” (1949), and, Kurt Vonnegut’s “Player Piano” (1952), to name the three at the top of my list.

My belief based on working on reliability and security of some such autonomous systems is that we are firmly in control. But also that we the developers of such systems, need to take some conscious decisions to make sure we do not usher in the age of autocracy of autonomous systems. In this article, I will discuss the first aspect out of three aspects of the topic.

1. What some feared future scenarios are

2. What we can do technologically to prevent such scenarios
3. What we can do policy-wise to prevent such scenarios

Some Dystopian Scenarios Please

Jeremy had had a great week, a great way to start the summer of 2050. He had delivered his software module on time and with all the promised functionality. His global team had come through with delivering the code with zero defects and ahead of its competitor. Getting a flying car had been in the popular imagination for 100 years now and their company had been the first to deliver on that imagination. And now this had become a hyper competitive industry with every point gain in market share being painfully won with a host of companies all running at breakneck speed. So it was good that they had shipped this peer-to-peer communication mode of the passenger drone product before anybody else. This would let the drones figure out their flight paths without careful a priori planning and on the go as events unfolded. Better still, the human overlords of these machines would not have to get involved, at all, and could not get involved even if they got it into their head to peek inside.

Jeremy also realized that a great week can turn into a nightmare week with 5 lines of code. The operators of the passenger drones started getting increasingly frantic calls almost all at the same time. The vehicle would be in mid-air and would stall and no amount of lever pressing and knob twiddling by the passengers inside would do any good. As the aircraft, or as it is more common to call such things these days, simply vehicles stalled and hovered above ground and as the gas gauge moved toward empty, the calls became increasingly frantic. And then there was the robotic voice that would come on over the peer-to-peer communication channel saying in how many minutes the vehicle would plummet to the ground, destroying all inside and any who happen to be on the ground on its path. But wait, there was a solution. A moderate amount of money paid in the digital currency that was the flavor of the day would get the vehicle back in motion. And it hit Jeremy which 5 lines of code had left the door open, though just a notch, for such criminal activity.

So Jeremy wants to go to the code repo, do the change himself (yes, as a team leader, he could still do it), and push it out to the millions of vehicles up in the air around the world. The online software upgrade feature would be put to its test but it should come through. But he had the sinking feeling in the pit of his stomach when the code repo rejected his credentials and would not let him in. His efforts to call members of his team on his super-smart calling device did not work either: there was the same robotic voice telling him his outgoing calling privileges had been suspended. So the vehicles running his code would continue to remain suspended in mid-air till the requisite amount of currency changed digital hands.

So what are some guiding principles for us technologists to avoid dystopian scenarios like the one above, and even worse. Autonomy will be around us in ever increasing pervasiveness. How do we make it less a toxic miasma and more an enabling springboard? We will theorize about some high-level principles in the next post.