Automation Ethics

Many people start their journey into the ethics of automation and robotics by looking at Asimov's Laws of Robotics.

A robot may not injure a human being or, through inaction, allow a human being to come to harm (etc. etc.)

As I've said before, I believe Asimov's Laws are problematic as a basis for ethical principles. Given that Asimov's stories demonstrate numerous ways in which the Laws don't actually work as intended. I have always regarded Asimov's work as being satirical rather than prescriptive.

While we usually don't want robots to harm people (although some people may argue for this principle to be partially suspended in the event of a "just war"), notions of harm are not straightforward. For example, a robot surgeon would have to cut the patient (minor harm) in order to perform an essential operation (major benefit). How essential or beneficial does the operation need to be, in order to justify it? Is the patient's consent sufficient?

Harm can be individual or collective. One potential harm from automation is that even if it creates wealth overall, it may shift wealth and employment opportunities away from some people, at least in the short term. But perhaps this can be justified in terms of the broader social benefit, or in terms of technological inevitability.

And besides the avoidance of (unnecessary) harm, there are some other principles to think about.

• Human-centred work - Humans should be supported by robots, not the other way around. 

• Whole system solutions - Design the whole system or process, don’t just optimize a robot as a single component.  

• Self-correcting - Ensure that the system is capable of detecting and learning from errors. 

• Open - Providing space for learning and future disruption. Don't just pave the cow-paths.

• Transparent - The internal state and decision-making processes of a robot are accessible to (some) users.  

Let's look at each of these in more detail.


Human-Centred Work

Humans should be supported by robots, not the other way around. So we don't just leave humans to handle the bits and pieces that can't be automated, but try to design coherent and meaningful jobs for humans, with robots to make them more powerful, efficient, and effective.

Organization theorists have identified a number of job characteristics associated with job satisfaction, including skill variety, task identity, task significance, autonomy and feedback. So we should be able to consider how a given automation project affects these characteristics.


Whole Systems

When we take an architectural approach to planning and designing new technology, we can look at the whole system rather than merely trying to optimize a single robotic component.

• Look across the business and technology domains (e.g. POLDAT).
• Look at the total impact of a collection of automated devices, not at each device separately.
• Look at this as a sociotechnical system, involving humans and robots collaborating on the business process.

Self-Correcting

Ensure that the (whole) system is capable of detecting and learning from errors (including near misses).

This typically requires a multi-loop learning process. The machines may handle the inner learning loops, but human intervention will be necessary for the outer loops.
 

Open

Okay, so do you improve the process first and then automate it, or do you automate first? If you search the Internet for "paving the cow-paths", you can find strong opinions on both sides of this argument. But the important point here is that automation shouldn't close down all possibility of future change. Paving the cow-paths may be okay, but not just paving the cow-paths and thinking that's the end of the matter.

In some contexts, this may mean leaving a small proportion of cases to be handled manually, so that human know-how is not completely lost. (Lewis Mumford argued that it is generally beneficial to retain some "craft" production alongside automated "factory" production, as a means to further insight, discovery and invention.)


Transparency

The internal state and decision-making processes of a robot are accessible to (some) users. Provide ways to monitor and explain what the robots are up to, or to provide an audit trail in the event of something going wrong.

---

Related posts

How Soon Might Humans Be Replaced At Work? (November 2015) Could we switch the algorithms off? (July 2017), How many ethical principles? (April 2019), Responsible Transparency (April 2019), Process Automation and Intelligence (August 2019), RPA - Real Value or Painful Experimentation? (August 2019)

Links

Jim Highsmith, Paving Cow Paths (21 June 2005)

Wikipedia

Job Characteristic Theory
Just War Theory

From Enabling Prejudices to Sedimented Principles

In my post From Sedimented Principles to Enabling Prejudices (March 2013)  I distinguished the category of design heuristics from other kinds of principle. Following Gadamer (via Peter Rowe) I call these Enabling Prejudices.

Rowe also uses the concept of Sedimented Principles, which he attributes to the French philosopher Maurice Merleau-Ponty, one of the key figures of phenomenology. As far as I can make out, Merleau-Ponty never used the exact term "sedimented principles", but he does talk a great deal about "sedimentation".

In phenomenology, the word "sedimentation" generally refers to cultural habitations that settle out of awareness into prereflective practices. Something like the "unconscious". (Professor James Morley, personal communication)

"On the basis of past experience, I have learned that doorknobs are to be turned. This ‘knowledge’ has sedimentated into my habitual body. While learning to play the piano, or to dance, I am intensely focused on what I am doing, and subsequently, this ability to play or to dance sedimentates into an habitual disposition." (Stanford Encyclopedia of Philosophy: Merleau-Ponty)

This relates to some notions of tacit knowledge, which is attributed to Michael Polanyi. There are two models that are used in the knowledge management world that talk about tacit/explicit knowledge, and present two slightly different notions of internalization.

• Nonaka's SECI model (Wikipedia)
• Boisot's I-Space (Wikipedia)

Some critics (notably Wilson) regard the SECI model as flawed, arguing that Nonaka has confused Polanyi's notion of tacit knowledge with the much weaker concept of implicit knowledge. There are some deep notions of "unconscious" here, which may produce conceptual traps for the unwary.

Conceptual quibbles aside, there are several important points here. Firstly, enabling prejudices may start as consciously learned patterns, but can gradually become internalized, and perhaps not just implicit and habitual but tacit and unconscious. (The key difference here is how easily the practitioner can explain and articulate the reasoning behind some design decision.)

Secondly, to extent that these learned patterns are regarded as "best practices", it may be necessary to bring them back into full consciousness (whatever that means) so they can be replaced by "next practices". 

---

Bryan Lawson, How Designers Think (1980, 4th edition 2005)

Peter Rowe, Design Thinking (MIT Press 1987)

Wilson, T.D. (2002) "The nonsense of 'knowledge management'" Information Research, 8(1), paper no. 144

Stanford Encyclopedia of Philosophy: Gadamer, Knowledge How, Merleau-Ponty

Wikipedia: Boisot's I-Space, Nonaka's SECI model, Tacit Knowledge

Thanks to my friend Professor James Morley for help with Merleau-Ponty and sedimentation.

Related posts: From Black Belt to White Belt (September 2009), Three Notions of Maturity (March 2013), From Sedimented Principles to Enabling Prejudices (March 2013)

The Power of Principles (Not)

Discussing The Enclosure of the Commons with @umairh.

@umairh Here's some more stuff we can pimp--oh, sorry, I mean "privatize"--while we're at it. The atmosphere, the oceans, our grandkids. Oh, wait...

In response, I pointed to @owenbarder 's blog Enclosure of the Commons – 21st Century Edition.

@umairh Exactly. That's why fighting back with "open-source"/commons principles is so disruptive--and important.

But the history of enclosure doesn't suggest that it can be defeated by "principles". Wikipedia: Enclosure. Fighting back may be important and disruptive, but surely disruption needs more than principles? After all, people have often defended enclosure with another set of equally plausible principles - protecting the environment, increasing agricultural productivity, or whatever.

---

Here's a more general question - to what extent have "principles" ever contributed significantly to social or political change. Many key historical changes - examples might include the abolition of slavery in the USA, the enfranchisement of women, and the independence of India - were heralded by strong and principled campaigns. But why were these campaigns more successful than those against enclosure?

We might note that in each case of successful progressive change, there is an alternative explanation for the event, based on socioeconomic and geopolitical forces. For example, with the availability of cheap quinine (reducing the economic dependence on labour of West African origin), slavery ceased to be the cheapest form of labour in malaria-ridden plantations. Such socioeconomic explanations should caution us against regarding the forceful articulation of principles as the sole driver of social change.

In business and engineering, as well as politics, it is customary to appeal to "principles" to justify some business model, some technical solution, or some policy. But these principles are usually so vague that they provide very little concrete guidance. Profitability, productivity, efficiency, which can mean almost anything you want them to mean. And when principles interfere with what we really want to do, we simply come up with a new interpretation of the principle, or another overriding principle, which allows us to do exactly what we want while dressing up the justification in terms of "principles".

The BBC Moral Maze programme this week discussed a recent case of a Christian couple in the UK who refused bed-and-breakfast to a gay couple, thereby offending against recent anti-discrimination legislation. This case appears to involve two conflicting applications of the same principle - tolerance and human rights. Listening to the programme, I thought how easy it might have been for the Christian couple to turn away guests they regarded as undesirable by appealing instead to the principle of security, and how often "security" and "risk" is used as a reason for being unpleasant or unhelpful to other people. I also remembered FakeSteveJobs' recent rant against Christian intolerance, in which he offered the following interpretation of the Good Samaritan story. "Jesus, your big hero, was saying that if you have some rule or conventional wisdom that causes you to do harm to people, violate the goddamn rule." [FSJ December 2010]

So much for principles then.

---

I have previously written about the over-emphasis on principles within the discourse of enterprise architecture: What's Wrong With Principles, What's Wrong With Principles 2.