On the invisibility of infrastructure

Infrastructure is boring, expensive, and usually someone else's responsibility/problem. Which is perhaps how the UK finds itself at what Jeremy Fleming, head of GCHQ, describes as a moment of reckoning. Simon Wardley analyses this in terms of digital sovereignty.

Digital sovereignty is all about us (as a collective) deciding which parts of this competitive space that we want to own, compete, defend, dominate and represent our values and our behaviours in. It's all about where are our borders in this space. ... Our responses all seem to include a slide into protectionism with claims that we need to build our own cloud industries.

Fleming is particularly focused on "the growing challenge from China", expresses concern about UK potentially losing control of  "standards that shape our technology environment" which apparently "make sure that our liberal Western democratic views are baked into our technology". Whatever that means. Fleming's technological examples include digital currency and smart cities.

Fleming talks about the threats from Russia and China, and regards China's potential control of the underlying infrastructure as more fundamentally challenging than potential attacks from Russia as well as non-state actors.

Fleming notes the following characteristics of those he labels adversaries:

• Potential to control the global operating system.
• Early implementors of many of the emerging technologies that are changing the digital environment.
• Bringing all elements of [...] power to control, influence, design and dominate markets. Often with the effect of pushing out smaller players and reducing innovation. 
• Concerted campaigns to dominate international standards.
  

And continues

If [any of this] turns out to be insecure or broken or undemocratic, everyone is going to be facing a very difficult future.

It would be easy to hear these remarks as referring solely to China. But he also sounds a warning about corporate power, acknowledging that their commercial interests sometimes (!?) don't align with the interests of ordinary citizens. And with that in mind, it's easy to see how some of the adversarial characteristics listed above would apply equally to some of the Western tech giants.

If the goal is to bake Western values (whatever they are) into our technology infrastructure, it is not obvious that the Western tech giants can be trusted to do this. Smart City initiatives associated with Google's Sidewalk Labs have been cancelled in Portland and Toronto, following (although perhaps not entirely as a consequence of) democratic concerns about surveillance capitalism. However, Sidewalk Labs appears to be still active in a number of smaller smart city initiatives, as are Amazon Web Services, IBM and other major technology firms.

Fleming talks about standards, but at the same time he acknowledges that standards alone are too slow-changing and too weak to keep the adversaries at bay. "The nature of cyberspace makes the rules and standards more open to abuse." He talks about evolutionary change, using a version of Leon Megginson's formulation of natural selection: "it's those that are most able to adjust that prosper". (See my post on Arguments from Nature). But that very formulation seems to throw the initiative over to those tech firms that preach moving fast and breaking things. Can we therefore complain if our infrastructure is insecure, broken, and above all undemocratic?

For most of us, most of the time, infrastructure needs to be just there, taken for granted, ready to hand. Organizations providing these services are often established as monopolies, or turn into de facto monopolies, controlled not only (if at all) by market forces but by democratically accountable regulators and/or by technocratic specialists. However, the Western tech giants devote significant resources to lobbying against external regulation, resisting democratic control. And Smart City initiatives typically embed much the same values everywhere (civic paternalism, biopower).

So here is Fleming's dilemma. If you don't want China to make the running on smart cities, you have to forge alliances with other imperfectly trusted players, whose values are sometimes (!?) not aligned with yours. This moves away from the kind of positional strategy described in Wardley's maps, towards a more relational strategy.

 

---

Gordon Corera, GCHQ chief warns of tech 'moment of reckoning' (BBC News, 23 April 2021) via @sukhigill and @swardley

Jeremy Fleming, A world of possibilities: Leading the way in cyber and technology (Vincent Briscoe Lecture @ Imperial College, 23 April 2021) via YouTube.

Susan Leigh Star and Karen Ruhleder, Steps Toward an Ecology of Infrastructure: Design and Access for Large Information Spaces (Information Systems Research 7/1, March 1996)

Simon Wardley, Digital Sovereignty (22 October 2020)

Related posts: The Allure of the Smart City (April 2021)

The Social Dilemma

Just watched the documentary The Social Dilemma on Netflix, which takes a critical look at some of the tech giants that dominate our world today (although not Netflix itself, for some reason), largely from the perspective of some former employees who helped them achieve this dominance and are now having second thoughts. One of the most prominent members of this group is Tristan Harris, formerly with Google, now the president of an organization called the Center for Humane Technology. He and others have been airing these concerns for several years already - see for example Noah Kulwin's 2018 article (link below).

The documentary opens by asking the contributors to state the problem, and shows them all initially hesitating. By the end of the documentary, however, they are mostly making large statements about the morality of encouraging addictive behaviour, the propagation of truth and lies, the threat to democracy, the ease with which these platforms can be used by authoritarian rulers and other bad actors, and the need for regulation.

Quantity becomes quality. To some extent, the phenomena and affordances of social media can be regarded as merely scaled-up versions of previous social tools, including advertising and television: the maxim If you aren't paying, you are the product derives from a 1973 video about the power of commercial television. However, several of the contributors to the documentary observed that the power of the modern platforms and the wealth of the businesses that control these platforms is unprecedented, while noting that social media is far less regulated than other mass communication enterprises, including television and telecommunications.

Contributors doubted whether we could expect these enterprises, or the technology sector generally, to fix these problems on their own - especially given the focus on profit, growth and shareholder value that drives all enterprises within the capitalist system. Is it fair to ask them to reform capitalism? (Many years ago, the architect J.P. Eberhard noted a tendency to escalate even small problems to the point where the entire capitalist system comes into question, and argued that We Ought To Know The Difference.) So is regulation the answer?

Surprisingly enough, Facebook doesn't think so. In its response to the documentary, it complains

The film’s creators do not include insights from those currently working at the companies or any experts that take a different view to the narrative put forward by the film.

As Pranav Malhotra notes, it's not hard to find experts who would offer a different perspective, in many cases offering far more fundamental and far-reaching criticisms of Facebook and its peers. Hey Facebook, careful what you wish for!

Last year, Tristan Harris appeared to call for a new interdisciplinary field of research, focused on exploring the interaction between technology and society. Several people including @ruchowdh pointed out that such a field was already well-established. (In response he said he already knew this, and apologized for his poor choice of words, blaming the Twitter character limit.)

So there is already an abundance of deep and interesting work that can help challenge the simplistic thinking of Silicon Valley in a number of areas including

• Truth and Objectivity
• Technological Determinism
• Custodianship of Technology (for example Latour's idea that we should Love Our Monsters - see also article by Adam Briggle)
  

These probably deserve a separate post each, if I can find time to write them. 

---

The Social Dilemma (dir Jeff Orlowski, Netflix 2020)

Wikipedia: The Social Dilemma, Television Delivers People

Stanford Encyclopedia of Philosophy: Ethics of Artificial Intelligence and Robotics, Phenomenological Approaches to Ethics and Information Technology, Philosophy of Technology

 

Adam Briggle, What can be done about our modern-day Frankensteins? (The Conversation, 26 December 2017)

Robert L. Carneiro, The transition from quantity to quality: A neglected causal mechanism in accounting for social evolution  (PNAS 97:23, 7 November 2000)

Rumman Chowdhury, To Really 'Disrupt,' Tech Needs to Listen to Actual Researchers (Wired, 26 June 2019)

Facebook, What the Social Dilemma Gets Wrong (2020)

Tristan Harris, “How Technology Is Hijacking Your Mind—from a Magician and Google Design Ethicist”, Thrive Global, 18 May 2016

Noah Kulwin, The Internet Apologizes (New York Magazine, 16 April 2018)

John Lanchester, You Are The Product (London Review of Books, Vol. 39 No. 16, 17 August 2017)

Bruno Latour, Love Your Monsters: Why we must care for our technologies as we do our children (Breakthrough, 14 February 2012) 

Pranav Malhotra, The Social Dilemma Fails to Tackle the Real Issues in Tech (Slate, 18 September 2020)

Richard Serra and Carlota Fay Schoolman, Television Delivers People (1973) 

Zadie Smith, Generation Why? (New York Review of Books, 25 November 2010)

Siva Vaidhyanathan, Making Sense of the Facebook Menace (The New Republic, 11 January 2021)

Related posts: The Perils of Facebook (February 2009), We Ought to Know the Difference (April 2013), Rhyme or Reason: The Logic of Netflix (June 2017), On the Nature of Platforms (July 2017), Ethical Communication in a Digital Age (November 2018), Shoshana Zuboff on Surveillance Capitalism (February 2019)

Whom does the change serve?

In my writings on technology ethics, riffing on the fact that so many cool technologies are presented as the Holy Grail of something or other, I have frequently invoked the mediaeval question that Parsifal failed to ask: Whom does the Grail Serve?

• Chatbot ethics - Whom does the chatbot serve? (May 2019)
  
• Driverless cars - Whom does the technology serve? (May 2019)
• The Road Less Travelled - Whom Does the Algorithm Serve? (June 2019)
  

The same question can be asked of other changes and transformations, where technology might be part of the story but is not the primary story.

• Is Organizational Integration a Good Thing? (November 2012)
• The Ethics of Disruption (August 2019)
• What difference does technology make? (October 2019)
  
• Bold Restless Experimentation (June 2020)
  

 

In response to Francis Fukuyama's statement on Big Tech's information monopoly

Almost every abuse these platforms are accused of perpetrating can be simultaneously defended as economically efficient

@mireillemoret argues

Efficiency is important, but it is NOT the holy grail

 

Important for whom? When I get involved in economic discussions of efficiency or productivity or whatever, I always try to remember the ethical dimension - efficiency for whom, productivity for whom, predictability and risk reduction for whom, innovation for whom.

Note: I just started reading Adrian Daub's new book, but I haven't got to the Disruption chapter yet.

---

Chris Bruce, Environmental Decision-Making as Central Planning: FOR WHOM is Production to Occur? (Environmental Economics Blog, 19 August 2005)

Adrian Daub, What tech calls thinking (Farrar Straus and Giroux, 2020) 

Adrian Daub, The disruption con: why big tech’s favourite buzzword is nonsense (The Guardian, 24 September 2020)

Francis Fukuyama, Barak Richman, and Ashish Goel, How to Save Democracy From Technology - Ending Big Tech’s Information Monopoly (Foreign Affairs, January/February 2021) 

Further posts

• For Whom (November 2006)
  
• From SOA to better judgement (January 2009)
  
• Redesigning the Banana (July 2009)
• Arguing with Drucker (April 2015) 
• Evolution or Revolution (December 2020)

Mapping out the entire world of objects

ImageNet is a large crowd-sourced database of coded images, widely used for machine learning. This database can be traced to an idea articulated by Fei-Fei Li in 2006: We’re going to map out the entire world of objects. In a blogpost on the Limitations of Machine Learning, I described this idea as naive optimism.

Such datasets raise both ethical and epistemological issues. One of the ethical problems thrown up by these image databases is that objects are sometimes also subjects. Bodies and body parts are depicted (often without consent) and labelled (sometimes offensively); people are objectified; and the objectification embedded in these datasets are then passed on to the algorithms that use them and learn from them. Crawford and Paglen argue convincingly that categorizing and classifying people is not just a technical process but a political act. And thanks to some great detective work by Vinay Prabhu and Abeba Birhane, MIT has withdrawn Tiny Images, another large image dataset widely used for machine learning.

But in this post, I'm going to focus on the epistemological and metaphysical issues - what constitutes the world, and how can we know about it. Li is quoted as saying Data will redefine how we think about models. The reverse should also be true, as I explain in my blogpost on the Co-Production of Data and Knowledge.

What exactly is meant by the phrase the entire world of objects and what would mapping this world really entail? Although I don't believe that philosophy is either necessary or sufficient to correct all of the patterns of sloppy thinking by computer scientists, even a casual reading of Wittgenstein, Quine and other 20th century philosophers might prompt people to question some simplistic assumptions of the relationships between Word and Object underpinning these projects. According to Donna Haraway, what counts as an object is precisely what world history turns out to be about.

The first problem with these image datasets is the assumption that images can be labelled according to the objects that are depicted in them. But as Prabhu and Birhane note, real-world images often contain multiple objects. Crawford and Paglen argue that images are laden with potential meanings, irresolvable questions, and contradictions and that ImageNet’s labels often compress and simplify images into deadpan banalities.

One photograph shows a dark-skinned toddler wearing tattered and dirty clothes and clutching a soot-stained doll. The child’s mouth is open. The image is completely devoid of context. Who is this child? Where are they? The photograph is simply labeled toy. Crawford and Paglen

Implicit in the labelling of this photograph is some kind of ontological precedence - that the doll is more significant than the child. As for the emotional and physical state of the child, ImageNet doesn't seem to regard these states as objects at all. (There are other image databases that do attempt to code emotions - see my post on Affective Computing.)

Given that much of the Internet is funded by companies that want to sell us things, it would not be surprising if there is an ontological bias towards things that can be sold. (This is what the word everything means in the Everything Store.) So that might explain why ImageNet chooses to focus on the doll rather than the child. But similar images are also used to sell washing powder. Thus the commercially relevant label might equally have been dirt.

But not only do concepts themselves (such as toys and dirt) vary between different discourses and cultures (as explored by anthropologists such as Mary Douglas), the ontological precedence between concepts may vary. People from a different culture, or with a different mindset, will jump to different conclusions as to what is the main thing depicted in a given image.

The American philosopher W.V.O. Quine argued that translation was indeterminate. If a rabbit runs past, and a speaker of an unknown language, Arunta, utters the word gavegai, we might guess that this word in Arunta corresponds to the word rabbit in English. But there are countless other things that the Arunta speaker might have been referring to. And although over time we may be able to eliminate some of these possibilities, we can never be sure we have correctly interpreted the meaning of the word gavegai. Quine called this the inscrutability of reference. Similar indeterminacy would seem to apply to our collection of images.

The second problem has to do with the nature of classification. I have talked about this in previous posts - for example on Algorithms and Governmentality - so I won't repeat all that here.

Instead, I want to jump to the third and final problem, arising from the phrase the entire world of objects - what does this really mean? How many objects are there in the entire world, and is it even a finite number? We can't count objects unless we can agree what counts as an object. What are the implications of what is included in everything and what is not included?

I occasionally run professional workshops in data modelling. One of the exercises I use is to display a photograph and ask the students to model all the objects they can see in the picture. Students who are new to modelling can always produce a simple model, while more advanced students can produce much more sophisticated models. There doesn't seem to be any limit to how many objects people can see in my picture.

ImageNet boasts 14 million images, but that doesn't seem a particularly large number from a big data perspective. For example, I guess there must be around a billion dogs in the world - so how many words and images do you need to represent a billion dogs?

Bruhl found some languages full of detail
Words that half mimic action; but
generalization is beyond them, a white dog is
not, let us say, a dog like a black dog.
Pound, Cantos XXVIII

---

Kate Crawford and Trevor Paglen, Excavating AI: The Politics of Images in Machine Learning Training Sets (19 September 2019)

Mary Douglas, Purity and Danger (1966)

Dave Gershgorn, The data that transformed AI research—and possibly the world (Quartz, 26 July 2017)

Donna Haraway, Situated Knowledges (Feminist Studies 14/3, 1988) pp 575-99

Vinay Uday Prabhu and Abeba Birhane, Large Image Datasets: A pyrrhic win for computervision? (Preprint, 1 July 2020)

Katyanna Quach, MIT apologizes, permanently pulls offline huge dataset that taught AI systems to use racist, misogynistic slurs Top uni takes action after El Reg highlights concerns by academics (The Register, 1 July 2020) 

W.V.O. Quine, Word and Object (MIT Press, 1960)

Stanford Encyclopedia of Philosophy: Feminist Perspectives on Objectification, Quine on the Indeterminacy of Translation


Related posts:  Co-Production of Data and Knowledge (November 2012), Have you got big data in your underwear (December 2014), Affective Computing (March 2019), Algorithms and Governmentality (July 2019), Limitations of Machine Learning (July 2020)

What Difference Does Technology Make?

In his book on policy-making, Geoffrey Vickers talks about three related types of judgment – reality judgment (what is going on, also called appreciation or sense-making), value judgment and action judgment.

In his book on technology ethics, Hans Jonas notes "the excess of our power to act over our power to foresee and our power to evaluate and to judge" (p22). In other words, technology disrupts the balance between the three types of judgment identified by Vickers.

Jonas (p23) identifies some critical differences between technological action and earlier forms

• novelty of its methods
• unprecedented nature of some of its objects
• sheer magnitude of most of its enterprises
• indefinitely cumulative propagation of its effects

In short, this amounts to action at a distance - the effects of one's actions and decisions reach further and deeper, affecting remote areas more quickly, and lasting long into the future. Which means that accepting responsibility only for the immediate and local effects of one's actions can no longer be justified.

Jonas also notes that the speed of technologically fed developments does not leave itself the time for self-correction (p32). An essential ethical difference between natural selection, selective breeding and genetic engineering is not just that they involve different mechanisms, but that they operate on different timescales.

(Of course humans have often foolishly disrupted natural ecosystems without recourse to technologies more sophisticated than boats. For example, the introduction of rabbits into Australia or starlings into North America. But technology creates many new opportunities for large-scale disruption.)

Another disruptive effect of technology is that it affects our reality judgments. Our knowledge and understanding of what is going on (WIGO) is rarely direct, but is mediated (screened) by technology and systems. We get an increasing amount of our information about our social world through technical media: information systems and dashboards, email, telephone, television, internet, social media, and these systems in turn rely on data collected by a wide range of monitoring instruments, including IoT. These technologies screen information for us, screen information from us.

The screen here is both literal and metaphorical. It is a surface on which the data are presented, and also a filter that controls what the user sees. The screen is a two-sided device: it both reveals information and hides information.

Heidegger thought that technology tends to constrain or impoverish the human experience of reality in specific ways. Albert Borgmann argued that technological progress tends to increase the availability of a commodity or service, and at the same time pushes the actual device or mechanism into the background. Thus technology is either seen as a cluster of devices, or it isn't seen at all. Borgmann calls this the Device Paradigm.

But there is a paradox here. On the one hand, the device encourages to pay attention to the immediate affordance of the device, and ignore the systems that support the device. So we happily consume recommendations from media and technology giants, without looking too closely at the surveillance systems and vast quantities of personal data that feed into these recommendations. But on the other hand, technology (big data, IoT, wearables) gives us the power to pay attention to vast areas of life that were previously hidden.

In agriculture for example, technology allows the farmer to have an incredibly detailed map of each field, showing how the yield varies from one square metre to the next. Or to monitor every animal electronically for physical and mental welbeing.

And not only farm animals, also ourselves. As I said in my post on the Internet of Underthings, we are now encouraged to account for everything we do: footsteps, heartbeats, posture. (Until recently this kind of micro-attention to oneself was regarded as slightly obsessional, nowadays it seems to be perfectly normal.)

Technology also allows much more fine-grained action. A farmer no longer has to give the same feed to all the cows every day, but can adjust the composition of the feed for each individual cow, to maximize her general well-being as well as her milk production.

In the 1980s when Borgmann and Jonas were writing, there was a growing gap between the power to act and the power to foresee. We now have technologies that may go some way towards closing this gap. Although these technologies are far from perfect, as well as introducing other ethical issues, they should at least make it easier for the effects of new technologies to be predicted, monitored and controlled, and for feedback and learning loops to be faster and more effective. And responsible innovation should take advantage of this.

---

Albert Borgmann, Technology and the Character of Everyday Life (University of Chicago Press, 1984)

Hans Jonas, The Imperative of Responsibility (University of Chicago Press, 1984)

Geoffrey Vickers, The Art of Judgment: A Study in Policy-Making (Sage 1965)


Wikipedia: Rabbits in Australia, Starlings in North America

On the Scope of Ethics

I was involved in a debate this week, concerning whether ethical principles and standards should include weapons systems, or whether military purposes should be explicitly excluded.

On both sides of the debate, there were people who strongly disapproved of weapons systems, but this disapproval led them to two opposite positions. One side felt that applying any ethical principles and standards to such systems would imply a level of ethical approval or endorsement, which they would prefer to withhold. The other side felt that weapons systems called for at least as much ethical scrutiny as anything else, if not more, and thought that exempting weapons systems implied a free pass.

It goes without saying that people disapprove of weapons systems to different degrees. Some people think they are unacceptable in all circumstances, while others see them as a regrettable necessity, while welcoming the economic activity and technological spin-offs that they produce. It's also worth noting that there are other sectors that attract strong disapproval from many people, including gambling, hydrocarbon, nuclear energy and tobacco, especially where these appear to rely on disinformation campaigns such as climate science denial.

It's also worth noting that there isn't always a clear dividing line between those products and technologies that can be used for military purposes and those that cannot. For example, although the dividing line between peaceful nuclear power and nuclear weapons may be framed as a purely technical question, this has major implications for international relations, and technical experts may be subject to significant political pressure.

While there may be disagreements about the acceptability of a given technology, and legitimate suspicion about potential use, these should be capable of being addressed as part of ethical governance. So I don't think this is a good reason for limiting the scope.

However, a better reason for limiting the scope may be to simplify the task. Given finite time and resources, it may be better to establish effective governance for a limited scope, than taking forever getting something that works properly for everything. This leads to the position that although some ethical governance may apply to weapons systems, this doesn't mean that every ethical governance exercise must address such systems. And therefore it may be reasonable to exclude such systems from a specific exercise for a specific time period, provided that this doesn't rule out the possibility of extending the scope at a later date.

---

Update. The US Department of Defense has published a high-level set of ethical principles for the military use of AI. Following the difference of opinion outlined above, some people will think it matters how these principles are interpreted and applied in specific cases (since like many similar sets of principles, they are highly generic), while other people will think any such discussion completely misses the point.

David Vergun, Defense Innovation Board Recommends AI Ethical Guidelines (US Dept of Defense, 1 November 2019)

Ethics of Transparency and Concealment

Last week I was in Berlin at the invitation of the IEEE to help develop standards for responsible technology (P7000). One of the working groups (P7001) is looking at transparency, especially in relation to autonomous and semi-autonomous systems. In this blogpost, I want to discuss some more general ideas about transparency.

In 1986 I wrote an article for Human Systems Management promoting the importance of visibility. There were two reasons I preferred this word. Firstly, "transparency" is a contronym - it has two opposite senses. When something is transparent, this either means you don't see it, you just see through it, or it means you can really see it. And secondly, transparency appears to be merely a property of an object, whereas visibility is about the relationship between the object and the viewer - visibility to whom?

(P7001 addresses this by defining transparency requirements in relation to different stakeholder groups.)

Although I wasn't aware of this when I wrote the original article, my concept of visibility shares something with Heidegger's concept of Unconcealment (Unverborgenheit). Heidegger's work seems a good starting point for thinking about the ethics of transparency.

Technology generally makes certain things available while concealing other things. (This is related to what Albert Borgmann, a student of Heidegger, calls the Device Paradigm.)

In our time, things are not even regarded as objects, because their only important quality has become their readiness for use. Today all things are being swept together into a vast network in which their only meaning lies in their being available to serve some end that will itself also be directed towards getting everything under control. Levitt

Goods that are available to us enrich our lives and, if they are technologically available, they do so without imposing burdens on us. Something is available in this sense if it has been rendered instantaneous, ubiquitous, safe, and easy. Borgmann

I referred above to the two opposite meanings of the word "transparent". For Heidegger and his followers, the word "transparent" often refers to tools that can be used without conscious thought, or what Heidegger called ready-to-hand (zuhanden). In technology ethics, on the other hand, the word "transparent" generally refers to something (product, process or organization) being open to scrutiny, and I shall stick to this meaning for the remainder of this blogpost.

We are surrounded by technology, we rarely have much idea how most of it works, and usually cannot be bothered to find out. Thus when technological devices are designed to conceal their inner workings, this is often exactly what the users want. How then can we object to concealment?

The ethical problems of concealment depend on what is concealed by whom and from whom, why it is concealed, and whether, when and how it can be unconcealed.

Let's start with the why. Sometimes people deliberately hide things from us, for dishonest or devious reasons. This category includes so-called defeat devices that are intended to cheat regulations. Less clear-cut is when people hide things to avoid the trouble of explaining or justifying them.

(If something is not visible, then we may not be aware that there is something that needs to be explained. So even if we want to maintain a distinction between transparency and explainability, the two concepts are interdependent.)

People may also hide things for aesthetic reasons. The Italian civil engineer Riccardo Morandi designed bridges with the steel cables concealed, which made them difficult to inspect and maintain. The Morandi Bridge in Genoa collapsed in August 2018, killing 43 people.

And sometimes things are just hidden, not as a deliberate act but because nobody has thought it necessary to make them visible. (This is one of the reasons why a standard could be useful.)

We also need to consider the who. For whose benefit are things being hidden? In particular, who is pulling the strings, where is the funding coming from, and where are the profits going - follow the money. In technology ethics, the key question is Whom Does The Technology Serve?

In many contexts, therefore, the main focus of unconcealment is not understanding exactly how something works but being aware of the things that people might be trying to hide from you, for whatever reason. This might include being selective about the available evidence, or presenting the most common or convenient examples and ignoring the outliers. It might also include failing to declare potential conflicts of interest.

For example, the #AllTrials campaign for clinical trial transparency demands that drug companies declare all clinical trials in advance, rather than waiting until the trials are complete and then deciding which ones to publish.

Now let's look at the possibility of unconcealment. Concealment doesn't always mean making inconvenient facts impossible to discover, but may mean making them so obscure and inaccessible that most people don't bother, or creating distractions that divert people's attention elsewhere. So transparency doesn't just entail possibility, it requires a reasonable level of accessibility.

Sometimes too much information can also serve to conceal the truth. Onora O'Neill talks about the "cult of transparency" that fails to produce real trust.

Transparency can produce a flood of unsorted information and misinformation that provides little but confusion unless it can be sorted and assessed. It may add to uncertainty rather than to trust. Transparency can even encourage people to be less honest, so increasing deception and reducing reasons for trust. O'Neill

Sometimes this can be inadvertent. However, as Chesterton pointed out in one of his stories, this can be a useful tactic for those who have something to hide.

Where would a wise man hide a leaf? In the forest. If there were no forest, he would make a forest. And if he wished to hide a dead leaf, he would make a dead forest. And if a man had to hide a dead body, he would make a field of dead bodies to hide it in. Chesterton

Stohl et al call this strategic opacity (via Ananny and Crawford).

Another philosopher who talks about the "cult of transparency" is Shannon Vallor. However, what she calls the "Technological Transparency Paradox" seems to be merely a form of asymmetry: we are open and transparent to the social media giants, but they are not open and transparent to us.

In the absence of transparency, we are forced to trust people and organizations - not only for their honesty but also their competence and diligence. Under certain conditions, we may trust independent regulators, certification agencies and other institutions to verify these attributes on our behalf, but this in turn depends on our confidence in their ability to detect malfeasance and enforce compliance, as well as believing them to be truly independent. (So how transparent are these institutions themselves?) And trusting products and services typically means trusting the organizations and supply chains that produce them, in addition to any inspection, certification and official monitoring that these products and services have undergone.

Instead of seeing transparency as a simple binary (either something is visible or it isn't), it makes sense to discuss degrees of transparency, depending on stakeholder and context. For example, regulators, certification bodies and accident investigators may need higher levels of transparency than regular users. And regular users may be allowed to choose whether to make things visible or invisible. (Thomas Wendt discusses how Heideggerian thinking affects UX design.)

Finally, it's worth noting that people don't only conceal things from others, they also conceal things from themselves, which leads us to the notion of self-transparency. In the personal world this can be seen as a form of authenticity; in the corporate world, it translates into ideas of responsibility, due diligence, and a constant effort to overcome wilful blindness.

If transparency and openness is promoted as a virtue, then people and organizations can make their virtue apparent by being transparent and open, and this may make us more inclined to trust them. We should perhaps be wary of organizations that demand or assume that we trust them, without providing good evidence of their trustworthiness. (The original confidence trickster asked strangers to trust him with their valuables.) The relationship between trust and trustworthiness is complicated. 

---

UK Department of Health and Social Care, Response to the House of Commons Science and Technology Committee report on research integrity: clinical trials transparency (UK Government Policy Paper, 22 February 2019) via AllTrials

Mike Ananny and Kate Crawford, Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability (new media and society 2016) pp 1–17

Albert Borgmann, Technology and the Character of Contemporary Life (University of Chicago Press, 1984)

G.K. Chesterton, The Sign of the Broken Sword (The Saturday Evening Post, 7 January 1911)

Martin Heidegger, The Question Concerning Technology (Harper 1977) translated and with an introduction by William Lovitt

Onora O'Neill, Trust is the first casualty of the cult of transparency (Telegraph, 24 April 2002)

Cynthia Stohl, Michael Stohl and P.M. Leonardi, Managing opacity: Information visibility and the paradox of transparency in the digital age (International Journal of Communication Systems 10, January 2016) pp 123–137.

Richard Veryard, The Role of Visibility in Systems (Human Systems Management 6, 1986) pp 167-175 (this version includes some further notes dated 1999)

Thomas Wendt, Designing for Transparency and the Myth of the Modern Interface (UX Magazine, 26 August 2013)

Stanford Encyclopedia of Philosophy: Heidegger, Technological Transparency Paradox

Wikipedia: Confidence Trick, Follow The Money, Ponte Morandi, Regulatory Capture,Willful Blindness


Related posts: Defeating the Device Paradigm (October 2015), Transparency of Algorithms (October 2016), Pax Technica (November 2017), Responsible Transparency (April 2019), Whom Does The Technology Serve (May 2019)

What Does Diversion Mean?

Diversion has various different meanings in the world of ethics.

Distraction. An idea or activity serves as a distraction from what's important. For example, @juliapowles uses the term "captivating diversion" to refer to ethicists becoming preoccupied with narrow computational puzzles that distract them from far more important issues. See my post on The Game of Wits Between Technologists and Ethics Professors (June 2019)

Substitution. People are redirected from something harmful to something supposedly less harmful. For example, switching from smoking to vaping. See my post on the Ethics of Diversion - Tobacco Example (September 2019). And in the 1840s, a Baptist preacher and temperance activist organized excursions to divert people from drinking. His name: Thomas Cook.

Unauthorized Utilization. Using products for some purpose other than that approved or prescribed for a given purpose in a given market. There are various forms of this, some of which are both illegal and unethical, while others may be ethically justifiable.

• Drug diversion, the transfer of any legally prescribed controlled substance from the individual for whom it was prescribed to another person for any illicit use.

• Grey imports. Drug companies try to control shipments of drugs between markets, especially when this is done to undercut the official drug prices. However, some people regard the tactics of the drug companies as unethical. Médecins Sans Frontières, the medical charity, has accused one pharma giant of promoting overly-intrusive patient surveillance to stop a generic drug being diverted to patients in developed countries.

• Off-label use. Doctors may prescribe drugs for a purpose or patient group outside the official approval, with various degrees of justification. For more discussion, see my post Off-Label (March 2005)

Exploiting Regulatory Divergence. Carrying out activities (for example, conducting trials) in countries with underdeveloped ethics and weak regulatory oversight. See debate between Wertheimer and Resnick.

---

Amy Kazmin, Pharma combats diversion of cheap drugs (FT 12 April 2015)

Julia Powles, The Seductive Diversion of ‘Solving’ Bias in Artificial Intelligence (7 December 2018)

David B. Resnik, Addressing diversion effects (Journal of Law and the Biosciences, 2015) 428–430

Alan Wertheimer, The ethics of promulgating principles of research ethics: the problem of diversion effects (J Law Biosci. 2(1) Feb 2015) 2-32

Wikipedia: Drug Diversion, Thomas Cook

The Ethics of Diversion - Tobacco Example

What are the ethics of diverting people from smoking to vaping?

On the one hand, we have the following argument.

• E-cigarettes ("vaping") offer a plausible substitute for smoking cigarettes.
• Smoking is dangerous, and vaping is probably much less dangerous.
• Many smokers find it difficult to give up, even if they are motivated to do so. So vaping provides a plausible exit route.
• Observed reductions in the level of smoking can be partially attributed to the availability of alternatives such as vaping. (This is known as the diversion hypothesis.)
• It is therefore justifiable to encourage smokers to switch from cigarettes to e-cigarettes.

Critics of this argument make the following points.

• While the dangers of smoking are now well-known, some evidence is now emerging to suggest that vaping may also be dangerous. In the USA, a handful of people have died and hundreds have been hospitalized.
• While some smokers may be diverted to vaping, there are also concerns that vaping may provide an entry path to smoking, especially for young people. This is known as the gateway or catalyst hypothesis.

Some defenders of vaping blame the potential health risks and the gateway effect not on vaping itself but on the wide range of flavours that are available. While these may increase the attraction of vaping to children, the flavour ingredients are chemically unstable and may produce toxic compounds. For this reason, President Trump has recently proposed a ban on flavoured e-cigarettes.

Juul, which dominates the e-cigarette market in the US, is currently being investigated by the FDA and federal prosecutors for its marketing, and the inappropriately named Mr Burns has just stepped down as CEO.

And elsewhere in the world, significant differences in regulation are emerging between countries. While some countries are looking to ban e-cigarettes altogether, the UK position (as presented by Public Health England and the MHRA) is to encourage e-cigarettes as a safe alternative to smoking. At some point in the future presumably, UK data can be compared with data from other countries to provide evidence for or against the UK position. Professor Simon Capewell of Liverpool University (quoted in the Observer) calls this a "bizarre national experiment".

While we await convincing data about outcomes, ethical reasoning may appeal to several different principles.

Firstly, the minimum interference principle. In this case, this means not restricting people's informed choice without good reason.

Secondly, the utilitarian principle. The benefit of helping a large number of people to reduce a known harm outweighs the possibility of causing a lesser but unknown harm to a smaller number of people.

Thirdly, the cautionary principle. Even if vaping appears to be safer than traditional smoking, Professor Capewell reminds us of other things that were assumed to be safe - until we discovered that they weren't safe at all.

And finally, the conflict of interest principle. Elliott Reichardt, a researcher at the University of Calvary and a campaigner against vaping, argues that any study, report or campaign funded by the tobacco industry should be regarded with some suspicion.

I wouldn't be rushing to defend the vaping industry right now. Caution may well prove to be unfounded, but highly profitable products have a habit of turning out to be unsavoury as events unfold. This would not be a good one to be on the wrong side of history for, either way. https://t.co/vNeYX9u8Uz

— Jon Ayre (@Jon_Ayre) September 16, 2019

Meanwhile, the traditional tobacco industry is hedging its bets - investing in e-cigarettes but doing well when vaping falters.

---

US Food and Drug Administration, Warning Letter to Juul Labs (FDA, 9 September 2019) via BBC News

Allan M. Brandt, Inventing Conflicts of Interest: A History of Tobacco Industry Tactics (Am J Public Health 102(1) January 2012) 63–71

Tom Chivers, Stop Hating on Vaping (Unherd, 13 September 2019) via @IanDunt

Jamie Doward, After six deaths in the US and bans around the world – is vaping safe? (Observer, 15 September 2019)

David Heath, Contesting the Science of Smoking (Atlantic, 4 May 2016)

Angelica Lavito, Juul built an e-cigarette empire. Its popularity with teens threatens its future (CNBC 4 August 2018)

Levy DT, Warner KE, Cummings KM, et al, Examining the relationship of vaping to smoking initiation among US youth and young adults: a reality check (Tobacco Control 20 November 2018)

Jennifer Maloney, Federal Prosecutors Conducting Criminal Probe of Juul (Wall Street Journal, 23 September 2019)

Elliott Reichardt and Juliet Guichon, Vaping is an urgent threat to public health (The Conversation, 13 March 2019)

The Ethics of Disruption

In a recent commentary on #Brexit, Simon Jenkins notes that

"disruption theory is much in vogue in management schools, so long as someone else suffers".

Here is Bruno Latour making the same point.

"Don't be fooled for a second by those who preach the call of wide-open spaces, of  'risk-taking', those who abandon all protection and continue to point at the infinite horizon of modernization for all. Those good apostles take risks only if their own comfort is guaranteed. Instead of listening to what they are saying about what lies ahead, look instead at what lies behind them: you'll see the gleam of the carefully folded golden parachutes, of everything that ensures them against the random hazards of existence." (Down to Earth, p 11)

Anyone who advocates "moving fast and breaking things" is taking an ethical position: namely that anything fragile enough to break deserves to be broken. This position is similar to the economic view that companies and industries that can't compete should be allowed to fail.

This position may be based on a combination of specific perceptions and general observations. The specific perception is when something is weak or fragile, protecting and preserving it consumes effort and resources that could otherwise be devoted to other more worthwhile purposes, and makes other things less efficient and effective. The general observation is that when something is failing, efforts to protect and preserve it may merely delay the inevitable collapse.

These perceptions and observations rely on a particular worldview or lens, in which things can be perceived as successful or otherwise, independent of other things. As Gregory Bateson once remarked (via Tim Parks),

"There are times when I catch myself believing there is something which is separate from something else."

Perceptions of success and failure are also dependent on timescale and time horizon. The dinosaurs ruled the Earth for 140 million years.

There may also be strong opinions about which things get protection and which don't. For example, some people may think it is more important to support agriculture or to rescue failing banks than to protect manufacturers. On the other hand, there will always be people who disagree with the choices made by governments on such matters, and who will conclude that the whole project of protecting some industry sectors (and not others) is morally compromised.

Furthermore, the idea that some things are "too big to fail" may also be problematic, because it implies that small things don't matter so much.

A common agenda of the disruptors is to tear down perceived barriers, such as regulations. This is subject to a fallacy known as Chesterton's Fence, assuming that anyone whose purpose is not immediately obvious must be redundant.

---

Simon Jenkins, Boris Johnson and Jeremy Hunt will have to ditch no deal – or face an election (Guardian, 28 June 2019)

Bruno Latour, Down to Earth: Politics in the New Climatic Regime (Polity Press, 2018)

Tim Parks, Impossible Choices (Aeon, 15 July 2019)

Rory Sutherland, Chesterton’s fence – and the idiots who rip it out (Spectator, 10 September 2016)


Related posts: Shifting Paradigms and Disruptive Technology (September 2008), Arguments from Nature (December 2010), Low-Hanging Fruit (August 2019)

Automation Ethics

Many people start their journey into the ethics of automation and robotics by looking at Asimov's Laws of Robotics.

A robot may not injure a human being or, through inaction, allow a human being to come to harm (etc. etc.)

As I've said before, I believe Asimov's Laws are problematic as a basis for ethical principles. Given that Asimov's stories demonstrate numerous ways in which the Laws don't actually work as intended. I have always regarded Asimov's work as being satirical rather than prescriptive.

While we usually don't want robots to harm people (although some people may argue for this principle to be partially suspended in the event of a "just war"), notions of harm are not straightforward. For example, a robot surgeon would have to cut the patient (minor harm) in order to perform an essential operation (major benefit). How essential or beneficial does the operation need to be, in order to justify it? Is the patient's consent sufficient?

Harm can be individual or collective. One potential harm from automation is that even if it creates wealth overall, it may shift wealth and employment opportunities away from some people, at least in the short term. But perhaps this can be justified in terms of the broader social benefit, or in terms of technological inevitability.

And besides the avoidance of (unnecessary) harm, there are some other principles to think about.

• Human-centred work - Humans should be supported by robots, not the other way around. 

• Whole system solutions - Design the whole system or process, don’t just optimize a robot as a single component.  

• Self-correcting - Ensure that the system is capable of detecting and learning from errors. 

• Open - Providing space for learning and future disruption. Don't just pave the cow-paths.

• Transparent - The internal state and decision-making processes of a robot are accessible to (some) users.  

Let's look at each of these in more detail.


Human-Centred Work

Humans should be supported by robots, not the other way around. So we don't just leave humans to handle the bits and pieces that can't be automated, but try to design coherent and meaningful jobs for humans, with robots to make them more powerful, efficient, and effective.

Organization theorists have identified a number of job characteristics associated with job satisfaction, including skill variety, task identity, task significance, autonomy and feedback. So we should be able to consider how a given automation project affects these characteristics.


Whole Systems

When we take an architectural approach to planning and designing new technology, we can look at the whole system rather than merely trying to optimize a single robotic component.

• Look across the business and technology domains (e.g. POLDAT).
• Look at the total impact of a collection of automated devices, not at each device separately.
• Look at this as a sociotechnical system, involving humans and robots collaborating on the business process.

Self-Correcting

Ensure that the (whole) system is capable of detecting and learning from errors (including near misses).

This typically requires a multi-loop learning process. The machines may handle the inner learning loops, but human intervention will be necessary for the outer loops.
 

Open

Okay, so do you improve the process first and then automate it, or do you automate first? If you search the Internet for "paving the cow-paths", you can find strong opinions on both sides of this argument. But the important point here is that automation shouldn't close down all possibility of future change. Paving the cow-paths may be okay, but not just paving the cow-paths and thinking that's the end of the matter.

In some contexts, this may mean leaving a small proportion of cases to be handled manually, so that human know-how is not completely lost. (Lewis Mumford argued that it is generally beneficial to retain some "craft" production alongside automated "factory" production, as a means to further insight, discovery and invention.)


Transparency

The internal state and decision-making processes of a robot are accessible to (some) users. Provide ways to monitor and explain what the robots are up to, or to provide an audit trail in the event of something going wrong.

---

Related posts

How Soon Might Humans Be Replaced At Work? (November 2015) Could we switch the algorithms off? (July 2017), How many ethical principles? (April 2019), Responsible Transparency (April 2019), Process Automation and Intelligence (August 2019), RPA - Real Value or Painful Experimentation? (August 2019)

Links

Jim Highsmith, Paving Cow Paths (21 June 2005)

Wikipedia

Job Characteristic Theory
Just War Theory

Algorithms and Auditability

@ruchowdh objects to an article by @etzioni and @tianhuil calling for algorithms to audit algorithms. The original article makes the following points.

• Automated auditing, at a massive scale, can systematically probe AI systems and uncover biases or other undesirable behavior patterns. 

• High-fidelity explanations of most AI decisions are not currently possible. The challenges of explainable AI are formidable.  

• Auditing is complementary to explanations. In fact, auditing can help to investigate and validate (or invalidate) AI explanations.

• Auditable AI is not a panacea. But auditable AI can increase transparency and combat bias. 

Rumman Chowdhury points out some of the potential imperfections of a system that relied on automated auditing, and does not like the idea that automated auditing might be an acceptable substitute for other forms of governance. Such a suggestion is not made explicitly in the article, and I haven't seen any evidence that this was the authors' intention. However, there is always a risk that people might latch onto a technical fix without understanding its limitations, and this risk is perhaps what underlies her critique.

In a recent paper, she calls for systems to be "taught to ignore data about race, gender, sexual orientation, and other characteristics that aren’t relevant to the decisions at hand". But how can people verify that systems are not only ignoring these data, but also being cautious about other data that may serve as proxies for race and class, as discussed by Cathy O'Neil? How can they prove that a system is systematically unfair without having some classification data of their own?

And yes, we know that all classification is problematic. But that doesn't mean being squeamish about classification, it just means being self-consciously critical about the tools you are using. Any given tool provides a particular lens or perspective, and it is important to remember that no tool can ever give you the whole picture. Donna Haraway calls this partial perspective.

With any tool, we need to be concerned about how the tool is used, by whom, and for whom. Chowdhury expects people to assume the tool will be in some sense "neutral", creating a "veneer of objectivity"; and she sees the tool as a way of centralizing power. Clearly there are some questions about the role of various stakeholders in promoting algorithmic fairness - the article mentions regulators as well as the ACLU - and there are some major concerns that the authors don't address in the article.

Chowdhury's final criticism is that the article "fails to acknowledge historical inequities, institutional injustice, and socially ingrained harm". If we see algorithmic bias as merely a technical problem, then this leads us to evaluate the technical merits of auditable AI, and acknowledge its potential use despite its clear limitations. And if we see algorithmic bias as an ethical problem, then we can look for various ways to "solve" and "eliminate" bias. @juliapowles calls this a "captivating diversion". But clearly that's not the whole story.

Some stakeholders (including the ACLU) may be concerned about historical and social injustice. Others (including the tech firms) are primarily interested in making the algorithms more accurate and powerful. So obviously it matters who controls the auditing tools. (Whom shall the tools serve?)

What algorithms and audits have in common is that they deliver opinions. A second opinion (possibly based on the auditing algorithm) may sometimes be useful - but only if it is reasonably independent of the first opinion, and doesn't entirely share the same assumptions or perspective. There are codes of ethics for human auditors, so we may want to ask whether automated auditing would be subject to some ethical code.

---

Paul R. Daugherty, H. James Wilson, and Rumman Chowdhury, Using Artificial Intelligence to Promote Diversity (Sloan Management Review, Winter 2019)

Oren Etzioni and Michael Li, High-Stakes AI Decisions Need to Be Automatically Audited (Wired, 18 July 2019)

Donna Haraway, Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. In Simians, Cyborgs and Women (Free Association, 1991)

Cathy O'Neil, Weapons of Math Destruction

Julia Powles, The Seductive Diversion of ‘Solving’ Bias in Artificial Intelligence (7 December 2018)

Related posts: Whom Does the Technology Serve? (May 2019), Algorithms and Governmentality (July 2019)

Algorithms and Governmentality

In the corner of the Internet where I hang out, it is reasonably well understood that big data raises a number of ethical issues, including data ownership and privacy.

There are two contrasting ways of characterizing these issues. One way is to focus on the use of big data to target individuals with increasingly personalized content, such as precision nudging. Thus mass surveillance provides commercial and governmental organizations with large quantities of personal data, allowing them to make precise calculations concerning individuals, and use these calculations for the purposes of influence and control.

Alternatively, we can look at how big data can be used to control large sets or populations - what Foucault calls governmentality. If the prime job of the bureaucrat is to compile lists that could be shuffled and compared (Note 1), then this function is increasingly being taken over by the technologies of data and intelligence - notably algorithms and so-called big data.

Although Deleuze challenges this dichotomy.

We no longer find ourselves dealing with the mass/individual pair. Individuals have become 'dividuals' and masses, samples, data, markets, or 'banks'.

Foucault's version of Bentham's panopticon is often invoked in discussions of mass surveillance, but what was equally important for Foucault was what he called biopower - a type of power that presupposed a closely meshed grid of material coercions rather than the physical existence of a sovereign. [Foucault 2003 via Adams]

People used to talk metaphorically about faceless bureaucracy being a machine, but now we have a real machine, performing the same function with much greater efficiency and effectiveness. And of course, scale.

The machine tended increasingly to dictate the purpose to be served, and to exclude other more intimate human needs. Lewis Mumford

Bureaucracy is usually regarded as a Bad Thing, so it's worth remembering that it is a lot better than some of the alternatives. Bureaucracy should mean you are judged according to an agreed set of criteria, rather than whether someone likes your face or went to the same school as you. Bureaucracy may provide some protection against arbitrary action and certain forms of injustice. And the fact that bureaucracy has sometimes been used by evil regimes for evil purposes isn't sufficient grounds for rejecting all forms of bureaucracy everywhere.

What bureaucracy does do is codify and classify, and this has important implications for discrimination and diversity.

Sometimes discrimination is considered to be a good thing. For example, recruitment should discriminate between those who are qualified to do the job and those who are not, and this can be based either on a subjective judgement or an agreed standard. But even this can be controversial. For example, the College of Policing is implementing a policy that police recruits in England and Wales should be educated to degree level, despite strong objections from the Police Federation.

Other kinds of discrimination such as gender and race are widely disapproved of, and many organizations have an official policy disavowing such discrimination, or affirming a belief in diversity. Despite such policies, however, some unofficial or inadvertent discrimination may often occur, and this can only be discovered and remedied by some form of codification and classification. Thus if campaigners want to show that firms are systematically paying women less than men, they need payroll data classified by gender to prove the point.

Organizations often have a diversity survey as part of their recruitment procedure, so that they can monitor the numbers of recruits by gender, race, religion, sexuality, disability or whatever, thereby detecting any hidden and unintended bias, but of course this depends on people's willingness to place themselves in one of the defined categories. (If everyone ticks the prefer not to say box, then the diversity statistics are not going to be very helpful.)

Daugherty, Wilson and Chowdhury call for systems to be taught to ignore data about race, gender, sexual orientation, and other characteristics that aren’t relevant to the decisions at hand. But there are often other data (such as postcode/zipcode) that are correlated with the attributes you are not supposed to use, and these may serve as accidental proxies, reintroducing discrimination by the back door. The decision-making algorithm may be designed to ignore certain data, based on training data that has been carefully constructed to eliminate certain forms of bias, but perhaps you then need a separate governance algorithm to check for any other correlations.

Bureaucracy produces lists, and of course the lists can either be wrong or used wrongly. For example, King's College London recently apologized for denying access to selected students during a royal visit.

Big data also codifies and classifies, although much of this is done on inferred categories rather than declared ones. For example, some social media platforms infer gender from someone's speech acts (or what Judith Butler would call performativity). And political views can apparently be inferred from food choice. The fact that these inferences may be inaccurate doesn't stop them being used for targetting purposes, or population control.

Cathy O'Neil's statement that algorithms are opinions embedded in code is widely quoted. This may lead people to think that this is only a problem if you disagree with these opinions, and that the main problem with big data and algorithmic intelligence is a lack of perfection. For example, criticizing such technologies as affective computing (to detect emotional state) if they fail to deal with ethnic diversity.

And of course technology companies encourage ethics professors to look at their products from this perspective, firstly because they welcome any ideas that would help them make their products more powerful, and secondly because it distracts the professors from the more fundamental question as to whether they should be doing things like facial recognition in the first place. @juliapowles calls this a "captivating diversion".

But a more fundamental question concerns the ethics of codification and classification. Following a detailed investigation of this topic, published under the title Sorting Things Out, Bowker and Star conclude that "all information systems are necessarily suffused with ethical and political values, modulated by local administrative procedures" (p321).

Black boxes are necessary, and not necessarily evil. The moral questions arise when the categories of the powerful become the taken for granted; when policy decisions are layered into inaccessible technological structures; when one group's visibility comes at the expense of another's suffering. (p320)

At the end of their book (pp324-5), they identify three things they want designers and users of information systems to do. (Clearly these things apply just as much to algorithms and big data as to older forms of information system.)

• Firstly, allow for ambiguity and plurality, allowing for multiple definitions across different domains. They call this recognizing the balancing act of classifying.

• Secondly, the sources of the classifications should remain transparent. If the categories are based on some professional opinion, these should be traceable to the profession or discourse or other authority that produced them. They call this rendering voice retrievable.

• And thirdly, awareness of the unclassified or unclassifiable other. They call this being sensitive to exclusions, and note that residual categories have their own texture that operates like the silences in a symphony to pattern the visible categories and their boundaries (p325).

---

Note 1: This view is attributed to Bruno Latour by Bowker and Star (1999 p 137). However, although Latour talks about paper-shuffling bureaucrats (1987 pp 254-5), I have been unable to find this particular quote.

Rachel Adams, Michel Foucault: Biopolitics and Biopower (Critical Legal Thinking, 10 May 2017)

Geoffrey Bowker and Susan Leigh Star, Sorting Things Out (MIT Press 1999).

Paul R. Daugherty, H. James Wilson, and Rumman Chowdhury, Using Artificial Intelligence to Promote Diversity (Sloan Management Review, Winter 2019)

Gilles Deleuze, Postscript on the Societies of Control (October, Vol 59, Winter 1992), pp. 3-7

Michel Foucault, ‘Society Must be Defended’ Lecture Series at the Collège de France, 1975-76 (2003) (trans. D Macey)

Maša Galič, Tjerk Timan and Bert-Jaap Koops, Bentham, Deleuze and Beyond: An Overview of Surveillance Theories from the Panopticon to Participation (Philos. Technol. 30:9–37, 2017)

Bruno Latour, Science in Action (Harvard University Press 1987)

Lewis Mumford, The Myth of the Machine (1967)

Samantha Murphy, Political Ideology Linked to Food Choices (LiveScience, 24 May 2011)

Julia Powles, The Seductive Diversion of ‘Solving’ Bias in Artificial Intelligence (7 December 2018)

Antoinette Rouvroy and Thomas Berns (translated by Elizabeth Libbrecht), Algorithmic governmentality and prospects of emancipation (Réseaux No 177, 2013)

BBC News, All officers 'should have degrees', says College of Policing (13 November 2015), King's College London sorry over royal visit student bans (4 July 2019)


Related posts

Quotes on Bureaucracy (June 2003), Crude Categories (August 2009), What is the Purpose of Diversity? (January 2010), Affective Computing (March 2019), The Game of Wits between Technologists and Ethics Professors (June 2019), Algorithms and Auditability (July 2019), On the Performativity of Data (August 2021), The Corporate Sorting Hat (September 2021)


Updated 16 July 2019

With Strings Attached

@rachelcoldicutt notes that "Google Docs new grammar suggestion tool doesn’t like the word 'funding' and prefers 'investment' ".

Many business people have an accounting mindset, in which all expenditure must be justified in terms of benefit to the organization, measured in financial terms. When they hear the word "investment", they hold their breath until they hear the word "return".

So when Big Tech funds the debate on AI ethics (Oscar Williams, New Statesman, 6 June 2019), can we infer that Big Tech sees this as an "investment", to which it is entitled to a return or payback?



Related post: The Game of Wits Between Technologists and Ethics Professors (June 2019)

The Game of Wits between Technologists and Ethics Professors

What does #TechnologyEthics look like from the viewpoint of your average ethics professor? 

Not surprisingly, many ethics professors believe strongly in the value of ethics education, and advocate ethics awareness training for business managers and engineers. Provided by people like themselves, obviously.

There is a common pattern among technologists and would-be enterpreneurs to first come up with a "solution", find areas where the solution might apply, and then produce self-interested arguments to explain why the solution matches the problem. Obviously there is a danger of confirmation bias here. Proposing ethics education as a solution for an ill-defined problem space looks suspiciously like the same pattern. Ethicists should understand why it is important to explain what this education achieves, and how exactly it solves the problem.

Please note that I am not arguing against the value of ethics education and training as such, merely complaining that some of the programmes seem to involve little more than meandering through a randomly chosen reading list. @ruchowdh recently posted a particularly egregious example - see below.


Ethics professors may also believe that people with strong ethical awareness, such as themselves, can play a useful role in technology governance - for example, participating in advisory councils.

Some technology companies may choose to humour these academics, engaging them as a PR exercise (ethics washing) and generously funding their research. Fortunately, many of them lack deep understanding of business organizations and of technology, so there is little risk of them causing any serious challenge or embarrasment to these companies.

Professors are always attracted to the kind of work that lends itself to peer-reviewed articles in leading Journals. So it is fairly easy to keep their attention focused on theoretically fascinating questions with little or no practical relevance, such as the Trolley Problem.

Alternatively, they can be engaged to try and "fix" problems with real practical relevance, such as algorithmic bias. @juliapowles calls this a "captivating diversion", distracting academics from the more fundamental question, whether the algorithm should be built at all.

It might be useful for these ethics professors to have deeper knowledge of technology and business, in their social and historical context, enabling them to ask more searching and more relevant questions. (Although some ethics experts have computer science degrees or similar, computer science generally teaches people about specific technologies, not about Technology.) 

The problem isn’t that we are lacking a conceptual tool kit. The problem is people emerging from the technology fields are *aggressively* ignoring the existing vast body of knowledge and existing ethnical frameworks that we can and should apply to the current situation/technology.

— zeynep tufekci (@zeynep) June 11, 2019

But if only a minority of ethics professors possess sufficient knowledge and experience, these will be overlooked for the plum advisory jobs. I therefore advocate compulsory technology awareness training for ethics professors, especially "prominent" ones. Provided by people like myself, obviously.

---

Simon Beard, The Problem with the Trolley Problem (27 September 2019)

Stephanie Burns, Solution Looking For A Problem (Forbes, 28 May 2019)

Casey Fiesler, Tech Ethics Curricula: A Collection of Syllabi (5 July 2018), What Our Tech Ethics Crisis Says About the State of Computer Science Education (5 December 2018)

Mark Graban, Cases of Technology “Solutions” Looking for a Problem? (26 January 2011)

Julia Powles, The Seductive Diversion of ‘Solving’ Bias in Artificial Intelligence (7 December 2018)

Oscar Williams, How Big Tech funds the debate on AI ethics (New Statesman, 6 June 2019)

Related posts:

Leadership and Governance (May 2019), Selected Reading List - Science and Technology Studies (June 2019), With Strings Attached (June 2019)

Updated 27 September 2019

The Nudge as a Speech Act

As I said in my previous post, I don't think we can start to think about the ethics of technology nudges without recognizing the complexity of real-world nudges. So in this post, I shall look at how nudges are communicated in the real world, before considering what their artificial analogues might look like.

---

Once upon a time, nudges were physical rather than verbal - a push on the shoulder perhaps, or a dig in the ribs with an elbow. The meaning was elliptical and depended almost entirely on context. "Nudge nudge, wink wink", as Monty Python used to say.

Even technologically mediated nudges can sometimes be physical, or what we should probably call haptic. For example, the fitness band that vibrates when it thinks you have been sitting for too long.

But many of the acts we now think of as nudges are delivered verbally, as some kind of speech act. But which kind?

The most obvious kind of nudge is a direct suggestion, which may take the form of a weak command. ("Try and eat a little now.") But nudges can also take other illocutionary forms, including questions ("Don't you think the sun is very hot here?") and statements / predictions ("You will find that new nose of yours very useful to spank people with.").

(Readers familiar with Kipling may recognize my examples as the nudges given by the Bi-Coloured-Python-Rock-Snake to the Elephant's Child.)

The force of a suggestion may depend on context and tone of voice. (A more systematic analysis of what philosophers call illocutionary force can be found in the Stanford Encyclopedia of Philosophy, based on Searle and Vanderveken 1985.)

@tonyjoyce raises a good point about tone of voice in electronic messages. Traditionally robots don't do tone of voice, and when a human being talks in a boring monotone we may describe their speech as robotic. But I can't see any reason why robots couldn't be programmed with more varied speech patterns, including tonality, if their designers saw the value of this.

Meanwhile, we already get some differentation from electronic communications. For example I should expect an electronic announcement to "LEAVE THE BUILDING IMMEDIATELY" to have a tone of voice that conveys urgency, and we might think it is inappropriate or even unethical to use the same tone of voice for selling candy. We might put this together with other attention-seeking devices, such as flashing red text. The people who design clickbait clearly understand illocutionary force (even if they aren't familiar with the term). 

A speech act can also gain force by being associated with action. If I promise to donate money to a given charity, this may nudge other people to do the same; but if they see me actually putting the money in the tin, the nudge might be much stronger. But then the nudge might be just as strong if I just put the money in the tin without saying anything, as long as everyone sees me do it. The important point is that some communication takes place, whether verbal or non-verbal, and this returns us to something closer to the original concept of nudge.

From an ethical point of view, there are particular concerns about unobtrusive or subliminal nudges. Yeung has introduced the concept of the Hypernudge, which combines three qualities: nimble, unobtrusive and highly potent. I share her concerns about this combination, but I think it is helpful to deal with these three qualities separately, before looking at the additional problems that may arise when they are combined.

Proponents of the nudge sometimes try to distinguish between unobtrusive (acceptable) and subliminal (unacceptable), but this distinction may be hard to sustain, and many people quote Luc Bovens' observation that nudges "typically work better in the dark". See also Baldwin.


I'm sure there's more to say on this topic, so I may update this post later. Relevant comments always welcome.

---

Robert Baldwin, From regulation to behaviour change: giving nudge the third degree (The Modern Law Review 77/6, 2014) pp 831-857

Luc Bovens, The Ethics of Nudge. In Mats J. Hansson and Till Grüne-Yanoff (eds.), Preference Change: Approaches from Philosophy, Economics and Psychology. (Berlin: Springer, 2008) pp. 207-20

John Danaher, Algocracy as Hypernudging: A New Way to Understand the Threat of Algocracy (Institute for Ethics and Emerging Technologies, 17 January 2017)

J. Searle and D. Vanderveken, Foundations of Illocutionary Logic (Cambridge: Cambridge University Press, 1985)

Karen Yeung, ‘Hypernudge’: Big Data as a Mode of Regulation by Design (Information, Communication and Society (2016) 1,19; TLI Think! Paper 28/2016)


Stanford Encyclopedia of Philosophy: Speech Acts

Related posts: On the Ethics of Technologically Mediated Nudge (May 2019), Nudge Technology (July 2019)


Updated 28 May 2019. Many thanks to @tonyjoyce