Ethics of Transparency and Concealment

Last week I was in Berlin at the invitation of the IEEE to help develop standards for responsible technology (P7000). One of the working groups (P7001) is looking at transparency, especially in relation to autonomous and semi-autonomous systems. In this blogpost, I want to discuss some more general ideas about transparency.

In 1986 I wrote an article for Human Systems Management promoting the importance of visibility. There were two reasons I preferred this word. Firstly, "transparency" is a contronym - it has two opposite senses. When something is transparent, this either means you don't see it, you just see through it, or it means you can really see it. And secondly, transparency appears to be merely a property of an object, whereas visibility is about the relationship between the object and the viewer - visibility to whom?

(P7001 addresses this by defining transparency requirements in relation to different stakeholder groups.)

Although I wasn't aware of this when I wrote the original article, my concept of visibility shares something with Heidegger's concept of Unconcealment (Unverborgenheit). Heidegger's work seems a good starting point for thinking about the ethics of transparency.

Technology generally makes certain things available while concealing other things. (This is related to what Albert Borgmann, a student of Heidegger, calls the Device Paradigm.)

In our time, things are not even regarded as objects, because their only important quality has become their readiness for use. Today all things are being swept together into a vast network in which their only meaning lies in their being available to serve some end that will itself also be directed towards getting everything under control. Levitt

Goods that are available to us enrich our lives and, if they are technologically available, they do so without imposing burdens on us. Something is available in this sense if it has been rendered instantaneous, ubiquitous, safe, and easy. Borgmann

I referred above to the two opposite meanings of the word "transparent". For Heidegger and his followers, the word "transparent" often refers to tools that can be used without conscious thought, or what Heidegger called ready-to-hand (zuhanden). In technology ethics, on the other hand, the word "transparent" generally refers to something (product, process or organization) being open to scrutiny, and I shall stick to this meaning for the remainder of this blogpost.

We are surrounded by technology, we rarely have much idea how most of it works, and usually cannot be bothered to find out. Thus when technological devices are designed to conceal their inner workings, this is often exactly what the users want. How then can we object to concealment?

The ethical problems of concealment depend on what is concealed by whom and from whom, why it is concealed, and whether, when and how it can be unconcealed.

Let's start with the why. Sometimes people deliberately hide things from us, for dishonest or devious reasons. This category includes so-called defeat devices that are intended to cheat regulations. Less clear-cut is when people hide things to avoid the trouble of explaining or justifying them.

(If something is not visible, then we may not be aware that there is something that needs to be explained. So even if we want to maintain a distinction between transparency and explainability, the two concepts are interdependent.)

People may also hide things for aesthetic reasons. The Italian civil engineer Riccardo Morandi designed bridges with the steel cables concealed, which made them difficult to inspect and maintain. The Morandi Bridge in Genoa collapsed in August 2018, killing 43 people.

And sometimes things are just hidden, not as a deliberate act but because nobody has thought it necessary to make them visible. (This is one of the reasons why a standard could be useful.)

We also need to consider the who. For whose benefit are things being hidden? In particular, who is pulling the strings, where is the funding coming from, and where are the profits going - follow the money. In technology ethics, the key question is Whom Does The Technology Serve?

In many contexts, therefore, the main focus of unconcealment is not understanding exactly how something works but being aware of the things that people might be trying to hide from you, for whatever reason. This might include being selective about the available evidence, or presenting the most common or convenient examples and ignoring the outliers. It might also include failing to declare potential conflicts of interest.

For example, the #AllTrials campaign for clinical trial transparency demands that drug companies declare all clinical trials in advance, rather than waiting until the trials are complete and then deciding which ones to publish.

Now let's look at the possibility of unconcealment. Concealment doesn't always mean making inconvenient facts impossible to discover, but may mean making them so obscure and inaccessible that most people don't bother, or creating distractions that divert people's attention elsewhere. So transparency doesn't just entail possibility, it requires a reasonable level of accessibility.

Sometimes too much information can also serve to conceal the truth. Onora O'Neill talks about the "cult of transparency" that fails to produce real trust.

Transparency can produce a flood of unsorted information and misinformation that provides little but confusion unless it can be sorted and assessed. It may add to uncertainty rather than to trust. Transparency can even encourage people to be less honest, so increasing deception and reducing reasons for trust. O'Neill

Sometimes this can be inadvertent. However, as Chesterton pointed out in one of his stories, this can be a useful tactic for those who have something to hide.

Where would a wise man hide a leaf? In the forest. If there were no forest, he would make a forest. And if he wished to hide a dead leaf, he would make a dead forest. And if a man had to hide a dead body, he would make a field of dead bodies to hide it in. Chesterton

Stohl et al call this strategic opacity (via Ananny and Crawford).

Another philosopher who talks about the "cult of transparency" is Shannon Vallor. However, what she calls the "Technological Transparency Paradox" seems to be merely a form of asymmetry: we are open and transparent to the social media giants, but they are not open and transparent to us.

In the absence of transparency, we are forced to trust people and organizations - not only for their honesty but also their competence and diligence. Under certain conditions, we may trust independent regulators, certification agencies and other institutions to verify these attributes on our behalf, but this in turn depends on our confidence in their ability to detect malfeasance and enforce compliance, as well as believing them to be truly independent. (So how transparent are these institutions themselves?) And trusting products and services typically means trusting the organizations and supply chains that produce them, in addition to any inspection, certification and official monitoring that these products and services have undergone.

Instead of seeing transparency as a simple binary (either something is visible or it isn't), it makes sense to discuss degrees of transparency, depending on stakeholder and context. For example, regulators, certification bodies and accident investigators may need higher levels of transparency than regular users. And regular users may be allowed to choose whether to make things visible or invisible. (Thomas Wendt discusses how Heideggerian thinking affects UX design.)

Finally, it's worth noting that people don't only conceal things from others, they also conceal things from themselves, which leads us to the notion of self-transparency. In the personal world this can be seen as a form of authenticity; in the corporate world, it translates into ideas of responsibility, due diligence, and a constant effort to overcome wilful blindness.

If transparency and openness is promoted as a virtue, then people and organizations can make their virtue apparent by being transparent and open, and this may make us more inclined to trust them. We should perhaps be wary of organizations that demand or assume that we trust them, without providing good evidence of their trustworthiness. (The original confidence trickster asked strangers to trust him with their valuables.) The relationship between trust and trustworthiness is complicated. 

---

UK Department of Health and Social Care, Response to the House of Commons Science and Technology Committee report on research integrity: clinical trials transparency (UK Government Policy Paper, 22 February 2019) via AllTrials

Mike Ananny and Kate Crawford, Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability (new media and society 2016) pp 1–17

Albert Borgmann, Technology and the Character of Contemporary Life (University of Chicago Press, 1984)

G.K. Chesterton, The Sign of the Broken Sword (The Saturday Evening Post, 7 January 1911)

Martin Heidegger, The Question Concerning Technology (Harper 1977) translated and with an introduction by William Lovitt

Onora O'Neill, Trust is the first casualty of the cult of transparency (Telegraph, 24 April 2002)

Cynthia Stohl, Michael Stohl and P.M. Leonardi, Managing opacity: Information visibility and the paradox of transparency in the digital age (International Journal of Communication Systems 10, January 2016) pp 123–137.

Richard Veryard, The Role of Visibility in Systems (Human Systems Management 6, 1986) pp 167-175 (this version includes some further notes dated 1999)

Thomas Wendt, Designing for Transparency and the Myth of the Modern Interface (UX Magazine, 26 August 2013)

Stanford Encyclopedia of Philosophy: Heidegger, Technological Transparency Paradox

Wikipedia: Confidence Trick, Follow The Money, Ponte Morandi, Regulatory Capture,Willful Blindness


Related posts: Defeating the Device Paradigm (October 2015), Transparency of Algorithms (October 2016), Pax Technica (November 2017), Responsible Transparency (April 2019), Whom Does The Technology Serve (May 2019)

Leadership versus Governance

@j2bryson has commented on her blog about the fate of Google's Advanced Technology External Advisory Council (ATEAC), to which she had been appointed.

She argues that the people who were appointed to the ATEAC were selected because they were "prominent" in the field. She notes that "although being prominent doesn't mean you're the best, it probably does mean you're at least pretty good, at least at something".

Ignoring the complexities of university politics, academics generally achieve prominence because they are pretty good at having interesting and original ideas, publishing papers and books, coordinating research, and supervising postgraduate work, as well as representing the field in wider social and intellectual forums (e.g. TED talks). Clearly that can be regarded as an important type of leadership.

Bryson argues that leading is about problem-solving. And clearly there are some aspects of problem-solving in what has brought her to prominence, although that's certainly not the whole story.

But that argument completely misses the point. The purpose of the ATEAC was not problem-solving. Google does not need help with problem-solving, it employs thousands of extremely clever people who spend all day solving problems (although it may sometimes need a bit of help in the diversity stakes).

The stated purpose of the ATEAC was to help Google implement its AI principles. In other words, governance.

When Google published its AI principles last year, the question everyone was asking was about governance:

• @mer__edith (Twitter 8 June 2018, tweet no longer available) called for "strong governance, independent external oversight and clarity"

• @katecrawford (Twitter 8 June 2018) asked "How are they implemented? Who decides? There's no mention of process, or people, or how they'll evaluate if a tool is 'beneficial'. Are they... autonomous ethics?" 

• and @EricNewcomer (Bloomberg 8 June 2018) asked "who decides if Google has fulfilled its commitments".

Google's appointment of an "advisory" council was clearly a half-hearted attempt to answer this question.

Bryson points out that Kay Coles James (the most controversial appointee) had some experience writing technology policy. But what a truly independent governance body needs is experience monitoring and enforcing policy, which is not the same thing at all.

People talk a lot about transparency in relation to technology ethics. Typically this refers to being able to "look inside" an advanced technological product, such as an algorithm or robot. But transparency is also about process and organization - ability to scrutinize the risk assessment and the design and the potential conflicts of interest. There are many people performing this kind of scrutiny on a full-time basis within large organizations or ecosystems, with far more experience of extremely large and complex development programmes than your average professor.

Had Google really wanted a genuinely independent governance body to scrutinize them properly, could they have appointed a different set of experts? Can people appointed and paid by Google ever be regarded as genuinely independent? And doesn't the word "advisory" give the game away? As Brustein and Bergen point out, the actual decisions are made by an internal body, the Advanced Technology Review Council, and external critics doubt that this body will ever seriously challenge Google's commercial or strategic interests.

Veena Dubal suggests that the most effective governance over Google is currently coming from Google's own workforce. It seems that their protests were significant in getting Google to disband the ATEAC, while earlier protests (re Project Maven) had led to the production of the AI principles in the first place. Clearly the kind of courageous leadership demonstrated by people like Meredith Whittaker isn't just about problem-solving.

---

Joshua Brustein and Mark Bergen, The Google AI Ethics Board With Actual Power Is Still Around (Bloomberg, 6 April 2019)

Joanna Bryson, What we lost when we lost Google ATEAC (7 April 2019), What leaders are actually for (13 May 2019)

Veena Dubal, Who stands between you and AI dystopia? These Google activists (The Guardian, 3 May 2019)

Bobbie Johnson and Gideon Lichfield, Hey Google, sorry you lost your ethics council, so we made one for you (MIT Technology Review 6 April 2019

Abner Li, Google details formal review process for enforcing AI Principles, plans external advisory group (9to5 Google, 18 December 2018

Eric Newcomer, What Google's AI Principles Left Out (Bloomberg 8 June 2018)

Kent Walker, An external advisory council to help advance the responsible development of AI (Google, 26 March 2019, updated 4 April 2019)


Related post: Data and Intelligence Principles From Major Players (June 2018)

Updated 15 May 2019

Responsible Transparency

It is difficult to see how we can achieve an ethical technology without some kind of transparency, although we are still trying to work out how this could be achieved in an effective yet responsible manner. There are several concerns that are thought to conflict with transparency, including commercial advantage, security, privacy, and the risk of the device being misused or "gamed" by adversaries. There is a good summary of these issues in Mittelstadt et al (2016).

An important area where demands for transparency conflict with demands for confidentiality is with embedded software that serves the interests of the manufacturer rather than the consumer or the public. For example, a few years ago we learned about a "defeat device" that VW had built in order to cheat the emissions regulations; similar devices have been discovered in televisions to falsify energy consumption ratings.

Even when the manufacturers aren't actually breaking the law, they have a strong commercial interest in concealing the purpose and design of these systems, and they use Digital Rights Management (DRM) and the US Digital Millenium Copyright Act (DMCA) to prevent independent scrutiny. In what appears to be an example of regulatory capture, car manufacturers were abetted by the US EPA, which was persuaded to inhibit transparency of engine software, on the grounds that this would enable drivers to cheat the emissions regulations.

Defending the EPA, David Golumbia sees a choice between two trust models, which he calls democratic and cyberlibertarian. For him, the democratic model "puts trust in bodies specifically chartered and licensed to enforce regulations and laws", such as the EPA, whereas in the cyberlibertarian model, it is the users themselves who get the transparency and can scrutinize how something works. In other words, trusting the wisdom of crowds, or what he patronizingly calls "ordinary citizen security researchers".

(In their book on Trust and Mistrust, John Smith and Aidan Ward describe four types of trust. Golumbia's democratic model involves top-down trust, based on the central authority of the regulator, while the cyberlibertarian model involves decentralized network trust.)

Golumbia argues that the cyberlibertarian position is incoherent. 

"It says, on the one hand, we should not trust manufacturers like Volkswagen to follow the law. We shouldn’t trust them because people, when they have self-interest at heart, will pursue that self-interest even when the rules tell them not to. But then it says we should trust an even larger group of people, among whom many are no less self-interested, and who have fewer formal accountability obligations, to follow the law."

One problem with this argument is that it appears to confuse scrutiny with compliance. Cyberlibertarians may be strongly in favour of deregulation, but increasing transparency isn't only advocated by cyberlibertarians and doesn't necessarily imply deregulation. It could be based on a recognition that regulatory scrutiny and citizen scrutiny are complementary, given two important facts. Firstly, however powerful the tools at their disposal the regulators don't always spot everything; and secondly, regulators are sometimes subject to improper influence from the companies they are supposed to be regulating (so-called regulatory capture). Therefore having independent scrutiny as well as central regulation increases the likelihood that hazards will be discovered and dealt with. This could include the detection of algorithmic bias or previously unidentified hazards/vulnerabilities/malpractice.

Another small problem with his argument is that the defeat device had already hoodwinked the EPA and other regulators for many years.

Golumbia claims that "what the cyberlibertarians want, even demand, is for everyone to have the power to read and modify the emissions software in their cars" and complains that "the more we put law into the hands of those not specifically entrusted to follow it, the more unethical behavior we will have". It is certainly true that some of the advocates of open source are also advocating "right to repair" and customization rights. But there were two separate requests for exemptions to DMCA - one for testing and one for modification. And the researchers quoted by Kyle Wiens, who were disadvantaged by the failure of the EPA to mandate a specific exemption to DMCA to allow safety and security tests, were not casual libertarians or "ordinary citizens" but researchers at the International Council of Clean Transportation and West Virginia University.

It ought to be possible for regulators and academic researchers to collaborate productively in scrutinizing an industry, provided that clear rules, protocols and working practices are established for responsible scrutiny. Perhaps researchers might gain some protection from regulatory action or litigation by notifying a regulator in advance, or by prompt notification of any discovered issues. For example, the UK Data Protection Act 2018 (section 172) defines what it calls "effectiveness testing conditions", under which researchers can legitimately attempt to crack the anonymity of deidentified personal data. Among other things, a successful attempt must be notified to the Information Commissioner within 72 hours.

Meanwhile, in the cybersecurity world there are fairly well-established protocols for responsible disclosure of vulnerabilities, and in some cases rewards are paid to the researchers who find them, provided they are disclosed responsibly. Although not all of us have the expertise to understand the technical detail, the existence of this kind of independent scrutiny should make us all feel more confident about the safety, reliability and general trustworthiness of the products in question.

---

David Golumbia, The Volkswagen Scandal: The DMCA Is Not the Problem and Open Source Is Not the Solution (6 October 2015)

Brent Mittelstadt et al, The ethics of algorithms: Mapping the debate (Big Data and Society July–December 2016)

Jonathan Trull, Responsible Disclosure: Cyber Security Ethics (CSO Cyber Security Pulse, 26 February 2015)

Aidan Ward and John Smith, Trust and Mistrust (Wiley 2003)

Kyle Wiens, Opinion: The EPA shot itself in the foot by opposing rules that could've exposed VW (The Verge, 25 September 2015)


Related posts: Four Types of Trust (July 2004), Defeating the Device Paradigm (October 2015)

The Transparency of Algorithms

Algorithms have been getting a bad press lately, what with Cathy O'Neil's book and Zeynap Tufekci's TED talk. Now the German Chancellor, Angela Merkel, has weighed into the debate, calling for major Internet firms (Facebook, Google and others) to make their algorithms more transparent.

There are two main areas of political concern. The first (raised by Mrs Merkel) is the control of the news agenda. Politicians often worry about the role of the media in the political system when people only pick up the news that fits their own point of view, but this is hardly a new phenomenon. Even in the days before the Internet, few people used to read more than one newspaper, and most people would prefer to read the newspapers that confirm their own prejudices. Furthermore, there have been recent studies that show that even when you give different people exactly the same information, they will interpret it differently, in ways that reinforce their previous beliefs. So you can't blame the whole Filter Bubble thing on Facebook and Google.

But they undoubtedly contribute further to the distortion. People get a huge amount of information via Facebook, and Facebook systematically edits out the uncomfortable stuff. It aroused particular controversy recently when its algorithms decided to censor a classic news photograph from the Vietnam war.

Update: Further criticism from Tufekci and others immediately following the 2016 US Election

2016 was a close election where filter bubbles & algorithmic funneling was weaponized for spreading misinformation. https://t.co/QCb4KG1gTV pic.twitter.com/cbgrj1TqFb

— Zeynep Tufekci (@zeynep) November 9, 2016

---

The second area of concern has to do with the use of algorithms to make critical decisions about people's lives. The EU regards this as (among other things) a data protection issue, and privacy activists are hoping for provisions within the new General Data Protection Regulation (GDPR) that will confer a "right to an explanation" upon data subjects. In other words, when people are sent to prison based on an algorithm, or denied a job or health insurance, it seems reasonable to allow them to know what criteria these algorithmic decisions were based on.

Reasonable but not necessarily easy. Many of these algorithms are not coded in the old-fashioned way, but developed using machine learning. So the data scientists and programmers responsible for creating the algorithm may not themselves know exactly what the criteria are. Machine learning is basically a form of inductive reasoning, using data about the past to predict the future. As Hume put it, this assumes that “instances of which we have had no experience resemble those of which we have had experience”.

In a Vanity Fair panel discussion entitled “What Are They Thinking? Man Meets Machine,” a young black woman tried unsuccessfully to explain the problem of induction and biased reasoning to Sebastian Thrun, formerly head of Google X.

At the end of the panel on artificial intelligence, a young black woman asked Thrun whether bias in machine learning “could perpetuate structural inequality at a velocity much greater than perhaps humans can.” She offered the example of criminal justice, where “you have a machine learning tool that can identify criminals, and criminals may disproportionately be black because of other issues that have nothing to do with the intrinsic nature of these people, so the machine learns that black people are criminals, and that’s not necessarily the outcome that I think we want.”

In his reply, Thrun made it sound like her concern was one about political correctness, not unconscious bias. “Statistically what the machines do pick up are patterns and sometimes we don’t like these patterns. Sometimes they’re not politically correct,” Thrun said. “When we apply machine learning methods sometimes the truth we learn really surprises us, to be honest, and I think it’s good to have a dialogue about this.”

In other words, Thrun assumed that whatever the machine spoke was Truth, and he wasn't willing to acknowledge the possibility that the machine might latch onto false patterns. Even if the algorithm is correct, it doesn't take away the need for transparency; but if there is the slightest possibility that the algorithm might be wrong, the need for transparency is all the greater. And evidence is that some of the algorithms are grossly wrong.


In this post, I've talked about two of the main concerns about algorithms - firstly the news agenda filter bubble, and secondly the critical decisions affecting individuals. In both cases, people are easily misled by the apparent objectivity of the algorithm, and are often willing to act as if the algorithm is somehow above human error and human criticism. Of course algorithms and machine learning are useful tools, but an illusion of infallibility is dangerous and ethically problematic.

---

Rory Cellan-Jones, Was it Facebook 'wot won it'? (BBC News, 10 November 2016)

Ethan Chiel, EU citizens might get a ‘right to explanation’ about the decisions algorithms make (5 July 2016)

Kate Connolly, Angela Merkel: internet search engines are 'distorting perception' (Guardian, 27 October 2016)

Bryce Goodman, Seth Flaxman, European Union regulations on algorithmic decision-making and a "right to explanation" (presented at 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016), New York, NY)

Mike Masnick, Activists Cheer On EU's 'Right To An Explanation' For Algorithmic Decisions, But How Will It Work When There's Nothing To Explain? (Tech Dirt, 8 July 2016)

Fabian Reinbold, Warum Merkel an die Algorithmen will (Spiegel, 26 October 2016)

Nitasha Tiku, At Vanity Fair’s Festival, Tech Can’t Stop Talking About Trump (BuzzFeed, 24 October 2016) HT @noahmccormack

Julia Carrie Wong, Mark Zuckerberg accused of abusing power after Facebook deletes 'napalm girl' post (Guardian, 9 September 2016)

New MIT technique reveals the basis for machine-learning systems’ hidden decisions (Kutzweil News, 31 October 2016) HT @jhagel

Video: When Man Meets Machine (Vanity Fair, 19 October 2016)

See Also
The Problem of Induction (Stanford Encyclopedia of Philosophy, Wikipedia)


Related Posts
The Shelf-Life of Algorithms (October 2016)
Weapons of Math Destruction (October 2016)

Updated 10 November 2016