Leadership versus Governance

@j2bryson has commented on her blog about the fate of Google's Advanced Technology External Advisory Council (ATEAC), to which she had been appointed.

She argues that the people who were appointed to the ATEAC were selected because they were "prominent" in the field. She notes that "although being prominent doesn't mean you're the best, it probably does mean you're at least pretty good, at least at something".

Ignoring the complexities of university politics, academics generally achieve prominence because they are pretty good at having interesting and original ideas, publishing papers and books, coordinating research, and supervising postgraduate work, as well as representing the field in wider social and intellectual forums (e.g. TED talks). Clearly that can be regarded as an important type of leadership.

Bryson argues that leading is about problem-solving. And clearly there are some aspects of problem-solving in what has brought her to prominence, although that's certainly not the whole story.

But that argument completely misses the point. The purpose of the ATEAC was not problem-solving. Google does not need help with problem-solving, it employs thousands of extremely clever people who spend all day solving problems (although it may sometimes need a bit of help in the diversity stakes).

The stated purpose of the ATEAC was to help Google implement its AI principles. In other words, governance.

When Google published its AI principles last year, the question everyone was asking was about governance:

• @mer__edith (Twitter 8 June 2018, tweet no longer available) called for "strong governance, independent external oversight and clarity"

• @katecrawford (Twitter 8 June 2018) asked "How are they implemented? Who decides? There's no mention of process, or people, or how they'll evaluate if a tool is 'beneficial'. Are they... autonomous ethics?" 

• and @EricNewcomer (Bloomberg 8 June 2018) asked "who decides if Google has fulfilled its commitments".

Google's appointment of an "advisory" council was clearly a half-hearted attempt to answer this question.

Bryson points out that Kay Coles James (the most controversial appointee) had some experience writing technology policy. But what a truly independent governance body needs is experience monitoring and enforcing policy, which is not the same thing at all.

People talk a lot about transparency in relation to technology ethics. Typically this refers to being able to "look inside" an advanced technological product, such as an algorithm or robot. But transparency is also about process and organization - ability to scrutinize the risk assessment and the design and the potential conflicts of interest. There are many people performing this kind of scrutiny on a full-time basis within large organizations or ecosystems, with far more experience of extremely large and complex development programmes than your average professor.

Had Google really wanted a genuinely independent governance body to scrutinize them properly, could they have appointed a different set of experts? Can people appointed and paid by Google ever be regarded as genuinely independent? And doesn't the word "advisory" give the game away? As Brustein and Bergen point out, the actual decisions are made by an internal body, the Advanced Technology Review Council, and external critics doubt that this body will ever seriously challenge Google's commercial or strategic interests.

Veena Dubal suggests that the most effective governance over Google is currently coming from Google's own workforce. It seems that their protests were significant in getting Google to disband the ATEAC, while earlier protests (re Project Maven) had led to the production of the AI principles in the first place. Clearly the kind of courageous leadership demonstrated by people like Meredith Whittaker isn't just about problem-solving.

---

Joshua Brustein and Mark Bergen, The Google AI Ethics Board With Actual Power Is Still Around (Bloomberg, 6 April 2019)

Joanna Bryson, What we lost when we lost Google ATEAC (7 April 2019), What leaders are actually for (13 May 2019)

Veena Dubal, Who stands between you and AI dystopia? These Google activists (The Guardian, 3 May 2019)

Bobbie Johnson and Gideon Lichfield, Hey Google, sorry you lost your ethics council, so we made one for you (MIT Technology Review 6 April 2019

Abner Li, Google details formal review process for enforcing AI Principles, plans external advisory group (9to5 Google, 18 December 2018

Eric Newcomer, What Google's AI Principles Left Out (Bloomberg 8 June 2018)

Kent Walker, An external advisory council to help advance the responsible development of AI (Google, 26 March 2019, updated 4 April 2019)


Related post: Data and Intelligence Principles From Major Players (June 2018)

Updated 15 May 2019

Decentred Regulation and Responsible Technology

In 2001-2, Julia Black published some papers discussing the concept of Decentred Regulation, with particular relevance to the challenges of globalization. In this post, I shall summarize her position as I understand it, and apply it to the topic of responsible technology.

Black identifies a number of potential failures in regulation, which are commonly attributed to command and control (CAC) regulation - regulation by the state through the use of legal rules backed by (often criminal) sanctions.

• instrument failure - the instruments used (laws backed by sanctions) are inappropriate and unsophisticated

• information and knowledge failure - governments or other authorities have insufficient knowledge to be able to identify the causes of problems, to design solutions that are appropriate, and to identify non-compliance

• implementation failure - implementation of the regulation is inadequate

• motivation failure and capture theory - those being regulated are insufficiently inclined to comply, and those doing the regulating are insufficiently motivated to regulated in the public interest

For Black, decentred regulation represents an alternative to CAC regulation, based on five key challenges. These challenges echo the ideas of Michel Foucault around governmentality, which Isabell Lorey (2005, p23) defines as "the structural entanglement between the government of a state and the techniques of self-government in modern Western societies".

• complexity - emphasising both causal complexity and the complexity of interactions between actors in society (or systems), which are imperfectly understood and change over time

• fragmentation - of knowledge, and of power and control. This is not just a question of information asymmetry; no single actor has sufficient knowledge, or sufficient control of the instruments of regulation.

• interdependencies - including the co-production of problems and solutions by multiple actors across multiple jurisdictions (and amplified by globalization)

• ungovernability - Black explains this in terms of autopoiesis, the self-regulation, self-production and self-organisation of systems. As a consequence of these (non-linear) system properties, it may be difficult or impossible to control things directly

• the rejection of a clear distinction between public and private - leading to rethinking the role of formal authority in governance and regulation

In response to these challenges, Black describes a form of regulation with the following characteristics

• hybrid - combining governmental and non-governmental actors

• multifaceted - using a number of different strategies simultaneously or sequentially

• indirect - this appears to link to what (following Teubner) she calls reflexive regulation - for example setting the decision-making procedures within organizations in such a way that the goals of public policy are achieved

And she asks if it counts as regulation at all, if we strip away much of what people commonly associate with regulation, and if it lacks some key characteristics, such as intentionality or effectiveness. Does regulation have to be what she calls "cybernetic", which she defines in terms of three functions: standard-setting, information gathering and behaviour modification? (Other definitions of "cybernetic" are available, such as Stafford Beer's Viable Systems Model.)

Meanwhile, how does any of this apply to responsible technology? Apart from the slogan, what I'm about to say would be true of any large technology company, but I'm going to talk about Google, for no other reason than its former use of the slogan "Don't Be Evil". (This is sometimes quoted as "Do No Evil", but for now I shall ignore the difference between being evil and doing evil.) What holds Google to this slogan is not primarily government regulation (mainly US and EU) but mostly an interconnected set of other forces, including investors, customers (much of its revenue coming from advertising), public opinion and its own workforce. Clearly these stakeholders don't all have the same view on what counts as Evil, or what would be an appropriate response to any specific ethical concern.

If we regard each of these stakeholder domains as a large-scale system, each displaying complex and sometimes apparently purposive behaviour, then the combination of all of them can be described as a system of systems. Mark Maier distinguished between three types of System of System (SoS), which he called Directed, Collaborative and Virtual; Philip Boxer identifies a fourth type, which he calls Acknowledged.

• Directed - under the control of a single authority

• Acknowledged - some aspects of regulation are delegated to semi-autonomous authorities, within a centrally planned regime 

• Collaborative - under the control of multiple autonomous authorities, collaborating voluntarily to achieve an agreed purpose

• Virtual - multiple authorities with no common purpose

Black's notion of "hybrid" clearly moves from the Directed type to one of the other types of SoS. But which one? Where technology companies are required to interpret and enforce some rules, under the oversight of a government regulator, this might belong to the Acknowledged type. For example, social media platforms being required to enforce some rules about copyright and intellectual property, or content providers being required to limit access to those users who can prove they are over 18. (Small organizations sometimes complain that this kind of regime tends to favour larger organizations, which can more easily absorb the cost of building and implementing the necessary mechanisms.) 

However, one consequence of globalization is that there is no single regulatory authority. In Data Protection, for example, the tech giants are faced with different regulations in different jurisdictions, and can choose whether to adopt a single approach worldwide, or to apply the stricter rules only where necessary. (So for example, Microsoft has announced it will apply GDPR rules worldwide, while other technology companies have apparently migrated personal data of non-EU citizens from Ireland to the US in order to avoid the need to apply GDPR rules to these data subjects.)

But although the detailed rules on privacy and other ethical issues vary significantly between countries and jurisdictions, there is a reasonably broad acceptance of the principle that some privacy is probably a Good Thing. Similarly, although dozens of organizations have published rival sets of ethical principles for AI or robotics or whatever, there appears to be a fair amount of common purpose between them, indicating that all these organizations are travelling (or pretending to travel) in more or less the same direction. Therefore it seems reasonable to regard this as the Collaborative type.


Decentred regulation raises important questions of agency and purpose. And if it is to be maintain relevance and effectiveness in a rapidly changing technological world, there needs to be some kind of emergent / collective intelligence conferring the ability to solve not only downstream problems (making judgements on particular cases) but also upstream problems (evolving governance principles and practices).

---

Julia Black, Decentring Regulation: Understanding the Role of Regulation and Self-Regulation in a ‘Post-Regulatory’ World (Current Legal Problems, Volume 54, Issue 1, 2001) pp 103–146

Julia Black, Decentred Regulation (LSE Centre for Analysis of Risk and Regulation, 2002)

Philip Boxer, Architectures that integrate differentiated behaviours (Asymmetric Leadership, August 2011)

Martin Innes, Bethan Davies and Morag McDermont, How Co-Production Regulates (Social and Legal Studies, 2008)

Mark W. Maier, Architecting Principles for Systems-of-Systems (Systems Engineering, Vol 1 No 4, 1998)

Isabell Lorey, State of Insecurity (Verso 2015)

Gunther Teubner, Substantive and Reflexive Elements in Modern Law (Law and Society Review, Vol. 17, 1983) pp 239-285

Wikipedia: Don't Be Evil,


Related posts: How Many Ethical Principles (April 2019), Algorithms and Governmentality (July 2019)

Ethics committee raises alarm

Dr @BenGoldacre was the keynote speaker at an IT conference I attended recently. In the context of the growing interest in technology ethics, especially AI ethics, I asked him what IT could learn from medical ethics. He responded by criticising the role of the ethics committee, and mentioned a recent case in which an ethics committee had blocked an initiative that could have collected useful data concerning the effectiveness of statins. This is an example of what Goldacre calls the ethical paradox. As he wrote in 2008,

"You can do something as part of a treatment program, entirely on a whim, and nobody will interfere, as long as it’s not potty (and even then you’ll probably be alright). But the moment you do the exact same thing as part of a research program, trying to see if it actually works or not, adding to the sum total of human knowledge, and helping to save the lives of people you’ll never meet, suddenly a whole bunch of people want to stuck their beaks in."

Within IT, there is considerable controversy about the role of the ethics committee, especially after Google appointed and then disbanded its Ethics Board. In a recent article for Slate, @internetdaniel complains about company ethics boards offering "advice" rather than meaningful oversight, and calls this ethics theatre. @ruchowdh prefers to call it ethics washing.

So I was particularly interested to find a practical example of an ethics committee in action in this morning's Guardian. While the outcome of this case is not yet clear, there seem to be some positive indicators in @sloumarsh's report.

Firstly, the topic (predictive policing) is clearly an important and difficult one. It is not just about applying a simplistic set of ethics principles, but balancing a conflicting set of interests and concerns. (As @oscwilliams reports, this topic has already got the attention of the Information Commissioner's Office.)

Secondly, the discussion is in the open, and the organization is making the right noises. “This is an important area of work, that is why it is right that it is properly scrutinised and those details are made public.” (This contrasts with some of the bad examples of medical ethics cited by Goldacre.)

Thirdly, the ethics committee is (informally) supported by a respected external body (Liberty), which adds weight to its concerns, and has helped bring the case to public attention. (Credit @Hannah_Couchman)

Fourthly, although the ethics committee mandate only applies to a single police force (West Midlands), its findings are likely to be relevant to other police forces across the UK. For those forces that do not have a properly established governance process of their own, the default path may be to follow the West Midlands example.


So it is possible (although not guaranteed) that this particular case may produce a reasonable outcome, with a valuable contribution from the ethics committee and its external supporters. But it is worrying if this is what it takes for governance to work, because this happy combination of positive indicators will not be present in most other cases.

---

Ben Goldacre, Where’s your ethics committee now, science boy? (Bad Science Blog,23 February 2008), When Ethics Committees Kill (Bad Science Blog, 26 March 2011), Taking transparency beyond results: ethics committees must work in the open (Bad Science Blog, 23 September 2016)

Sarah Marsh, Ethics committee raises alarm over 'predictive policing' tool (The Guardian, 20 April 2019)

Daniel Susser, Ethics Alone Can’t Fix Big Tech (Slate, 17 April 2019)

Jane Wakefield, Google's ethics board shut down (BBC News, 5 April 2019)

Oscar Williams, Some of the UK’s biggest police forces are using algorithms to predict crime (New Statesman, 4 February 2019)

Upstream Ethics

We can roughly characterize two places where ethical judgements are called for, which I shall call upstream and downstream. There is some inconsistency about how these terms are used in the literature; here are my definitions.

I use the term upstream ethics to refer to

• Establishing priorities and goals - for example, emphasising precaution and prevention
• Establishing general principles, processes and practices
• Embedding these in standards, policies and codes of practices
• Enacting laws and regulations
• Establishing governance - monitoring and enforcement
• Training and awareness - enabling, encouraging and empowering people to pay due attention to ethical concerns
• Approving and certifying technologies, products, services and supply chains. 

Some people call these (or some of them) "pre-normative" ethics.


I use the term downstream ethics to refer to

• Making judgements about a specific instance
• Eliciting values and concerns in a specific context as part of the requirements elicitation process
• Detecting ethical warning signals
• Applying, interpreting and extending upstream ethics to a specific case or challenge
• Auditing compliance with upstream ethics

There is also a feedback and learning loop, where downstream issues and experiences are used to evaluate and improve the efficacy of upstream ethics.


Downstream ethics does not take place at a single point in time. I use the term early downstream to mean paying attention to ethical questions at an early stage of an initiative. Among other things, this may involve picking up early warning signals of potential ethical issues affecting a particular case. Early downstream means being ethically proactive - introducing responsibility by design - while late downstream means reacting to ethical issues only after they have been forced upon you by other stakeholders.

However, some writers regard what I'm calling early downstream as another type of upstream. Thus Ozdemir and Knoppers talk about Type 1 and Type 2 upstream. And John Paul Slosar writes

"Early identification of the ethical dimensions of person-centered care before the point at which one might recognize the presence of a more traditionally understood “ethics case” is vital for Proactive Ethics Integration or any effort to move ethics upstream. Ideally, there would be a set of easily recognizable ethics indicators that would signal the presence of an ethics issue before it becomes entrenched, irresolvable or even just obviously apparent."

For his part, as a lawyer specializing in medical technology, Christopher White describes upstream ethics as a question of confidence and supply - in other words, having some level of assurance about responsible sourcing and supply of component technologies and materials. He mentions a range of sourcing issues, including conflict minerals, human slavery, and environmentally sustainable extraction.

Extending this point, advanced technology raises sourcing issues not only for physical resources and components, but also for intangible inputs like data and knowledge. For example, medical innovation may be dependent upon clinical trials, while machine learning may be dependent on large quantities of training data. So there are important questions of upstream ethics as to whether these data were collected properly and responsibly, which may affect the extent to which these data can be used responsibly, or at all. As Rumman Chowdhury asks, "How do we institute methods of ethical provenance?"

There is a trade-off between upstream effort and downstream effort. If you take more care upstream, you should hope to experience fewer difficulties downstream. Conversely, some people may wish to invest little or no time upstream, and face the consequences downstream. One way of thinking about responsibility is shifting the balance of effort and attention upstream. But obviously you can't work everything out upstream, so you will always have further stuff to do downstream.

So it's about getting the balance right, and joining the dots. Wherever we choose to draw the line between "upstream" and "downstream", with different institutional arrangements and mobilizing different modes of argumentation and evidence at different stages, "upstream" and "downstream" still need to be properly connected, as part of a single ethical system.

---

(In a separate post, Ethics - Soft and Hard, I discuss Luciano Floridi's use of the terms hard and soft ethics, which covers some of the same distinctions I'm making here but in a way I find more confusing.)

Os Keyes, Nikki Stevens, and Jacqueline Wernimont, The Government Is Using the Most Vulnerable People to Test Facial Recognition Software (Slate 17 March 2019) HT @ruchowdh

Vural Ozdemir and Bartha Maria Knoppers, One Size Does Not Fit All: Toward “Upstream Ethics”? (The American Journal of Bioethics, Volume 10 Issue 6, 2010) https://doi.org/10.1080/15265161.2010.482639

John Paul Slosar, Embedding Clinical Ethics Upstream: What Non-Ethicists Need to Know (Health Care Ethics, Vol 24 No 3, Summer 2016)

Christopher White, Looking the Other Way: What About Upstream Corporate Considerations? (MedTech, 29 Mar 2017)


Updated 18 March 2019

Intelligence and Governance

Katy Steward of @TheKingsFund asks What Makes a Board Effective? (Feb 2013). She's looking specifically at the role of the Board in the National Health Service, but there is much that can be generalized to other contexts. She asks some key questions for any given board.

• Are its members individually effective and do they communicate effectively – for example, do they challenge themselves and others?
• Do they use energetic presentations and have insightful conversations?
• Do they support their colleagues and have good decision-making skills?

In this post, I want to develop this line of thinking further by exploring what the concept of organizational intelligence implies for boards.


1. Boards need to know what is going on.

• Multiple and diverse sources of information - both quantitative and qualitative
• Understanding how information is filtered, and a willingness to view unfiltered information as necessary. 
• Ability to identify areas of concern, and initiate detailed investigation 

2. Boards need to make sense of what is going on.

• Ability to see things from different perspectives - patient quality, professional excellence, financial accountability, social accountability. 
• Ability to see the detail as well as the big picture. 
• Courage to investigate and explore any discrepancies, and not to be satisfied with easy denial.

3. Boards need to ensure that all decisions, policies and procedures are guided by both vision and reality. This includes decisions taken by the board itself, as well as decisions taken at all levels of management.

• Decisions and actions are informed by values and priorities, and reinforce these values. (People both inside and outside the organization will infer your true values not from your words but from your actions.) 
• Decisions and actions are guided by evidence wherever possible. Ongoing decisions and policies are open to revision according to the outcomes they yield.
• Decision-making by consent (Robertson)

4. Boards need to encourage learning.

• Effective feedback loops are established, monitoring outcomes and revising decisions and policies where necessary. 
• Courage to experiment. Ability to tolerate temporary reduction in productivity during problem-solving and learning curve. Supporting people and teams when they are out of their comfort zone. 
• Willingness to learn lessons from anywhere, not just a narrow set of approved exemplars.

5. Boards need to encourage knowledge-sharing

• All kinds of experience and expertise may be relevant 
• Overcoming the "silos" and cultural differences 
• The collective memory should be strong and coherent enough to support the organization's values, but not so strong as to inhibit change.

6. Boards work as a team, and collaborate with other teams

• Effective communication and collaboration within the board - don't expect each board member to do everything. 
• Effective communication and collaboration with other groups and organizations.
• Circle Organization (Robertson)

---

Note: The six points I've discussed here correspond to the six core capabilities of organizational intelligence, as described in my Organizational Intelligence eBook.


See also

Brian Robertson, The Sociocratic Method. A Dutch model of corporate governance harnesses self-organization to provide agility and a voice to all participants (Strategy+Business Aug 2006)

Steve Waddell, Wicked Problems, Governance as Learning Systems (Feb 2013)


Updated 1 March 2013

Shareholder Democracy

Cross-posted from System Viability and Corporate Governance blog

---

According to John Plender, the biggest curiosity of US corporate governance is that shareholders' votes on directors' appointments at annual meetings have no force. Shareholders can "withhold" their votes, but even if more than 50 per cent do so the gesture is purely symbolic and the company can continue on the management's chosen path. Plender comments: "This shareholder democracy is about as democratic as Cuba or the old Soviet Union."

[Financial Times, November 13th 2004]

Meanwhile, Manchester United management is outraged that Malcolm Glazier should have used his substantial minority shareholding this week to vote against the appointment of directors. Apparently this is bad form; complaints have been made to various regulators. Glazier's intentions towards the football club are currently unclear - but does this mean he should be denied a vote?

Many people seem unable to distinguish the viability of a company from the perpetuation of its current management. Shareholder democracy in Europe is far from being an adequate control mechanism - it lacks power as well as requisite variety. However, many US stakeholders are strongly resistant to even this weak control mechanism; meanwhile News International's corporate move to Delaware is widely seen as an escape from the stricter corporate governance environment in Australia.

One of the preconditions for corporate governance is transparency, and systems engineering can certainly do something about this. But transparency is not enough. We need to engineer appropriate control mechanisms, so that transparent information can be properly acted upon in the interests of the real stakeholders.

---

Click here for Trevor's reply 
http://systemviabilityandcorporategovernance.blogspot.co.uk/2004/11/shareholder-democracy_16.html

Commitment and Self-Interest

Cross-posted from System Viability and Corporate Governance blog

---

Standard investment advice is to diversify. Among other things, this means that it's not a good idea to hold significant quantities of shares in your own company, since you are doubly exposed if the company fails. (Think Enron.)

Meanwhile, it is widely supposed that the company's interests are served if the directors and employees have an investment stake in the company as well as an employment stake. This is supposed to align the personal interests of the directors and employees with the interests of the company. There is personal commitment to the success of the company, with an inflated cost of exit.

Furthermore, by having large personal shareholdings, company directors demonstrate their confidence in the company's present state and future prospects, and their belief that the share price undervalues the true worth of the company.

There is therefore a structural conflict of interest between company (external shareholders) and employees (especially directors). How is this resolved?

Either the individual directors take an irrational stance in respect of their personal investments, accepting an unbalanced portfolio with a sub-optimal risk/reward ratio. However, we should not expect true alignment between the interests of a director with an unbalanced investment portfolio, and the majority of shareholders whose investments are (of course) properly balanced and diversified.

Or the directors cheat. For example, holding derivatives that hedge against the excess exposure to the failure of the company. For example, manipulating information. And as a privileged class, the directors hedge against failure by awarding themselves massive termination payments. (While not illegal, this is a morally corrupt practice.)

According to this argument, directors are driven by the system towards either madness or badness. (Some manage both at once.) The answer is not to recruit a new cadre of morally upright and selfless leaders, but to change the system.

---

Click here for Trevor's reply  
http://systemviabilityandcorporategovernance.blogspot.co.uk/2004/08/commitment-and-self-interest.html