Alphabetic Warning

In an interview with the BBC, Sundar Pichai, CEO of Alphabet, advises people to be cautious of what AI tells them, suggesting that AI should be combined with other tools such as, er, Google search (owned by Alphabet). 

But if we shouldn't blindly trust what AI tells us, the same advice surely applies to Google search as well. The question Can we trust Google? was raised on this blog over twenty years ago, and remains relevant today. 

There are also important questions of context and perspective. People are increasingly aware that the results they get from generative AI is highly dependent on how the enquiry is worded, hence the new discipline of prompt engineering. But the same is also true of search engines, as I discussed with David McCoy back in 2008. In his comment below my blogpost, You don't have to be smart to search here ..., David recommended what he called speculative cunning. 

One advantage of AI over search is that it offers much better levels of context-awareness (depending on the prompts). As I noted previously, standardized search systems ... will give you exactly the same answer whether you are a schoolchild or a BBC researcher or an LSE postgraduate. Someone at Google previously told the BBC about their Quest for the Perfect Search. But perhaps a perfectly personalized answer may have the effect of seducing the user into an unwarranted level of trust. 

Mr Pichai acknowledges that there are ethical issues here, saying that Alphabet aims to be bold and responsible at the same time.

---

 

Faisal Islam and Rachel Clun, Don't blindly trust what AI tells you, says Google's Sundar Pichai (BBC, 18 November 2025)

Related posts Trusting Commons (March 2005), Context-Aware Services (January 2006), You don't have to be smart to search here ... (November 2008), The Force of Goole (April 2016)

 

All our eyes on the disgraceful Horizon

The scandal at the British Post Office, details of which are now emerging in the Public Enquiry, provides illustrations of many important aspects of organizational behaviour as discussed on this blog. 

Willful blindness. There is a strong attachment to a false theory, despite mounting evidence to the contrary, as well as the appalling human consequences.

Misplaced trust. Trusting a computer system (Horizon) above hundreds of ordinary people. And both the legal system and government ministers trusting the evidence presented by a public corporation, despite the fact that contrary evidence from expert witnesses had been accepted in a small number of cases (see below).

Defensive denial as one of the symptoms of organizational stupidity. In July 2013, Post Office boss Paula Vennells was told about faults in the Horizon system, and advised that denying these would be dangerous and stupid. This is something the Post Office had denied for years. ITV March 2024

A detail that struck me yesterday was a failure to connect the dots. In 2011, the auditors (EY) raised concerns about data quality, warning that if Horizon was not accurate, then they would not be able to sign off Post Office company accounts. Ms Perkins, who was giving evidence at an inquiry into the scandal, said at the time she did not make a link between the two. BBC June 2024. The pattern I'm seeing here is of assuming the sole purpose of audit as satisfying some regulatory requirement, with zero operational (let alone ethical) implications of anything the auditors might find. And assuming the regulatory requirement itself to have no real purpose, being merely a stupid and meaningless piece of bureaucracy.

Another failure to connect the dots occurred after Julie Wolstenholme successfully challenged the Post Office in 2003 with the aid of an expert technical witness. Why didn't this prompt serious questions about all the other cases? When asked about this at the enquiry, David Mills said he had not properly assimilated the information and pleaded lack of intelligence, saying I wasn’t that clever. I’m sorry, I didn’t ask about it. ITV April 2024

In my other pieces about organizational intelligence, I have noted that stupid organizations may sometimes be composed of highly intelligent people. Now that's one pattern the Post Office doesn't seem to illustrate. Or have the Post Office bosses merely chosen to present themselves as naive and incompetent rather than evil?

---

Tom Espiner, Ex-Post Office chair was told of IT risks in 2011 (BBC 5 June 2024)

ITV, Secret tape shows Paula Vennells was told about problems with Horizon and warned not to cover it up (29 March 2024)

ITV, Former Post Office boss tells inquiry he was not 'clever' enough to question Horizon IT system (16 April 2024)

Other Sources: Post Office Horizon IT Enquiry, British Post Office scandal (Wikipedia), Post Office Project (University of Exeter)

On the invisibility of infrastructure

Infrastructure is boring, expensive, and usually someone else's responsibility/problem. Which is perhaps how the UK finds itself at what Jeremy Fleming, head of GCHQ, describes as a moment of reckoning. Simon Wardley analyses this in terms of digital sovereignty.

Digital sovereignty is all about us (as a collective) deciding which parts of this competitive space that we want to own, compete, defend, dominate and represent our values and our behaviours in. It's all about where are our borders in this space. ... Our responses all seem to include a slide into protectionism with claims that we need to build our own cloud industries.

Fleming is particularly focused on "the growing challenge from China", expresses concern about UK potentially losing control of  "standards that shape our technology environment" which apparently "make sure that our liberal Western democratic views are baked into our technology". Whatever that means. Fleming's technological examples include digital currency and smart cities.

Fleming talks about the threats from Russia and China, and regards China's potential control of the underlying infrastructure as more fundamentally challenging than potential attacks from Russia as well as non-state actors.

Fleming notes the following characteristics of those he labels adversaries:

• Potential to control the global operating system.
• Early implementors of many of the emerging technologies that are changing the digital environment.
• Bringing all elements of [...] power to control, influence, design and dominate markets. Often with the effect of pushing out smaller players and reducing innovation. 
• Concerted campaigns to dominate international standards.
  

And continues

If [any of this] turns out to be insecure or broken or undemocratic, everyone is going to be facing a very difficult future.

It would be easy to hear these remarks as referring solely to China. But he also sounds a warning about corporate power, acknowledging that their commercial interests sometimes (!?) don't align with the interests of ordinary citizens. And with that in mind, it's easy to see how some of the adversarial characteristics listed above would apply equally to some of the Western tech giants.

If the goal is to bake Western values (whatever they are) into our technology infrastructure, it is not obvious that the Western tech giants can be trusted to do this. Smart City initiatives associated with Google's Sidewalk Labs have been cancelled in Portland and Toronto, following (although perhaps not entirely as a consequence of) democratic concerns about surveillance capitalism. However, Sidewalk Labs appears to be still active in a number of smaller smart city initiatives, as are Amazon Web Services, IBM and other major technology firms.

Fleming talks about standards, but at the same time he acknowledges that standards alone are too slow-changing and too weak to keep the adversaries at bay. "The nature of cyberspace makes the rules and standards more open to abuse." He talks about evolutionary change, using a version of Leon Megginson's formulation of natural selection: "it's those that are most able to adjust that prosper". (See my post on Arguments from Nature). But that very formulation seems to throw the initiative over to those tech firms that preach moving fast and breaking things. Can we therefore complain if our infrastructure is insecure, broken, and above all undemocratic?

For most of us, most of the time, infrastructure needs to be just there, taken for granted, ready to hand. Organizations providing these services are often established as monopolies, or turn into de facto monopolies, controlled not only (if at all) by market forces but by democratically accountable regulators and/or by technocratic specialists. However, the Western tech giants devote significant resources to lobbying against external regulation, resisting democratic control. And Smart City initiatives typically embed much the same values everywhere (civic paternalism, biopower).

So here is Fleming's dilemma. If you don't want China to make the running on smart cities, you have to forge alliances with other imperfectly trusted players, whose values are sometimes (!?) not aligned with yours. This moves away from the kind of positional strategy described in Wardley's maps, towards a more relational strategy.

 

---

Gordon Corera, GCHQ chief warns of tech 'moment of reckoning' (BBC News, 23 April 2021) via @sukhigill and @swardley

Jeremy Fleming, A world of possibilities: Leading the way in cyber and technology (Vincent Briscoe Lecture @ Imperial College, 23 April 2021) via YouTube.

Susan Leigh Star and Karen Ruhleder, Steps Toward an Ecology of Infrastructure: Design and Access for Large Information Spaces (Information Systems Research 7/1, March 1996)

Simon Wardley, Digital Sovereignty (22 October 2020)

Related posts: The Allure of the Smart City (April 2021)

Ethics of Transparency and Concealment

Last week I was in Berlin at the invitation of the IEEE to help develop standards for responsible technology (P7000). One of the working groups (P7001) is looking at transparency, especially in relation to autonomous and semi-autonomous systems. In this blogpost, I want to discuss some more general ideas about transparency.

In 1986 I wrote an article for Human Systems Management promoting the importance of visibility. There were two reasons I preferred this word. Firstly, "transparency" is a contronym - it has two opposite senses. When something is transparent, this either means you don't see it, you just see through it, or it means you can really see it. And secondly, transparency appears to be merely a property of an object, whereas visibility is about the relationship between the object and the viewer - visibility to whom?

(P7001 addresses this by defining transparency requirements in relation to different stakeholder groups.)

Although I wasn't aware of this when I wrote the original article, my concept of visibility shares something with Heidegger's concept of Unconcealment (Unverborgenheit). Heidegger's work seems a good starting point for thinking about the ethics of transparency.

Technology generally makes certain things available while concealing other things. (This is related to what Albert Borgmann, a student of Heidegger, calls the Device Paradigm.)

In our time, things are not even regarded as objects, because their only important quality has become their readiness for use. Today all things are being swept together into a vast network in which their only meaning lies in their being available to serve some end that will itself also be directed towards getting everything under control. Levitt

Goods that are available to us enrich our lives and, if they are technologically available, they do so without imposing burdens on us. Something is available in this sense if it has been rendered instantaneous, ubiquitous, safe, and easy. Borgmann

I referred above to the two opposite meanings of the word "transparent". For Heidegger and his followers, the word "transparent" often refers to tools that can be used without conscious thought, or what Heidegger called ready-to-hand (zuhanden). In technology ethics, on the other hand, the word "transparent" generally refers to something (product, process or organization) being open to scrutiny, and I shall stick to this meaning for the remainder of this blogpost.

We are surrounded by technology, we rarely have much idea how most of it works, and usually cannot be bothered to find out. Thus when technological devices are designed to conceal their inner workings, this is often exactly what the users want. How then can we object to concealment?

The ethical problems of concealment depend on what is concealed by whom and from whom, why it is concealed, and whether, when and how it can be unconcealed.

Let's start with the why. Sometimes people deliberately hide things from us, for dishonest or devious reasons. This category includes so-called defeat devices that are intended to cheat regulations. Less clear-cut is when people hide things to avoid the trouble of explaining or justifying them.

(If something is not visible, then we may not be aware that there is something that needs to be explained. So even if we want to maintain a distinction between transparency and explainability, the two concepts are interdependent.)

People may also hide things for aesthetic reasons. The Italian civil engineer Riccardo Morandi designed bridges with the steel cables concealed, which made them difficult to inspect and maintain. The Morandi Bridge in Genoa collapsed in August 2018, killing 43 people.

And sometimes things are just hidden, not as a deliberate act but because nobody has thought it necessary to make them visible. (This is one of the reasons why a standard could be useful.)

We also need to consider the who. For whose benefit are things being hidden? In particular, who is pulling the strings, where is the funding coming from, and where are the profits going - follow the money. In technology ethics, the key question is Whom Does The Technology Serve?

In many contexts, therefore, the main focus of unconcealment is not understanding exactly how something works but being aware of the things that people might be trying to hide from you, for whatever reason. This might include being selective about the available evidence, or presenting the most common or convenient examples and ignoring the outliers. It might also include failing to declare potential conflicts of interest.

For example, the #AllTrials campaign for clinical trial transparency demands that drug companies declare all clinical trials in advance, rather than waiting until the trials are complete and then deciding which ones to publish.

Now let's look at the possibility of unconcealment. Concealment doesn't always mean making inconvenient facts impossible to discover, but may mean making them so obscure and inaccessible that most people don't bother, or creating distractions that divert people's attention elsewhere. So transparency doesn't just entail possibility, it requires a reasonable level of accessibility.

Sometimes too much information can also serve to conceal the truth. Onora O'Neill talks about the "cult of transparency" that fails to produce real trust.

Transparency can produce a flood of unsorted information and misinformation that provides little but confusion unless it can be sorted and assessed. It may add to uncertainty rather than to trust. Transparency can even encourage people to be less honest, so increasing deception and reducing reasons for trust. O'Neill

Sometimes this can be inadvertent. However, as Chesterton pointed out in one of his stories, this can be a useful tactic for those who have something to hide.

Where would a wise man hide a leaf? In the forest. If there were no forest, he would make a forest. And if he wished to hide a dead leaf, he would make a dead forest. And if a man had to hide a dead body, he would make a field of dead bodies to hide it in. Chesterton

Stohl et al call this strategic opacity (via Ananny and Crawford).

Another philosopher who talks about the "cult of transparency" is Shannon Vallor. However, what she calls the "Technological Transparency Paradox" seems to be merely a form of asymmetry: we are open and transparent to the social media giants, but they are not open and transparent to us.

In the absence of transparency, we are forced to trust people and organizations - not only for their honesty but also their competence and diligence. Under certain conditions, we may trust independent regulators, certification agencies and other institutions to verify these attributes on our behalf, but this in turn depends on our confidence in their ability to detect malfeasance and enforce compliance, as well as believing them to be truly independent. (So how transparent are these institutions themselves?) And trusting products and services typically means trusting the organizations and supply chains that produce them, in addition to any inspection, certification and official monitoring that these products and services have undergone.

Instead of seeing transparency as a simple binary (either something is visible or it isn't), it makes sense to discuss degrees of transparency, depending on stakeholder and context. For example, regulators, certification bodies and accident investigators may need higher levels of transparency than regular users. And regular users may be allowed to choose whether to make things visible or invisible. (Thomas Wendt discusses how Heideggerian thinking affects UX design.)

Finally, it's worth noting that people don't only conceal things from others, they also conceal things from themselves, which leads us to the notion of self-transparency. In the personal world this can be seen as a form of authenticity; in the corporate world, it translates into ideas of responsibility, due diligence, and a constant effort to overcome wilful blindness.

If transparency and openness is promoted as a virtue, then people and organizations can make their virtue apparent by being transparent and open, and this may make us more inclined to trust them. We should perhaps be wary of organizations that demand or assume that we trust them, without providing good evidence of their trustworthiness. (The original confidence trickster asked strangers to trust him with their valuables.) The relationship between trust and trustworthiness is complicated. 

---

UK Department of Health and Social Care, Response to the House of Commons Science and Technology Committee report on research integrity: clinical trials transparency (UK Government Policy Paper, 22 February 2019) via AllTrials

Mike Ananny and Kate Crawford, Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability (new media and society 2016) pp 1–17

Albert Borgmann, Technology and the Character of Contemporary Life (University of Chicago Press, 1984)

G.K. Chesterton, The Sign of the Broken Sword (The Saturday Evening Post, 7 January 1911)

Martin Heidegger, The Question Concerning Technology (Harper 1977) translated and with an introduction by William Lovitt

Onora O'Neill, Trust is the first casualty of the cult of transparency (Telegraph, 24 April 2002)

Cynthia Stohl, Michael Stohl and P.M. Leonardi, Managing opacity: Information visibility and the paradox of transparency in the digital age (International Journal of Communication Systems 10, January 2016) pp 123–137.

Richard Veryard, The Role of Visibility in Systems (Human Systems Management 6, 1986) pp 167-175 (this version includes some further notes dated 1999)

Thomas Wendt, Designing for Transparency and the Myth of the Modern Interface (UX Magazine, 26 August 2013)

Stanford Encyclopedia of Philosophy: Heidegger, Technological Transparency Paradox

Wikipedia: Confidence Trick, Follow The Money, Ponte Morandi, Regulatory Capture,Willful Blindness


Related posts: Defeating the Device Paradigm (October 2015), Transparency of Algorithms (October 2016), Pax Technica (November 2017), Responsible Transparency (April 2019), Whom Does The Technology Serve (May 2019)

Algorithms and Auditability

@ruchowdh objects to an article by @etzioni and @tianhuil calling for algorithms to audit algorithms. The original article makes the following points.

• Automated auditing, at a massive scale, can systematically probe AI systems and uncover biases or other undesirable behavior patterns. 

• High-fidelity explanations of most AI decisions are not currently possible. The challenges of explainable AI are formidable.  

• Auditing is complementary to explanations. In fact, auditing can help to investigate and validate (or invalidate) AI explanations.

• Auditable AI is not a panacea. But auditable AI can increase transparency and combat bias. 

Rumman Chowdhury points out some of the potential imperfections of a system that relied on automated auditing, and does not like the idea that automated auditing might be an acceptable substitute for other forms of governance. Such a suggestion is not made explicitly in the article, and I haven't seen any evidence that this was the authors' intention. However, there is always a risk that people might latch onto a technical fix without understanding its limitations, and this risk is perhaps what underlies her critique.

In a recent paper, she calls for systems to be "taught to ignore data about race, gender, sexual orientation, and other characteristics that aren’t relevant to the decisions at hand". But how can people verify that systems are not only ignoring these data, but also being cautious about other data that may serve as proxies for race and class, as discussed by Cathy O'Neil? How can they prove that a system is systematically unfair without having some classification data of their own?

And yes, we know that all classification is problematic. But that doesn't mean being squeamish about classification, it just means being self-consciously critical about the tools you are using. Any given tool provides a particular lens or perspective, and it is important to remember that no tool can ever give you the whole picture. Donna Haraway calls this partial perspective.

With any tool, we need to be concerned about how the tool is used, by whom, and for whom. Chowdhury expects people to assume the tool will be in some sense "neutral", creating a "veneer of objectivity"; and she sees the tool as a way of centralizing power. Clearly there are some questions about the role of various stakeholders in promoting algorithmic fairness - the article mentions regulators as well as the ACLU - and there are some major concerns that the authors don't address in the article.

Chowdhury's final criticism is that the article "fails to acknowledge historical inequities, institutional injustice, and socially ingrained harm". If we see algorithmic bias as merely a technical problem, then this leads us to evaluate the technical merits of auditable AI, and acknowledge its potential use despite its clear limitations. And if we see algorithmic bias as an ethical problem, then we can look for various ways to "solve" and "eliminate" bias. @juliapowles calls this a "captivating diversion". But clearly that's not the whole story.

Some stakeholders (including the ACLU) may be concerned about historical and social injustice. Others (including the tech firms) are primarily interested in making the algorithms more accurate and powerful. So obviously it matters who controls the auditing tools. (Whom shall the tools serve?)

What algorithms and audits have in common is that they deliver opinions. A second opinion (possibly based on the auditing algorithm) may sometimes be useful - but only if it is reasonably independent of the first opinion, and doesn't entirely share the same assumptions or perspective. There are codes of ethics for human auditors, so we may want to ask whether automated auditing would be subject to some ethical code.

---

Paul R. Daugherty, H. James Wilson, and Rumman Chowdhury, Using Artificial Intelligence to Promote Diversity (Sloan Management Review, Winter 2019)

Oren Etzioni and Michael Li, High-Stakes AI Decisions Need to Be Automatically Audited (Wired, 18 July 2019)

Donna Haraway, Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. In Simians, Cyborgs and Women (Free Association, 1991)

Cathy O'Neil, Weapons of Math Destruction

Julia Powles, The Seductive Diversion of ‘Solving’ Bias in Artificial Intelligence (7 December 2018)

Related posts: Whom Does the Technology Serve? (May 2019), Algorithms and Governmentality (July 2019)

Responsible Transparency

It is difficult to see how we can achieve an ethical technology without some kind of transparency, although we are still trying to work out how this could be achieved in an effective yet responsible manner. There are several concerns that are thought to conflict with transparency, including commercial advantage, security, privacy, and the risk of the device being misused or "gamed" by adversaries. There is a good summary of these issues in Mittelstadt et al (2016).

An important area where demands for transparency conflict with demands for confidentiality is with embedded software that serves the interests of the manufacturer rather than the consumer or the public. For example, a few years ago we learned about a "defeat device" that VW had built in order to cheat the emissions regulations; similar devices have been discovered in televisions to falsify energy consumption ratings.

Even when the manufacturers aren't actually breaking the law, they have a strong commercial interest in concealing the purpose and design of these systems, and they use Digital Rights Management (DRM) and the US Digital Millenium Copyright Act (DMCA) to prevent independent scrutiny. In what appears to be an example of regulatory capture, car manufacturers were abetted by the US EPA, which was persuaded to inhibit transparency of engine software, on the grounds that this would enable drivers to cheat the emissions regulations.

Defending the EPA, David Golumbia sees a choice between two trust models, which he calls democratic and cyberlibertarian. For him, the democratic model "puts trust in bodies specifically chartered and licensed to enforce regulations and laws", such as the EPA, whereas in the cyberlibertarian model, it is the users themselves who get the transparency and can scrutinize how something works. In other words, trusting the wisdom of crowds, or what he patronizingly calls "ordinary citizen security researchers".

(In their book on Trust and Mistrust, John Smith and Aidan Ward describe four types of trust. Golumbia's democratic model involves top-down trust, based on the central authority of the regulator, while the cyberlibertarian model involves decentralized network trust.)

Golumbia argues that the cyberlibertarian position is incoherent. 

"It says, on the one hand, we should not trust manufacturers like Volkswagen to follow the law. We shouldn’t trust them because people, when they have self-interest at heart, will pursue that self-interest even when the rules tell them not to. But then it says we should trust an even larger group of people, among whom many are no less self-interested, and who have fewer formal accountability obligations, to follow the law."

One problem with this argument is that it appears to confuse scrutiny with compliance. Cyberlibertarians may be strongly in favour of deregulation, but increasing transparency isn't only advocated by cyberlibertarians and doesn't necessarily imply deregulation. It could be based on a recognition that regulatory scrutiny and citizen scrutiny are complementary, given two important facts. Firstly, however powerful the tools at their disposal the regulators don't always spot everything; and secondly, regulators are sometimes subject to improper influence from the companies they are supposed to be regulating (so-called regulatory capture). Therefore having independent scrutiny as well as central regulation increases the likelihood that hazards will be discovered and dealt with. This could include the detection of algorithmic bias or previously unidentified hazards/vulnerabilities/malpractice.

Another small problem with his argument is that the defeat device had already hoodwinked the EPA and other regulators for many years.

Golumbia claims that "what the cyberlibertarians want, even demand, is for everyone to have the power to read and modify the emissions software in their cars" and complains that "the more we put law into the hands of those not specifically entrusted to follow it, the more unethical behavior we will have". It is certainly true that some of the advocates of open source are also advocating "right to repair" and customization rights. But there were two separate requests for exemptions to DMCA - one for testing and one for modification. And the researchers quoted by Kyle Wiens, who were disadvantaged by the failure of the EPA to mandate a specific exemption to DMCA to allow safety and security tests, were not casual libertarians or "ordinary citizens" but researchers at the International Council of Clean Transportation and West Virginia University.

It ought to be possible for regulators and academic researchers to collaborate productively in scrutinizing an industry, provided that clear rules, protocols and working practices are established for responsible scrutiny. Perhaps researchers might gain some protection from regulatory action or litigation by notifying a regulator in advance, or by prompt notification of any discovered issues. For example, the UK Data Protection Act 2018 (section 172) defines what it calls "effectiveness testing conditions", under which researchers can legitimately attempt to crack the anonymity of deidentified personal data. Among other things, a successful attempt must be notified to the Information Commissioner within 72 hours.

Meanwhile, in the cybersecurity world there are fairly well-established protocols for responsible disclosure of vulnerabilities, and in some cases rewards are paid to the researchers who find them, provided they are disclosed responsibly. Although not all of us have the expertise to understand the technical detail, the existence of this kind of independent scrutiny should make us all feel more confident about the safety, reliability and general trustworthiness of the products in question.

---

David Golumbia, The Volkswagen Scandal: The DMCA Is Not the Problem and Open Source Is Not the Solution (6 October 2015)

Brent Mittelstadt et al, The ethics of algorithms: Mapping the debate (Big Data and Society July–December 2016)

Jonathan Trull, Responsible Disclosure: Cyber Security Ethics (CSO Cyber Security Pulse, 26 February 2015)

Aidan Ward and John Smith, Trust and Mistrust (Wiley 2003)

Kyle Wiens, Opinion: The EPA shot itself in the foot by opposing rules that could've exposed VW (The Verge, 25 September 2015)


Related posts: Four Types of Trust (July 2004), Defeating the Device Paradigm (October 2015)

Ethics in Technology - FinTech

Last Thursday, @ThinkRiseLDN (Rise London, a FinTech hub) hosted a discussion on Ethics in Technology (15 November 2018).

Since many of the technologies under discussion are designed to support the financial services industry, the core ethical debate is strongly correlated to the business ethics of the finance sector and is not solely a matter of technology ethics. But like most other sectors, the finance sector is being disrupted by the opportunities and challenges posed by technological innovation, and this entails a professional and moral responsibility on technologists to engage with a range of ethical issues.

(Clearly there are many ethical issues in the financial services industry besides technology. For example, my friends in the @LongFinance initiative are tackling the question of sustainability.)

The Financial Services industry has traditionally been highly regulated, although some FinTech innovations may be less well regulated for now. So people working in this sector may expect regulation - specifically principles-based regulation - to play a leading role in ethical governance (Note: the UK Financial Services Authority has been pursuing a principles-based regulation strategy for over ten years.)

Whether ethical questions can be reduced to a set of principles or rules is a moot point. In medical ethics, principles are generally held to be useful but not sufficient for resolving difficult ethical problems. (See McCormick for a good summary. See also my post on Practical Ethics.)

Nevertheless, there are undoubtedly some useful principles for technology ethics. For example, the principle that you can never foresee all the consequences of your actions, so you should avoid making irreversible technological decisions. In science fiction, this issue can be illustrated by a robot that goes rogue and cannot be switched off. @moniquebachner made the point that with a technology like Blockchain, you were permanently stuck, for good or ill, with your original design choices.

Several of the large tech companies have declared principles for data and intelligence. (My summary here.) But declaring principles is the easy bit; these companies taking them seriously (or us trusting them to take them seriously) may be harder.

One of the challenges discussed by the panel was how to negotiate the asymmetry of power. If your boss or your client wants to do something that you are uncomfortable with, you can't just assert some ethical principles and expect her to change her mind. So rather than walk away from an interesting technical challenge, you give yourself an additional organizational challenge - how to influence the project in the right way, without sacrificing your own position.

Obviously that's an ethical dilemma in its own right. Should you compromise your principles in the hope of retaining some influence over the outcome, or could you persuade yourself that the project isn't so bad after all? There is an interesting play-off between individual responsibility and collective responsibility, which we are also seeing in politics (Brexit passim).

Sheryl Sandberg appears to offer a high-profile example of this ethical dilemma. She had been praised by feminists for being "the one reforming corporate boy’s club culture from the inside ... the civilizing force barely keeping the organization from tipping into the abyss of greed and toxic masculinity." Crispin now disagrees with this view. "It seems clear what Sandberg truly is instead: a team player. And her team is not the working women of the world. It is the corporate culture that has groomed, rewarded, and protected her throughout her career." "This is the end of corporate feminism", comments @B_Ehrenreich.

And talking of Facebook ...

The title of Cathy O'Neil's book Weapons of Math Destruction invites a comparison between the powerful technological instruments now in the hands of big business, and the arsenal of nuclear and chemical weapons that have been a major concern of international relations since the Second World War. During the so-called Cold War, these weapons were largely controlled by the two major superpowers, and it was these superpowers that dominated the debate. As these weapons technologies have proliferated however, attention has shifted to the possible deployment of these weapons by smaller countries, and it seems that the world has become much more uncertain and dangerous.

In the domain of data ethics, it is the data superpowers (Facebook, Google) that command the most attention. But while there are undoubtedly major concerns about the way these companies use their powers, we may at least hope that a combination of forces may help to moderate the worst excesses. Besides regulatory action, these forces might include public opinion, corporate risk aversion from the large advertisers that provide the bulk of the income, as well as pressure from their own employees.

And in FinTech as with Data Protection, it will always be easier for regulators to deal with a small number of large players than with a very large number of small players. The large players will of course try to lobby for regulations that suit them, and may shift some operations into less strongly regulated jurisdictions, but in the end they will be forced to comply, more or less. Except that the ethically dubious stuff will always turn out to be led by a small company you've never heard of, and the large players will deny that they knew anything about it.

As I pointed out in my previous post on The Future of Political Campaigning, the regulators only have limited tools at their disposal, so this slants their powers to deal with the ethical ecosystem as a whole. If I had a hammer ...

---

Financial Services Authority, Principles-Based Regulation - Focusing on the Outcomes that Matter (FSA, April 2007)

Jessa Crispin, Feminists gave Sheryl Sandberg a free pass. Now they must call her out (Guardian, 17 November 2018)

Ian Harris, Commercial Ethics: Process or Outcome (Z/Yen, 2008)

Thomas R. McCormick, Principles of Bioethics (University of Washington, 2013)

Chris Yapp, Where does the buck stop now? (Long Finance, 28 October 2018)


Related posts Practical Ethics (June 2018) Data and Intelligence Principles from Major Players (June 2018) The Future of Political Campaigning (November 2018)

Convergence - Symbolic, Imaginary or Real?

@Christian_BB responded to my post Does Rigour Matter? with a comment "Rigour matters when building. It matters less when trying to get people converging." I replied, "Rigour matters when building consensus. Unless you just want people to have a warm feeling of convergence."

Roughly speaking, there are three modes of convergence and consensus.

1. Symbolic. We have a formal agreement, and maybe set up some formal structures that perpetuate this agreement, but there is enough loophole and exception and wriggle-room that we don't need take it seriously.

2. Imaginary. We have a warm impression that we are all in agreement about something, and a vague hope that all the details will sort themselves out somehow.

3. Real. We have a tough negotiation around the details, and acknowledge the practical trade-offs and compromises that are required to implement the agreement.


Christian may be perfectly correct that a warm feeling (imaginary convergence) may be a useful and motivating step towards real convergence. But I have seen the converse too many times - when a meeting or workshop evades or fudges the details of some plan, leaves the details to be sorted out later, and then fails to follow through. This is a common feature of Management-by-Powerpoint, as notoriously practised by the Pentagon before the US invasion of Iraq.

And where there is a lot of mutual hostility and mistrust, it is probably unrealistic to expect warm feelings to emerge until long after a real agreement has been forged and implemented.

So an imaginary agreement is neither necessary not sufficient as a precondition for a real agreement.

Dangling Conversation

@markhillary asks "When you follow company Twitter accounts, do you like being able to see who runs the account, like a named person on the profile?"

I think that depends how gullible you are. When I get a letter signed by an Important Person, I generally assume it was written by his staff and signed in his absence. And when I get a mass-produced "personal letter" from an Important Person, I assume it was generated by a computer and signed by a programmer.

I got an email recently, which claimed to be a "A Personal Message from Dr. Richard Soley, Chairman and CEO, OMG and Keith Steele, CEO, PrismTech and OMG Board Member". I wrote back and thanked Richard personally - not to the address on the email (which was omg_marketing@omg.org) but to his real email address. For some reason, he ignored this. I hope he's not ill or anything.

And corporate communications sometimes use a fictional identity. Gerald Kaufman MP once tried to phone a person in the Prime Minister's office who had responded to a letter, only to discover that "Mrs E Adams" didn't actually exist. [Source: John Walsh: Beware letters from fictional civil servants (Independent May 2011)] This kind of thing is convenient for bureaucracies, because it allows incoming communications to be sorted by topic and redirected to whoever happens to be on duty that day. I'm sure the same thing often happens with Twitter, to prevent a corporate spokesperson ever being confused with a private individual.

As for company bosses, politicians and other celebrities, it would be naive to imagine that they always write their own tweets. "Of course they don't", tweets @markhillary, "but allowing helpers to do broadcast stuff is surely OK if the conversational is genuine?" Well, that depends on your idea of a genuine conversation.

It seems to me that there are some serious sociological and ethical problems here - of public/private identity, authenticity and trust - and we are only just learning how to operate in this new world.

---

@markhillary goes on to ask another question. "If you were interacting with a brand like Virgin Media, are you happy conversing with the brand?"

My answer to that question invokes Freud's concept of transference. Our psychological state (happiness, frustration) may depend on what we project onto a given brand or persona that we are conversing with. I generally try to separate my feelings about the company/brand from my feelings about the human being who is standing between me and the company/brand - but I don't always succeed. When we are really angry about something, it is difficult to avoid being rude or sarcastic to the junior employee that picks up the phone, even when we know it's not really their fault. Conversely, if the sales assistant is charming enough, it is tempting to buy something we don't really need.

Of course the CEO never picks up the phone herself. Funny that. When I'm conversing with the Virgin brand, I may fondly imagine that I'm getting Richard Branson's personal attention, but there is a little voice inside my head saying that's unlikely.

There is of course one thing that is likely to make me very unhappy indeed. Suppose I am naive enough to imagine I am having a personal conversation with Richard Branson or Richard Soley. Then the screen falls over and I see it is just some little functionary and not the Wizard of Oz at all. Isn't that just going to annoy me? Isn't it Richard, isn't it? #OMG.

Two Dimensions of Trust

In my post Magic Quadrant or Sorting Hat, I compared Gartner's Magic Quadrant (used to classify software vendors and products) with the Hogwarts Sorting Hat (used to classify young witches and wizards).

• Leaders: Gryffindor
• Challengers: Slytherin
• Visionaries: Ravenclaw
• Niche Players: Hufflepuff

Gartner's Magic Quadrant is a 2x2 matrix, whose two dimensions are Vision and Ability-to-Execute.

Following my previous post on Sharing Trust, I was thinking about a contrast between two key Hogwarts characters - Hagrid and Snape - based on the two dimensions of Trustworthiness and Ability-to-Execute.

Hagrid is regarded as extremely trustworthy. In the very first chapter of the first Harry Potter book, Dumbledore says he would trust Hagrid with his life. Professor McGonagall agrees, but points out that Hagrid can be a little unreliable. Later in the book, he is tricked by Voldemort into revealing a key vulnerability in the security arrangements protecting the Philosopher's Stone - security experts would call this "social engineering". So he doesn't score so well on ability-to-execute.

Snape, on the other hand, is a very accomplished and creative wizard, who scores extremely high on ability-to-execute. As we progress through the series, it becomes clear that he is successfully deceiving either Dumbledore or Voldemort - or possibly both. But this of course raises serious questions about his trustworthiness.

Trustworthiness - but for whom? Dumbledore trusts both Hagrid and Snape absolutely; other characters trust them with reservations, and only because Dumbledore does. And J.K. Rowling is careful not to present Dumbledore as omniscient - he is hoodwinked on several occasions, most notably by a clever impersonation in the Goblet of Fire.

So there are two ways of trusting people. We can regard them as trustworthy but fallible. Like Hagrid, or for that matter Dumbledore himself. Or we can regard them as reliable but remain suspicious of their true motivation and allegiance. Like Snape, or for that matter Voldemort. Ultimately, this is a question of authenticity.

Sharing Trust

@CoCreatr (Bernd Nurnberger) via @VenessaMiemis blogs about #trust.

"Being in business is basically about trust. Establishing and verifying trust, documenting it, so it can be shared, swiftly, without every business partner having to redo what led to the trust."

What I am slightly wary about here is the implication that trust can be passed around, like a parcel. I often find myself questioning the related notion that knowledge (content) can be passed around like a parcel, and I am wondering whether the same fallacy can be found in each of the five dimensions of VPEC-T.

Bernd also repeats some trust-builders and trust-destroyers that appear to originate in A Survey of Trust in the Workplace (pdf), carried out by Paul Bernthal of DDI.

Trust building behaviours:

• Communicates with me openly and honestly, without distorting any information.
• Shows confidence in my abilities by treating me as a skilled, competent associate.
• Keeps promises and commitments.
• Listens to and values what I say, even though he or she might not agree.
• Cooperates with me and looks for ways in which we can help each other.

Trust reducing behaviours:

• Acts more concerned about his or her own welfare than anything else.
• Sends mixed messages so that I never know where he or she stands.
• Avoids taking responsibility for action (“passes the buck” or “drops the ball”).
• Jumps to conclusions without checking the facts first.
• Makes excuses or blames others when things don’t work out (“finger-pointing”).

A commentary on this survey on the Challenge Network Forum (presumably by Oliver Sparrow) observes that fear appears to be a common factor of the trust destroyers.

"When you look over the trust-destroyers, that list sounds like the actions of people who are scared - scared of what might happen to them if they make mistakes in a company where mistakes are punished, rather than regarded as the occasional result of encouraging employees to take some initiative."

Again, I am wondering whether the same pattern of xxx-building and xxx-reducing behaviours applies to the other dimensions of VPEC-T.

---

There is another set of popular theories about trust, involving certain social activities (such as team-building exercises) that are supposed to promote trust. A quick internet search for "trust-building" will yield a large number of these exercises, together with companies that will happily take your money for running these exercises with you and your colleagues. Alternatively, why not just drip oxytocin into the air-conditioning?

See also Two Dimensions of Trust

---

Paul Bernthal, A Survey of Trust in the Workplace (pdf) (DDI, 1998)

Randy Borum, The Science of Interpersonal Trust (Mitre, 2010). Also available via Scribd.

Bernd Nurnberger, Community of practice and trust building (Feb 2012) - reposted by Venessa Miemis, 5 Trust Builders and 5 Trust Destroyers (March 2012)

Oliver Sparrow (?), Whom do we trust? (Challenge Network Forum, undated)

VPEC-T and pluralism

#vpect is a deceptively simple systems thinking lens developed by a couple of friends of mine (Carl Bate and Nigel Green), and described in their book Lost in Translation. The letters stand for Values, Policies, Events, Content and Trust. For a good brief description, see VPEC-T the 5D lens by @RoyGrubb.

---

VPEC-T is based on a profoundly radical philosophy of plurality. Instead of a single centralized value system (as found in top-down command-and-control organizations), we expect to find a range of different (overlapping, conflicting) value systems. Instead of a single coherent set of policies, we expect to find complex interaction between different kinds of policies (commercial, security, safety, corporate responsibility, and so on). Instead of a simple set of routine events, the post-modern organization is faced with a dynamic set of emerging events. Instead of a rigid set of database records, systems content is rich and evolving. And finally, the whole human activity system is underpinned by a complex set of trust relationships between people and organizations.

If you’ve spent any time working and thinking in business and systems, these statements will probably seem like common sense. However, many twentieth century systems methodologies were based on simplistic assumptions – for example, that there should be a single ordered set of goals and objectives, completely and consistently driving all system activity; that there should be a single ordered set of policies, rationally decided by management according to best-practice strategic principles, and so on.

And in our working lives and elsewhere we are surrounded by systems that have been designed according to these simplifying assumptions – systems that are inflexible, that serve a narrow set of stakeholders and to hell with everyone else. Not just software systems (for example, badly designed websites) but all sorts of human activity systems. Think of the hours of passenger time that are wasted because airline and airport procedures are designed for the convenience of their own operations rather than for the passengers. Think of the huge inefficiency of healthcare and other public services. Think of organizations whose attention is almost exclusively focused on internal squabbling rather than building external relationships, and which only survive because their competitors are no better. Think of counter-productive targets and expensive white elephants and all the other trappings of bureaucracy. These are the products of a certain way of thinking about business organizations and systems, which VPEC-T regards as obsolete and obscene.

So VPEC-T starts from the opposite assumption and expects trouble – there will be tension, there will be contradictions, and so on. Some people might think this is a pessimistic position. After all, wouldn’t it be wonderful if the world was as consistent and orderly and predictable as the 20th century methodologies assumed?

Actually I don’t think so – I think that kind of mechanical uniformity takes us towards the utopian nightmare described by countless science fiction writers. The plurality and richness which I find in VPEC-T is not only more realistic but also more human.

VPEC-T is not just a tool for making sense of complex human activity systems, but also a tool for communicating and telling stories about these systems. This is important for three reasons

• Firstly, because any single view of a complex system is necessarily unreliable and incomplete. We need multiple lenses and multiple voices to give us greater coverage of the issues, and greater confidence in the analysis.

• Secondly, because the philosophy of plurality makes it necessary for us to work collaboratively with other people, not only with different perspectives but with different value systems. So we need a language for collaboration.

• And thirdly, because of the reflexive nature of trust, which requires transparency of the models and assumptions on which the system is based. Ultimately, VPEC-T cannot be a private matter for a closed elite group of system designers without compromising trust principles.

---

See also POSIWID should be plural

Ethics and Intelligence

@flowchainsensei (Bob Marshall) argues that All Executives are Unethical (pdf).

More precisely, he argues that it is unethical to believe things without proper evidence. (He is particularly interested in beliefs about product and software development, but the argument applies more generally.)

As far as I can see, there are three steps in this argument.

1. People are ethically responsible for their beliefs. (According to Bob, this was the basis for a controversial paper presented to the Metaphysical Society by William Kingdon Clifford in 1876.)

2. An unfounded belief is unethical.

3. A person who holds unfounded beliefs is unethical.


Let's look at step 1 first. This appears to entail an ethical obligation to subject one's beliefs to some kind of "due diligence". However, most of our beliefs are based, not on evidence that we have personally collected and analysed, but at least partly on evidence that has been filtered through other sources. We may have reasons to trust certain sources more than others, but if it is unethical to believe things without proper evidence, it would also surely be unethical to trust things without proper evidence. We may accept an ethical obligation to subject our beliefs to "due diligence", but this is normally a collective obligation rather than an individual obligation.

Step 2 asserts that any failure to ground beliefs in proper evidence is an ethical failure. People are rightly held accountable for failing to act in certain circumstances (for example failing to save someone from drowning), but ethical censure generally assumes both awareness (knowing that someone needed rescue) and capability (being able to swim). So the problem with Step 2 is that the more complex the beliefs are, the greater the intellectual power (intelligence) that is required to appreciate and thoroughly investigate these beliefs. If the management team isn't individually or collectively intelligent enough to understand what proper evidence would look like, then believing things without proper evidence is a consequence of insufficient intelligence.

Does being stupid count as an ethical failure? (Being deliberately or avoidably stupid might, but most instances of stupidity are not deliberate.) Appointing people and teams who don't have enough intelligence might be unethical, but only if the appointment was deliberate or avoidable, and so on along the responsibility chain until we can find someone who should have known better.

Step 3 assumes that we can categorize people as ethical or unethical based on incidence of ethical or unethical behaviour. Once we have a hard-and-fast concept of sin, then we can define a sinner as a person who has committed (and not yet purged) at least one sin. The trouble with this is that if we are all sinners, the category of "sinner" ceases to have much value except for the purposes of hellfire rhetoric. Labelling all executives as unethical (and why stop at executives, by the way) becomes merely a rhetorical gesture.

---

So where does this leave the virtues of diligence, responsibility and probity? Firstly, I hold that these are collective virtues - executives display moral character in a particular organizational setting, and we may not know how their ethics would stand up in a different setting.

Secondly, I think character and intelligence are distinct virtues. We should not automatically suppose that intelligent people are more ethical than less intelligent people, and therefore we should not define "ethical" to mean something that only very intelligent or highly educated people can comply with.

Thirdly, there is a widespread belief (especially among consultants) in the value of knowledge (although I don't know exactly what would count as proper evidence for this belief - if executives are unethical, I dread to think where this leaves consultants). If we define knowledge as justified true belief, then knowledge is degraded to the extent that it is unjustified or untrue, or for that matter disbelieved. If it is unethical to believe something without proper evidence, it may sometimes also be unethical to disbelieve something. Sometimes excessive scepticism shades into cynicism and negativity, and maybe this can be just as unethical as unjustified optimism.

 

---

Related posts: Intelligence and Governance (February 2013), Ethics and Uncertainty (March 2019)

Social Proximity and Trust

Andrew McAfee describes a topography of social networking, which he describes as the Enterprise 2.0 Bullseye (November 2007), drawing on the sociologist Mark Granovetter’s theory of the 'strength of weak ties' (pdf). McAfee's topography, which he acknowledges to be a drastic simplification of a large and complex set of phenomena, is drawn as concentric circles with the strongest ties in the centre (presumably this is the "bullseye"). For a discussion of McAfee's topography, see my post From Weak Ties to Weak Signals.

The circles of social proximity may relate to circles of trust. We preferentially share information and ideas with people we trust.

• because we expect to be recognized and rewarded - either by benefiting directly from the use of our knowledge (e.g. as a member of the same team or organization) or by getting some favour in return
  
• because we expect to be informed about the use of our knowledge, allowing us to intervene if the knowledge is used inappropriately or out of context (this is particularly important if the knowledge is complex, uncertain or volatile, or we are unsure about the ability of our friend or colleague to appreciate its full implications)
• or conversely, because we expect them to get on with it without pestering us with follow-up questions

And we seek information and ideas from people we trust

• because we believe that the information and ideas will be good
• because we expect to be trusted with their information and ideas
  
• because we expect to be allowed to use the information and ideas without excessive constraint or tedious negotiation
  

There are several alternative reasons for trusting people. One reason for trusting people is because we know them personally. Another reason is that they work for the same organization - therefore there is some management hierarchy that can resolve any competing claims or other issues. A third reason is that they have some kind of public reputation to maintain. We tend to trust public figures either because we have a fancy of knowing them personally, or because we imagine they have little to gain and much to lose from tricking their fans.

What are the relative strengths of these reasons for trusting people? In some organizations, people trust outsiders (such as consultants) more than they trust their own colleagues - either because they believe that the consultants have access to superior knowledge and techniques, or because they believe that the consultants are disinterested observers of internal company politics rather than active players.

Early Warning Signals

As @bmichelson reports, rockstar Dave Lee Roth used to demand a bowl of M&Ms in his dressing room, with all the brown ones removed. Why? Because he wanted to test the venue's attention to detail. If they couldn't even get the sweets right, how could he trust them to correctly install the complex wiring and lighting required for the gig? If the test failed, what that triggered wasn't a childish tantrum but righteous anger and a thorough test of everything else before the band would go on the stage. [Business Advice from Van Halen, Fast Company March 2010 via Elemental Links] As @bmichelson points out, this can be regarded as a form of instrumentation.

I have seen a similar trick described somewhere else, possibly in one of Mark McCormack's books. When discussing a major event with a large hotel, casually ask for orange juice for all participants, freshly squeezed on the premises that morning. If hotel management appears to regard this requirement as trivial, then this is a warning that it may not take the other requirements seriously either.



Obviously these tricks only work when they are secret. As soon as people realise that these tricks are being used as tests or predictors of performance or quality, then they will alter (distort) their behaviour accordingly. (I'm imagining a scene from a Spinal Tap remake in which a rival rock band sends its roadies to tip off the venue and disrupt proceedings, so that the M&Ms are perfect, but the wiring electrocutes the guitarist and wreaks the concert.)

So if you are going to use these kinds of trick as a shortcut to detect what's going on, you need four things.

1. to design a test that is a good indicator of incompetence, laziness or inattention
2. to behave congruently as if the test really mattered
3. to act appropriately when the test fails (e.g. test the wiring, look for an alternative venue)
4. to watch out for signs that the test has been rumbled

And if you suspect that these kinds of tricks are being used against you, to test your competence and attention to detail, then you are probably already paying enough attention to detail to be able to do an all-round competent job. Maybe next time they'll trust you a bit more, okay?

Ice Nine

by Richard and Aidan

---

Earlier this week, Rachel was on her way to New Zealand via Heathrow. Here's how the interaction of several systems failed her.

1. Thanks to the latest security scare, it now takes two and a half hours to search all the handbaggage and get all the passengers onto the plane.

2. By which time the plane has frozen, and needs de-icing again. That takes another hour.

3. By which time the pilot and co-pilot have already spent so much time sitting on the plane that they no longer have enough flying hours remaining in this shift to take the plane to its destination. So the flight is cancelled.

4. The passengers are asked to return to the baggage hall, collect their checked-in baggage and start the process all over again. But there are many other flights that have been cancelled for similar reasons, and the baggage hall is already full-to-bursting with unloaded bags and frustrated passengers, so Rachel has to wait several hours before her unloaded bags appear on the carousel.

5. Then she has to queue to get onto the next available flight, and the process starts all over again.

By a happy fluke, the next plane Rachel boarded actually managed to take off, and she was on her way to New Zealand, but not before a last-minute search to find enough qualifying aircrew ...

---

Why does this kind of mess occur? Anyone can look at the whole system and see what could have been done differently. But each system is operated by a different organization, and there is a lack of trust and overall systems leadership.

As readers of Kurt Vonnegut's novel Cat's Cradle will recognize, Ice Nine was the name of a fictional crystal that was capable of bringing the whole world to a complete stand-still. Quite an apt metaphor for failed systems then.

Social Media and Political Action

Politicians are fascinated by social media, for several reasons. They are impressed by the apparent contribution of social media to the electoral success of Barack Obama (coordinating local initiative as well as fund-raising), and they are also attracted by the possibility of communicating directly with their supporters.

So I was interested to see the latest Conservative Party experiment in crowdsourcing. The Party has obtained a leaked copy of a government report on public sector IT, and has published it on a website called Make IT Better, with the following statement.

"We want to throw open the process and allow people to contribute their ideas on how policy should be designed. In the post-bureaucratic age, we believe that crowdsourcing and collaborative design can help us to make better policies."

As it happens, I do have some reactions to the leaked government report, which I may cover elsewhere, but what I want to cover here is not what the next government's IT policy should be, but the much more fundamental question "how policy should be designed".

The production of policy is an inescapably political process. In several recent cases, we can see how ministers attempt to steer an uncomfortable path between public opinion on the one hand and expert advice on the other.

• Bank bonuses. Banking experts insist that large bonuses are required to keep the banks operating effectively, but popular opinion is largely hostile to this proposition. 
  

• Drug classification. Scientists argue that drug classification should be based on the evidence of harm caused by each drug, but politicians fear that this would be politically dangerous and would "send the wrong message". 
  

• ID cards, databases and so on. At first, public opinion largely supported such schemes, in the belief that these security mechanisms would provide reliable protection against a range of social ills including illegal immigration and terrorism. However, several highly respected security experts have pointed out the flaws in the government schemes, and have indicated a strong likelihood that the costs will be far higher than the official estimates. Public opinion now seems to be shifting against these schemes.

It would be crazy to say either that public opinion should always trump expert opinion, or that expert opinion should always trump public opinion. And of course "public opinion" and "expert opinion" are not two separate worlds, but there are strong links between them. Thus opinion is rarely simple and consistent, but may contain vigorous disagreement. However, that cannot be an excuse to ignore opinion. Politicians cannot and must not abdicate from this arena.

What's the relevance of social media to this process?  There are two important points here.

Firstly, social media provide platforms for self-appointed experts of all kinds to share and attempt to mobilize their opinions. Sometimes these opinions can strike a chord with a broader audience, and feeds into a movement that subverts the established policy - whether by fostering popular suspicion about scientific issues (such as GM crops and mass vaccination), or by mobilizing local opposition to some central funding decision (such as closing a well-loved hospital).


Secondly, governments have traditionally received expert advice from a relatively small elite of professional scientists and businessmen. This has the result of pushing policy in certain directions, often to suit the vested interests of powerful lobbies. But these vested interests are increasingly hard to conceal from the public gaze (thanks in part to social media - think Trafigura), and public opinion can sometimes be roused against these vested interests (as we have seen in the case of bank bonuses). So some kind of crowdsourcing might conceivably offer alternative sources of advice.

There is an important trust issue here as well. Governments are not trusted to spend large amounts of money on IT; anyone who reads the IT press (Computer Weekly, The Register) will be able to quote lots of reasons for this lack of trust. This isn't just an IT issue of course: as Stephanie Flanders, the BBC's economics editor, puts it, "We all believe the savings are there to be had. We just don't trust the government to find them." (Efficiency Trap, 7 December 2009)

Crowdsourcing perhaps offers the possibility of forging a different kind of trust. So the challenge is not to find a better way of generating input for a traditional strategy report, but to find a better way of doing strategy. Politicians may wish to regard certain areas of policy as being purely technological (and not political at all, hem hem), and therefore be willing to delegate these policy areas to "friendly" technocrats, but this is essentially a Faustian pact in which the technocrats (generally senior representatives of the major IT firms) promise to solve all the technical problems in return for a shed-load of cash. Some politicians may have gone along with this kind of deal in the past, but there is an increasingly blatent history of project failure and cost over-runs. (Today it was announced that the NHS IT System is being "scaled back" [BBC News, 6 December 2009].) So there is a major strategic risk here that can no longer be swept under the carpet.

The political challenge for politicians in these situations is to forge a constituency that will support productive action. Not a small club of powerful players, but a broad range of stakeholders with varying levels of power, proximity and interest - and also a wide range of social ties to the people who will vote in the next election. That's the lesson of social media that politicians should learn from Barack Obama: use of the Internet not as a one-sided fund-raising mechanism but as a way to build a new kind of constituency.

And that's where I think the Conservative experiment in crowdsourcing should go - not just collecting negative comments from which to score debating points against the Government, but developing an entirely new way of producing policy out of a genuine conversation with well-informed public opinion. Not easy by any means, but (given the present situation) it has to be worth trying.

A Job Description for Systems Thinking

Michael Zang asked how to distill the essence of Systems Thinking into a job description?

Michael's question starts with the challenge of "selling" the idea of a Systems Director to an organization with little experience in this area. The job description therefore contributes to at least four different objectives.

1. To create vision and confidence that there is a job worth doing here (in other words selling).
2. To help select a suitable candidate for the job, without unduly narrowing the field.
3. To help determine a reasonable remuneration for the job.
4. To provide guidance and support to the job-holder, without unduly constraining initiative and innovation.

One of the challenges of a "job description" for systems thinking is that the traditional job description represents a fixed decomposition of responsibilities within the organization. The job-holder is required to carry out such-and-such specified activities, and produce such-and-such specified outcomes. This kind of job description comes out of a reductionist view of the organization. (This remains true even if the description is analysed in terms of systems-friendly "behavioural competences". Absolutely nothing wrong with a bit of reductionism, of course, as long as you don't imagine it's the whole story.)

Whereas a System Director will be working on the whole enterprise-as-a-system and the outcomes may be hard to define in advance. Maybe that's why there aren't many of them.

If you are going to have a System Director, that person will be a leader of systems thinking across the organization, not just going into a darkened room to "do systems thinking" with a small bunch of like-minded chums. In fact, you may follow her around the office and not see any activity that corresponds to a text-book description of what systems thinking is supposed to look like, but things just start to shift in interesting and positive ways.

So one way to explore the role of a System Director would be using the VPEC-T systems thinking framework. The "Content" of the job is presumably about system thinking and transformation, but the other elements (especially "Values" and "Trust") are perhaps more about Leadership.

A typical way of running an organization is that there is collective leadership exercised by all the directors together (notwithstanding the obvious fact that some directors will have more power than others) and in addition each director provides leadership in one specialist area. The role of Systems Director implies that one director is a specialist in systems thinking and systems practice, and brings this specialism (the Content in VPEC-T terms) to the general role of leadership.

Simply by opening up a discussion about the nature of leadership roles within an organization, and using a system-thinking lens like VPEC-T to provide a light structure to the discussion, could be a really good way of edging the organization into new ways of tackling complex problems.

The question of Trust is clearly a major issue for any organization. The System Director will draw a decent salary (presumably commensurate with her status in the organization), consume other resources, demand time and attention from her peers, push people out of their comfort zones, and so on, all for the sake of some uncertain and unquantifiable benefits to the organization. There is a much greater commitment here than employing an external consultant, so a considerable degree of trust is required.

But "selling" is not just getting an organization to accept the idea of "Systems Director". What's more important is for the organization to be able to trust the person occupying this role.

And in many unreflective organizations, people are trusted if and only if they fit the organization's stereotype of what people should be like. And yet someone who fits this stereotype may be unable to perform the role. So there is a critical tension to be confronted here.

For me then, what's most interesting about the job description is not the contents of the finished document (competences, outcomes, and so on) but the process of negotiating it - so that it provides a focus for critical discussions between stakeholders and their advisers that will help set appropriate expectations about the role, and start to build the trust that will be required.

---

Afterword

When this question was put to the Linked-In Systems Thinking group, some of the discussants went to some slightly unproductive places, perhaps responding in advance to positions they imagined others might take. For example, arguing which of the many available schools of systems thinking should be written into the job description. (My own opinion is that it would be better to keep the job description as neutral as possible rather than writing it in the language of any one school in particular.) There was also some (in my view wholly unnecessary) deprecation of people who don't share The Vision, dismissing them as left-brained Cartesians, with a special dig at accountants. One of the enemies of the systems approach (as identified by Churchman) is politics. Even in a Systems Thinking discussion group (which a naive person might imagine would know better) we can see how easily how schism emerges and the debate gets unnecessarily politicized.

Public Confidence in Healthcare

BBC medical correspondent Fergus Walsh advises How not to run an immunisation campaign.

It seems that there has been a failure of planning by the German government. As a result of different government departments making their purchases without coordinating - an innocent planning error, according to a government spokesman - two different vaccines are being distributed. Government officials and the armed forces are getting Celvapan, while the general public will receive Pandemrix.

People have a tendency to read deeper meaning into such differences, especially in the absence of trust. (Indeed, even random differences can cause suspicion, which in turn can undermine trust.) In this case, the difference in vaccines has led to complaints of a two-tier health service. Some politicians (including Angela Merkel) have announced their intention to have Pandemrix instead of Celvapan, just like they were ordinary German citizens.

Fergus sums up as follows:

'The row in Germany is a reminder of how easy it is to undermine public confidence in healthcare. Ministers and officials need to ask themselves, "what might people think?" with any decision they make.'

This example reinforces my belief in the importance of paying attention to Meaning.

Loyalty and Trust 2

@peterbregman has just posted a recommendation to Diversify Yourself (via @RSessions) prompted by the same topic as was my recent post on Loyalty and Trust, namely the alarming level of suicides at France Telecom.

Peter points out the danger of investing all your identity in your job. When a woman kills herself because she can't take the new reorganization, that sounds like a dreadful lack of work-life balance.

There is an apparent paradox here. In some organization cultures, loyalty and trust is understood to entail total commitment to the firm, which takes precedence over all other concerns. There is a sexist view that women cannot be expected to devote everything to the firm - not just the belief that women are more likely than men to take time off work at short notice because a child is sick, but the belief that women may be less willing to take the firm's agenda seriously. In such cultures, some women may feel the need to prove themselves all the more, either as individuals ("I'm not like other women") or collectively ("we women are not as men imagine"), and attempt to outdo the men in the loyalty and trust stakes.

And yet this kind of loyalty and trust is a brittle one. There is nothing loyal or trustworthy about jumping out of a window, or about stabbing oneself in the middle of a meeting. Authentic trust depends on a stable character, which depends on proper social and psychological foundations. 

When meeting new people at work, we often like to share some personal information, just enough to show that there is some life outside work. Part of the motivation for this sharing is because a more rounded and multi-faceted picture of a person generally makes it easier for us to trust them. Thus diversification of your identity makes you (among other things) more trustworthy.