Jaywalking

Until the arrival of the motor car, the street belonged to humans and horses. The motor car was regarded as an interloper, and was generally blamed for collisions with pedestrians. Cities introduced speed limits and other safety measures to protect pedestrians from the motor car.

The motor industry fought back. Their goal was to shift the blame for collisions onto the foolish or foolhardy pedestrian, who had crossed the road in the wrong place at the wrong time, or showed insufficient respect to our new four-wheeled masters. A new crime was invented, known as jaywalking, and newspapers were encouraged to describe road accidents in these terms.

In March 2018, a middle-aged woman was killed by a self-driving car. This is thought to be the first recorded death by a fully autonomous vehicle. According to the US National Safety Transportation Board (NTSB), the vehicle failed to recognise her as a pedestrian because she was not at an obvious designated crossing. In other words, she was jaywalking.

As I've observed before, ethics professors like to introduce the Trolley Problem into the ethics of self-driving cars, often carrying out opinion surveys (whom shall the vehicle kill?) because these are easily published in peer-reviewed journals. A recent study at MIT found that many people thought law-abiding pedestrians had more right to safety than jaywalkers. Therefore, if faced with this unlikely choice, the car should kill the jaywalker and spare the others. You have been warned.

 

---

Update Nov 2020

According to the New York Times, the motor industry is concerned that if pedestrians know they’ll never be run over, jaywalking could explode, grinding traffic to a halt. One solution, suggested by an automotive industry official, is gates at each corner, which would periodically open to allow pedestrians to cross. To compensate for the failures of technology, comments Greg Shill. In a later tweet he notes

Auto companies spent a century establishing a reputation for prioritizing the sale of their products—in other words, profits—over safety and other goals, and automated vehicles stand to continue that trend. Trust is earned.

(second screenshot: https://t.co/bz5tbevs7N) pic.twitter.com/KQDx0gVKo0

— Greg Shill (@greg_shill) November 9, 2020

---

Jack Denton, Is the Trolley Problem Derailing the Ethics of Self-Driving Cars? (Pacific Standard 29 November 2018)

Aidan Lewis, Jaywalking: How the car industry outlawed crossing the road (BBC News, 12 February 2014)

Aarian Marshall and Alex Davies, Uber’s Self-Driving Car Didn’t Know Pedestrians Could Jaywalk (Wired, 11 November 2020) HT @rickspence

Peter Norton, Street Rivals: Jaywalking and the Invention of the Motor Age Street (Technology and Culture, Vol 48, April 2007)

Katyanna Quach, Remember the Uber self-driving car that killed a woman crossing the street? The AI had no clue about jaywalkers (The Register, 6 November 2019)

Joseph Stromberg, The forgotten history of how automakers invented the crime of "jaywalking" (Vox, 4 November 2015)

Eric A Taub, How Jaywalking Could Jam Up the Era of Self-Driving Cars (New York Times, 1 August 2019) HT @greg_shill


Related posts: Whom Does The Technology Serve? (May 2019), The Game of Wits between Technologists and Ethics Professors (June 2019)

 

Updated 13 November 2020

Developing cultures of high-quality care

#kfleadership Excellent lecture at @TheKingsFund this evening by Professor Michael West. Here are some of my notes.

---

When he left college West was short of money, so he took a job in the coal mines. Productivity was important to everyone, and the pay at the end of the week depended on the quantity of coal extracted. But there was one thing more important than productivity, namely safety.

In many organizations this would just be lip service. But in the coal mines, safety was taken very seriously, and management actions were completely congruent with this.

West argued that the same should apply in the Health Service. Of course productivity is fundamentally important, but the number one priority should not be productivity but high-quality and safe patient care.

Valuing patients and staff turns out to be good management. West's argument is not merely based on rhetoric, but is supported by data. Patient outcomes and patient satisfaction are highly correlated with staff satisfaction and morale, and these in turn are correlated with staff engagement, which West defined in terms of three things: pride, intrinsic engagement and involvement in decisions. Ultimately this links back to improved productivity.

Someone in the audience objected that productivity must always be the top priority, otherwise you risk running out of money to pay for patient care. West replied that productivity follows from good people management. He agreed that the NHS has a great deal to learn from the private sector, and expressed a hope that private sector expertise (including non-executive board members) would not be limited to the Marketing and Finance perspectives.


West affirmed that the NHS is full of intelligent and highly motivated people, and said that the traditional command and control mode of leadership was such a waste of resource. The key role of leaders is to learn from staff, and to realize the potential of the people.

People at all levels require courage to accept challenging targets - in other words, to strive for things that they won't always achieve. The organization must accept and learn from failure to reach these targets. Blaming people for failure to excel is not only stupid and unfair, it is also counter-productive, because it makes people risk-averse and inhibits them from striving for anything that isn't guaranteed in advance.

Leadership includes the courage to seek unwelcome information - for example feedback that indicates things not going well.

After the lecture, I was chatting to a group from a London teaching hospital about accountability. As I see it, accountability doesn't only mean taking responsibility for the consequences of one's decisions (such as short-sighted cost-cutting) but also taking responsibility for what one chooses to pay attention to. One of the classic examples in Moral Philosophy concerns a ship owner who sends a ship to sea without bothering to check whether the ship was sea-worthy. Some argue that the ship owner cannot be held responsible for the deaths of the sailors, because he didn't actually know that the ship would sink. I think most people would see the ship owner having a moral duty of diligence, and would regard him as accountable for neglecting this duty.

In the current climate, the NHS leadership has a duty to achieve high quality patient care and productivity, and the evidence from Professor West is that this can best be achieved by engaging staff at all levels. Executive boards must surely be held accountable if they neglect to do this.

---

See also Culture of Fear (Storify, 27 Feb 2013)

The ship-owner example can be found in an essay called "The Ethics of Belief" (1877) by W.K. Clifford, in which he states that "it is wrong always, everywhere, and for anyone, to believe anything upon insufficient evidence".


Related post: Ethics and Intelligence (April 2010), Agility and Fear (February 2013)

Updated 28 Feb 2013

Quarantine

Question

Banks are highly aware of some types of threat, but seem to ignore other types of threat. How can you have a secure system in which one party is systematically blind to a particular class of threat. You would have to hold them in some sort of quarantine.

Answer

How can you have a secure system that only works if all the parties are completely free of conceptual limitations?

I think my children are systematically blind to certain things. (No doubt they think I'm systematically blind to certain things.) This means I trust them in certain contexts/situations and not in others.

A guard dog can provide some degree of security, can be involved in a secure system. That remains true despite the fact that dogs are unable to recognize certain classes of threat, and you certainly wouldn't delegate the design of the whole system to the guard dog. Why can't we say the same about a bank?

You put a dog into quarantine because you think it might have rabies, not because you think dogs are stupid. Banks aren't stupid either.

Quarantine may be a useful architectural pattern in certain situations. It protects against delayed attacks - such as a disease with an fixed incubation period, or a software virus. An entity remains in quarantine until it can be properly scanned and disinfected, or until the disease emerges and runs its course, or until the incubation period expires. (For example, a software artefact might be presumed free of a Friday 13th software virus if nothing detectable happens on Friday 13th.)

However, guard dogs need to be contained - for their own safety as well as the safety of others. They must be protected against specific attacks - the burglar who tries to feed them with drugged meat, or to confuse them with extreme smells. When dogs bark their heads off, these reactions need to be properly interpreted. And when a dog doesn't bark in the night, this may provide an important clue to what happened (Sherlock Holmes)

Similarly, banks might need to be contained, and their activity (and inactivity) interpreted. (But I don't think this counts as quarantine.) But whether this is necessary (or even meaningful) depends on the architecture of the whole collaborative system.

See also Quarantine 2 (October 2005)

Is Regulation a Drag?

Is regulation a drag on innovation?

Here's an example to explore. Following the ChoicePoint scandal, there have been some proposals in the USA to tighten privacy.

Adam Shostack writes: "A number of smart people (for example, Jim Harper writing on Politech) critique the drag on innovation that such a regime entails. I'm very sympathetic to this critique."

Innovation often generates risk, not just for the innovators but for other parties within the ecosystem. Some campaigners oppose some kinds of innovation altogether, on the grounds that the risks are unbounded and unacceptable. This argument has been used against nuclear energy and GM food.

But if we think a particular innovation should go ahead, there is still a question of risk management and governance. There is no reason why unnecessary and foreseeable risks should be taken, and it is surely appropriate to ensure that innocent bystanders are not unduly put at risk. This principle is now widely accepted in many domains.

There were many historical innovations that seriously endangered or poisoned the engineers and their neighbours. If present-day Health and Safety legislation and Anti-Pollution regulations had been in force, the industrial revolution might never have happened. Perhaps some engineers hanker for the old days of reckless entrepreneurial experiment, but I don't want to live next door to someone who is developing a new kind of explosive in his garden shed, thank you very much.

That doesn't mean I think regulation is the best mechanism for governing risk. And I don't for one moment imagine that we can remove all risk and uncertainty from our lives. But where innovation involves serious and obvious risk of loss, especially to other people, I think it is reasonable to insist on some kind of governance mechanism.

Feedback as Regulation

Can we trust software? Bruce Schneier has recently been advocating software liability as a trust mechanism. In the latest issue of Crypto-Gram (December 2004), Geoff Kuenning contributes a contrary opinion.

If you've never read it, you should track down the CACM article on the history of steam boilers that appeared some time in the 80's. The brief summary is that after steam power was invented, there were lots of nasty boiler explosions. In the UK, the problem was dealt with by regulation. In the U.S., free-market advocates succeeded in arguing that liability law was sufficient: boiler makers would lose a few lawsuits and would then have an incentive to develop safer boilers.



The result was that boiler-related deaths dropped to near zero in the UK and continued at high rates for 20 more years in the U.S., until finally we broke down and regulated the industry.



The problem with liability as a feedback mechanism is that the negative feedback is strongly disconnected from the original action. Liability can help, but it's not at all clear to me that simply making MS liable for all the worms in the world would cause them to start making secure software.

From a systems engineering point of view, feedback may be regarded as simply a different mode of regulation. One of the first feedback control systems, invented by James Watt, has become known as the Governor.



Feedback corresponds to a kind of Network Regulation, producing Network Trust. As Kuenning points out, this doesn't work effectively when there are strong disconnects between cause and effect. Kuenning contrasts this with a kind of Authority Regulation, and argues that because the latter was more effective at dealing with boiler safety, it would therefore be more effective at dealing with software quality and trustworthiness.



However, Authority Regulation can also suffer strong disconnects between cause and effect. Is there really an Authority anywhere in the world that can keep Microsoft in check?

Some cases where we might look for the basis of claims

originally posted by John

---

Early in the new year a report is published by US academics warning that eating Scottish farmed salmon potentially causes cancer. The industry erupts in righteous defence. UK scientists are wheeled out whose views refute the US report. UK politicians urge salmon eaters to disregard the warnings and assure us of the product's safety. The Scottish tourist industry worries about the impact on visitors. Salmon farming communities worry about the impact on jobs and local economies. The tabloids run scare stories. Supermarkets put the stuff on BOGOF. A week later and things have calmed down. Then the WHO puts forward its proposals on global diet, physical activity and health hoping to address growing obesity among the world's population - a figure currently standing at more than 3 billion and rising, quite literally. One of their recommendations is that sugar should be limited to a 10% maximum in any food product. The US demurs, saying that their 25% ceiling is perfectly satisfactory, and accuses the WHO of bad science. The fact that the average American daily consumes 34 teaspoonsful of sugar is, they claim, only marginally to blame for Americans' obesity. They fail to add that the $5b US sugar industry gave the Bush campaigns huge political donations out of its $3.5m lobby spend. Then there's another scientist - Prof Sir Roy Meadows. This one seems to have invented a medical condition all of his own - Munchausen's Syndrone by Proxy. Meadows's expertise - his ownership of knowledge - has resulted in upwards of 250 cases being identified as potentially unsafe. Meanwhile thousands more lives have been shattered. Who truly owns all this science? Who can own it? And where, among all these imposed values, all these constrained choices and all the phoney publicness that goes with owning knowledge, is there any space for trust?



John Smith

Rail Safety 3

originally posted by Aidan

---

What rail companies need to do to reestablish a focus on customer service and safety is to work closely together to solve practical problems. This is the deficit that customers actually see. But we note that it is tough and sometimes illegal to step outside the contractual boundaries that have been set. An industry that has been broken up is not the same thing as an industry that has grown up in a "competitive" form.



Aidan

Rail Safety 2

originally posted by Aidan

---

There is a political driver to the rail service and safety question which is disturbing from a trust perspective. There is a natural conflict of interest between the Treasury, the DTI and the MOT which led to the current structure and a political impasse which still exists. The political wrangle has nothing to offer customers and it makes no sense for customers to trust a service where policy is driven by concerns other than theirs.



Aidan

Rail Safety

The safety of the rail network depends (among other things) on our being able to trust in the rail maintenance process, and in those firms engaged in this process.

From this perspective (and not only this perspective) the history of the UK rail industry is seriously troubling.

---

What's wrong with Jarvis? What's wrong with Network Rail?

originally posted October 23rd, 2003


Three public events in close succession. The precise connections between these events have not been made public, but we can make some intelligent assumptions.

• The UK engineering firm Jarvis pulls out of the rail maintenance business.
• Jarvis is accused of falsifying records relating to rail maintenance contracts.
• Network Rail announces that all rail maintenance, previously outsourced to several engineering firms including Jarvis, will now be carried out inhouse.

One thing that stands out strongly is the relationship between safety and trust. The safety of the rail network depends (among other things) on our being able to trust in the rail maintenance process, and in those firms engaged in this process.

The allegations against Jarvis lead to a reduction in perceived trustworthiness - not just of Jarvis but of the whole industry. Even if the allegations are unfounded, and even if it is grossly unfair to paint all engineering firms with the same brush, trust has been damaged.

One of the most important lessons from this affair is that safety-critical engineering cannot be regarded as purely a technical matter, but requires attention to commercial / socioeconomic factors and stakeholder issues.