CIFAR Amii Summer Institute On AI And Society

These are my notes about the CIFAR Amii Summer Institute on AI and Society. This Institute ran July 22 - 24, 2019 at the Amii offices in Edmonton, Alberta.

As always my notes are severely limited by a) what I understand, b) my typing abilities, and c) the batter on my laptop. I can guarantee that I will miss stuff and get other things wrong.

The Twitter hashtag is https://twitter.com/hashtag/DLRLSS

Day 1: July 22

The Institute opened with a territorial acknowledgement and orientation. We are using Slack for sharing information.

Graham Taylor: Machine Learning and Deep Learning

Taylor talked about computer vision and reproducibility issues. He began on algorithms - step by step descriptions. This term wasn't embraced by computer scientists until 1960s. It was a way to encapsulate human knowledge (and replacing thinking). Machine learning systems take input and output and sent out algorithms (designed by the machine.) Traditional algorithms are being replaced by learning algorithms.

Now there are meta-learning systems that learn to learn. (See Pete Warden "Deep Learning is Eating Software".) Some people are calling this "software 2.0". Developers in this model are changing to trainers and teachers of learning system.

Then he talked about convolutional neural networks. This is an architecture that is very successful for computer vision. Note how hardware architecture is changing as ideas about how to do AI is changing. These networks have multiple layers. This architecture solves problems is a manner somewhat similar to how the human brain does. These systems date back to the 1980s, but only now do we have enough data.

Then he talked about computer vision in action (using CNN). There are different classification and labeling approaches. Do you want a single decision or a whole mess of tags. Then there is image segmentation. Instance segmentation, human pose estimation, dense pose are new types of image captioning. You can combine pipelines to do image captioning. Or one can reverse this and synthesize images from text. This is a generative task. Generative models are a popular area of research that also raise issues around deep fakes. Then you can get image question answering. All of these also scale to video.

Then he talked about reproducibility and Joelle Pineau's Checklist. He talked about how, within a field that embraces reproducibility, some are not releasing all the information in order to prevent people from being able to use research for nefarious purposes.

OpenAI is a prominent example of a group not releasing the whole codebase for fear of misuse (dual use.) Taylor mentioned how the barriers to entry for misuse of these algorithms is very low and that should give us pause. Publishing rocket information is less likely to be misused as it takes a lot of infrastructure and training to reproduce innovations. In the case of AI all sorts of people can start playing

We need to be able to anticipate ethical issues before they happen.

He mentioned all the standard answers to ethical questions like "it is just beautiful math", "this is not my area", "I'm not responsible for someone else's misuse"...

He answered questions about danger and gender recognition systems.

My question is whether alogorithms can have bias? Can they have ethics, or is it only in the data?

Dirk Hovy: Natural Language Processing

Hovy talked about NLP. Why is NLP important - because there is a lot of text everywhere. Humans tend to read about 9000 words a day, 200 million in a lifetime. 44 billion GB of new data is generated each day and mostly unstructured. We can't read all the stuff coming out, even in our own field. Text has become interesting. We use NLP systems like search every day.

He talked about topic modelling, sentiment analysis, word embeddings, visualization, and so on. He talked about what we can get out of a text like parts of speech. We might want to look only at nouns or syntactic structure. You can then do target based sentiment analysis. One can do authorship analysis.

These days NLP seems to be mostly machine learning applied to text. The linguistics is becoming less important. NLP can be generation of text now or understanding. Understanding can be prediction or exploration.

So, what's the problem? Language is incredibly complex. We have turned language into engineering problems. Now we have made language into non-linear problem. This is "bogus". We do all sorts of things in language. There is a real danger to oversimplifying language. This leads to trouble and communication breakdown. Language is influenced by demographics. Then there are issues like irony, and gestures, and context.

He showed a great visualization and the shifts of meaning for the word "gay" over time. Our systems don't pay attention to any of the historical context. NLP tens to work only for a very small segment of the population of English speakers. Does just using more people in the training solve the problem?

The sources of bias include:

• Semantics - the meaning and how represented
• Selection - where we get data for training
• Who we ask to annotate the outputs
• Models
• Design

He showed some of the problems that come out from word embeddings. Some of the problem is that sampling is too small. (Can one sample widely enough? Is selection something you can't avoid. Our systems, models have a bias to think the data is representative and the annotation good. Models amplify bias. This leads to problems.

We also have a problem that there isn't enough data for many languages. We also train too much on things we know. New York gets more attention than Lagos. He ended with:

"Am I comfortable with my system classifying me?"

Rich Sutton: Reinforcement Learning

He started by talking about how we are all trying to understand what it is to be a person. In AI we are doing that through modelling intelligence. Intelligence is the ability to achieve goals in the world. (McCarthy) Intelligence is the most powerful phenomena in the universe. (Kurzweil) "Intelligence in the eye of the beholder. (Sutton)

His main points are:

• Understanding intelligence is the greatest scientific prize of all time and it will probably come through creating AI. "Moore's law is strong and AI will come within decades."
• Reinforcement learning is machine learning configured to achieve autonomous AI. RL have goals, values, reason, and knowledge similar to those of human intelligence
• We should work towards a society that is open to all sentient persons, natural and designed.

He gave an example of Hajime Kimura's RL robots. RL acts in the world towards goals. (Is that what we do? On what scale?) You have the agent, the world - the agent has actions, responses, and stimulus. There are rewards. Agent learns a policy mapping states to actions and tries to maximize reward. He then gave examples of successes, including backgammon (1995). The system maps the world onto a network which is a value system. WAtson. RL and Deep Learning playing Atari games from score alone. It has been used for poker. It was used by AlphaGo to play Go. In all these cases, performance was better than could be obtained by other methods.

He said that there was no knowledge. Were the systems programmed with some sort of value? How did it know what was the better.

Reinforcement learning can be thought of as maximizing the reward signal. (Reward Hypothesis) Is this all there is to being human. Is intelligence essentially reward maximization. To do this RL uses "value functions" that are the key to solving decision problems. Agents learn to map states to values. These are not unlike human and societal value systems. (Really?) He showed a Gridworld example.

He summarized:

• RL is autonomous
• RL is more ambitious and therefore harder
• There are human analogues to these (are these analogues useful?)
• RL is based on

He then talked about AI and society. He feels it is the most human field - it is about understanding us - understanding mind. It is about amplifying us. Not exactly us, but the essential us. It is making our lives easier, better. At the same time it will bring radical change. There is a symmetry between man and machine. We have compatible goals (or could have compatible goals.) Some of the issues may drop away. People should not feel entitlement. AIs may not want to be slaves.

• Amplification is more important than replacement - this is less threatening
• AI will become part of what it disrupts. It will bring greater diversity of intelligence.
• We should be willing to give AIs
• The rise of great foresight in the in the universe is perhaps one of the few things that is probably clearly, generally good

I asked him if he tried pain as an alternative to reward. He sees pain as negative reward. I then asked about the body. He shifted to what AI can teach us which suggests that RL doesn't have to work like the human.

There was a question about whether the change is different now.

He seemed to distinguish between humans and persons - Sutton wants to achieve a world that is open to all persons, including persons that are not embodied as we are. But how are we to decide what are persons and what we should be doing to accomodate them. AIs are not yet autonomous persons so why do we need to fetishize their rights. We have hundreds of manifestly intelligent species disappearing yearly - by that logic we should prioritize their livelihood. Further, are we willing to share the right to adjudicate rights? What if machine intelligences don't treat rights as we do?

Elizabeth Joh: Uses of AI in policing and criminal justice.

Joh talked about data-driven policing. She mentioned how policing is not centralized. It is very local.

She focused on predictive policing. Predpol, for example, is designed to direct attention to some areas over others to maximize efficient policing. It is being used in a number of US cities.

Then there are systems that provide "threat scores". Threat analysis software like Beware will, based a variety of inputs, will estimate the threat posed by a person. Chicago has a system to provide a "heat list" that tries to identify who is at risk. Then there is facial recognition that can be direct or indirect. Axon, that used to be Taser, which is the largest provider of body cams. Axon's ethics board has announced that they have an agreement not to currently build recognition into body cams. Then there is doorbell ring cameras. There police departments giving Amazon Ring doorbells to people.

Something not talked about is policing and self-driving cars. The most likely encounter with police in the US is in cars. How does this change with self-driving cars. There could be automated enforcement. Police depts are very interested also in remotely moving cars.

Some of the issues that are coming up:

• Presumption in courts that police are humans and are making judgements - what happens when it is an algorithm or hybrid
• Expansion of policing power
• Bias in data that trains systems - data about things like arrests is based on human decisions that could be biased. These biases could be baked in that then reinforces bias.
• The systems are generated by private companies for police as consumers. These are not transparent as companies want to protect their property. Police often have to sign non-disclosure agreements.

She went back to self-driving cars.

Michael Karlin: Ethical issues in military uses of AI

Karlin has developed a number of policies for the Canadian government. He started with the question of what is the role of a medium or small sized country today. How can small powers project power. Can AI exploit data to arm a smaller state with more power than was previously able to wield. Militaries have become high data organizations which are now trying to figure out how to handle their data. How do we deal with all the data from Afghanistan. There is a bit of an arms race, but no one knows of what sort.

The mission of the DND and CAF is to defend Canada and allies. Provide peace operations. Help civil authorities with things like forest fires. Respond to disasters and conduct search and rescue operations. They also provide all sorts of services.

He then talked about the Terminator in the room - the issue of unpiloted kill machines. The DND has an ethics code that people have to adopt to join the military. He is working on these ethical shifts. He also talked about all the areas where AI can make a difference and he showed a complex governance slide.

The armed forces work in a complex legal and policy framework from the Constitution to international guidelines. He is a defense civilian and is under even different standards. The DND has develop a code of values and ethics. Other considerations include interoperability with allies, data management and governance across international alliances. Really important is the culture of the office class - they don't trust data. They trust their experience and intuitions.

Then there are cost issues. What if you build a fancy system that can be defeated by a cheap hack like a bit of face paint defeating face recognition. How can the costs be balanced to make sensible decisions.

Also, there are really issues about taking decisions away from commanders or civilian governance. Could AI nudge us towards force? There are also issues of the rules of engagement and how those can be gamed. The Taliban learned them and shifted continuously to game the system.

The challenge is to set practical, balanced rule sets.

Trooper Sanders: Extracting Value from Biased Data: Politics, Peril, Promise

Sanders has a policy background. He wanted to focus on bias and flip the question to "can we extract value from biased data?" He talked about examples of how data, even biased, data can be of help. He gave the example of the Flint water crisis. There were many levels to this from the issue of lead in the water to the systemic racism behind this to the issue of how to deal with long term effects. He mentioned that we are always dealing with history. History provides the context for interpreting what AI provides.

He talked about how the information/data can tell you about yourself. Does the bias tell you about what you have been doing. The data can help us understand whether there are systemic issues. It might even point to new frontiers - ML may create new classes of people facing discrimination. The use in health care could create new classes of people discriminated against. In short AI can mirror our hidden values.

We always have to strike a balance between competing public interest concerns.

He talked about how organizations don't want scrutiny and to be held accountable so they are likely not to look to their data for understanding. How can we encourage organizations to do this?

One way is to create sandboxes. Can we create an environment of consent degrees? Can we build the right regimes that will hold people accountable and yet not squelch innovation?

There was a question as to what is bias? It is perspective - a prejudice in favor of some subgroup.

Elana Zeide: Predictive analytics in education and employment: equity and opportunity implications

Zeide's work looks at platforms that make continuous decisions. She talked about systems in education that give continuous advice to students and teachers. These technologies create machine readable humans and identify promising candidates. They can watch linked in or other social media and then reach out to those they want. This can lead to the marginal candidates being ignored.

When we move from discrete systems to platforms there is an impact on visibility and timing of decisions. Protected groups can be officially not tracked, but there are all sorts of other signals that can be used that in fact end up being biased. She is interested in how we create the pipeline not just the decisions. The pipeline tends not to be regulated. The pipeline can limit what choices can be made further down the line.

Educational systems can track physical/behavioural cues over time that then become hard to erase if we want people to have the ability to erase their past or reinvent themselves.

She talked about rectification and contestability. One should be able to fix data about you and to contest decisions. With platforms it is hard to contest as the models and data are continuously changing. It is hard to go back and figure out why a decision was made. It is hard to be accurate. This is why we use proxies. We use grades as a representation of a person's knowledge.

The systems do embedded assessment, they are constructions, they use proxies, they are opaque and challenge traditional assessment laws. They are developing all sorts of new credentials. They will change the pathways to decisions, not just the decisions.

She talked about having a dominant market player like Linked-In. Now Linked-In is taking advantage of this to sell certificates and to then sell your data to someone else.

Like the previous speaker she talked about protected groups or classes and how there may be new forms of prejudice. Who decides what are the protected classes and then how do we make sure that indirect signals don't bake in bias.

Geoffrey Rockwell: AI Anxiety in historical context: Public discourse about automation, past and present

I spoke about reading past concerns and anxieties around automation. I drew on the history of the discourse around automation to reflect on what could happen now.

Gary Marchant: Limits of legal/regulatory approaches, soft-law and international alternatives

He started with the question of what we should regulate and how there are different types of regulation. There is top down, hard law in statutes and official law. There is also soft law, where one creates substantive obligations and requirements quickly and without the burden of hard regulation. AI is a case where there are so many agencies involved and things are moving so fast that it would be better to have faster and softer ways of developing regulations. Of all these reasons there probably won't be much law in the short term.

The advantages of soft law is that they can be developed quickly. There can be participation and even accountability. The problems are also great - it is hard to find hold orgs accountable. The public often doesn't have any confidence in them. The perception is that self-regulation doesn't work. Can we prove it works or not? See the Principled AI Project - from the Berkman Klein Center.

There is also the question of how to do soft law well. Part of it is soft law processes. You can have big companies using supply chain requirements. Corporate boards can play a role as can ethics committees. Trade associations, NGO Scorecards/Rankings, 3rd party audits, certification, funding agencies and so on. University ethics committees are examples of a sector that is self-regulating.

Insurers, journals, and other organizations can end up providing a surrogate form of regulation.

He concluded about how hard law doesn't work very well either so "soft law is the worst possible approach except for all the others."

He then looked at international governance. Everyone is trying to set up international governance. Why is everyone trying to develop international coordination? How can one get international coordination? Can these international efforts use soft processes? He showed an OECD matrix of 11 types of international regulatory cooperation.

Brenda Leong: AI Ethics Boards: Some humans need not apply

In March Google announced the formation of a form of ethics board. Many of the people appointed were reputable scientists and so on. One person was someone who was anti-trans in the hopes of having a diversity of opinions. That led others to bow out and then Google shut down the whole thing. That leads to questions about how one gets diversity and appropriate representation on ethics boards. These boards are examples of soft law. How can we do them well.

Why is this happening now? Why are they all publishing principles? How can a company do a believable job and not just appear to be doing corporate ethics washing. We have seen companies advertise themselves as ethical like Nike and Chick-fil-A. How can companies build credible boards and get the right people? Some of the challenges:

• Transparency of selection process
• Clear statements of power
• Prospective members need to know about the others
• What is the ideal makeup of the board. Should there be ethicists
• What is the role of the board and enforcement mechanisms
• Processes should be made public

They decided in their article on this subject that Human Rights is a good way to deal with issue. There is a literature and processes that can be used. She feels that one can ask participants on a board if they agree about human rights. Human rights principles would become a base level for selecting board members. She feels this would still allow for lots of diversity. Here is a quote from her article:

When I taught business ethics in the 1980s/90s there was a lot of talk about mission statements and companies auditing whether they were abiding by their missions and values. Now there is a focus on principles. What do principles leave out? What are they good for?

Craig Shank: Realizing the AI promise, avoiding the risk: How companies need to take greater responsibility, governments need to catch up, and multi-stakeholder governance processes need to bridge the gap

Shank talked about how he is no longer with Microsoft so he is at day 1 of figuring out what he thinks. He is impressed by both the iron man competition here in Edmonton and in our Institute. To get things right we need to think long haul and we need to be interdisciplinary. He talked about how the benefits of AI could be tremendous.

Technology is neither good, nor bad, nor is it neutral. He quoted Kranzberg. What matters is what we choose to do? Lets start asking what technology should do! We have to move from Can We, to Should We, to How Will We that can return to Can We. Ethics frameworks can be useful, but we are missing some core tools. How do we attach the sky to the ground? AI systems don't play by traditional rules. They will take a mix of technical, policy, and social science work.

He then talked about how ethics and practices will need to work across all forms of law. Ethics is not a box to be ticked.

Jason Millar: Autonomous and Connected Vehicles: Emergent issues and longer-term implications

Millar talked about a paper he has developed with a former student on An Ethics Evaluation Tool for Automating Ethical Decision-Making in Robots and Self-Driving Cars. He began with a personal intro so we could tell where he was coming from. He holds a research chair in engineering and ethics. He wants ethics built right into the design process. He smuggles ethics in through tools for his engineering students.

His argument is that we have a whole set of problems that self-driving cars raise. First, we are already automating human drivers. We are delegating navigating to Google maps. There is a history of this. He showed a turn-by-turn directions system from 1909. He showed the Chadwick Road Guide. We see it again and again. Actually these go back to the Vicarello Goblets. Now we can think about the values that will prioritize the routes.

He next talked about automating mobility systems. The systems coming from Google etc. could become tomorrow's mobility traffic flow managers. We are not only delegating individual mobility choices, we could soon delegate the whole traffic flow system management. Like net neutrality where we have seen how companies can manage packet flow, soon we could see companies managing flow of vehicle traffic. Who gets to control the traffic? On what grounds can we permissibly govern flow? Impermissible is discrimination that distrorts secondary markets. Permissible is anything that promotes integrity of the network. In the net neutrality discussion the flow was of data, now it is of people. Will people be comfortable with being managed?

So ... we need to develop the research areas:

• We need to develop the language and arguments that define the ethics/politics of automated mobility systems
• We need to translate that into design and policy tools

There is an interesting issue about infrastructure. Roads are public. Cars are private. The networks are private. Then we have to decide how to develop values. How can we provide local governance.

Evan Selinger: Facial recognition and implications: a pivotal governance moment?

Selinger started by referencing things said earlier. He is interested in transaction costs. He started with a quote by Evan Greer that biometric surveillance is categorically different. It changes scale and should therefore be banned.

He then talked about Amazon Rekognition - a cloud-based system that customers can use. It can do sentiment analysis and can extract text. And it is cheap to use. The entry costs are low. There are plenty of benefits. It could help the police catch people quickly and deter crime. There are also lots of concerns about both systems that work well and those that don't. There are due process problems - when recognition is used to convict someone how can it be challenged. Does the code have to be disclosed?

Then he asked about chilling effects. Then there is digital epidermalization and applied junk science. There is an erosion of obscurity. AIs could contribute to an unnecessary shooting by identifying someone as hostile.

What is interesting is that a year ago we heard a lot about banning face recognition. People said it would never happen. Now, all of a sudden, we may see bans. We have seen it in San Francisco. What does it mean to ban a technology? If the technology is accessible is it too late to ban it? Can one ban it anyway? Bans can have a symbollic value.

Then he talked about the different types of bans:

• Federal
• Local
• Commercial
• Tech companies refuse to service government

He identified three approaches to banning/regulation:

• Keep calm and stop the cycle of privacy/tech panics - the libertarian response
• Cost-benefit balancing to the rescue
• Tech-neutral regulation or bust

Part of the way things get discussed is by being compared to previous panics. Face recognition gets compared to fingerprinting. But are they similar. Faces can be captured easily and in bulk. It is harder to hide your face. Photography is permissionless. Faces are more high value information because there is so much information we can get from them like mood/sentiment. FR is likely to spread much faster than fingerprinting. FR is a real case for function creep. There is a slippery slope here and a non-falacious one.

We might want to use pause buttons (banning) in cases where there are similar structural slippery slopes.

He talked about the economic argument that others are doing it so we have to or we will lose out. His response is that we need moral leadership.

Osonde Osoba and Casey Bouskill: Technology and Culture: Neglected interactions in AI impacts and governance

We then had a double feature. Oshoba started by talking about the clash of civilization thesis that post-cold war conflicts are more likely to stem from cultural frictions. What is the unit of civilization in the networked world. Can we map cultures? What is culture? For them culture is the values, norms, individuality and accepted behaviours.

He has a paper at: https://escholarship.org/content/qt26s3d0mz/qt26s3d0mz.pdf?t=po1osn&nosplash=4ee9b37ee0ade799f518d4027effb034

He talked about the intersections:

• The data is cultural detailed
• The problems chosen are culturally determined

Any tool is inert until picked up and used. The use is socio-culturally determined. Technology also influences culture. She raised the issue of whether AI is really new or a repeat of something else.

What are the sociocultural norms that will influence on AI development and use? How can we think about risk and security in the context of cultural pluralism?

They then compared US doctrine of use for defense and Chinese doctrine. The US want meaningful human oversight while the Chinese wants quick use.

They talked about the plurality of points-of-view on technology use. There is a civilian military divide. Then the HHS in the US has the authority to impose ethics on researchers getting their funding. The new Final Rule and Regulation for US IRBs includes language that has to be in consent forms to the effect that investigators could share the data. IRBs are a cultural system. Some will deal with sharing issues. They typically don't follow up.

The willingness to share data is changing. People over time are less willing to share. Is broad consent broad or consent?

AIs are fed by data. What are the cultural lives of data.

She then talked about cultural problems in systems. AIs can automate diagnostics. But, getting cured is often up to a specialist who may not trust the AI. There is a culture to specialists.

They closed with a visualization of the different types of social media used around the world. Different cultures are using different mixes of systems.Winning the tech cold war is more about cooperating than dominating.

Rob Lempert: Algorithmic Design and Societal Impact: Choosing our Scenario?

Lempert is talking ahout the longer futures. How do we thinking about longer range futures. We consider a multitude of futures. We seek robust rather than optimal strategies. We also want adaptive strategies. We use the computer to facilitate discussion. Traditionally people use predictions and then act. This doesn't work for some problems. Uncertainties can be underestimated and one can get gridlock. Lempert and colleagues reverse the flow and run the analysis backwards. The analytics then tell you about possible scenarios.

Exploratory models will map assumptions onto consequences without priviledging assumptions. Consolidative models try to merge all the models. They can use scenarios to craft strategies and adapt over time to new information.

At RAND they use these tools for decision making for wicked problems. It is more collaborative and engaged more groups of diverse interests.

He then transitioned to how do we measure what we are doing? He showed a list of capabilities developed by Martha Nussbaum as to what constitutes a meaningful life. One can turn these lists into quantifiable metrics.

He then compared the list of capabilities against AI. Can AI help with these? Some yes, some no.

Finally, the relationships among markets, equality and popular sovereignity has changed over time. If we are having another revolution - will AI reinforce corporate power, or will individual? Could AI be used by people to do without companies? Can we learn from earlier utopian experiments?

Pitches

We then brainstormed a number of pitches for what teams in the institute could do. Some of the ones I can think of include:

• How can principles be operationalized by organizations?
• Can we create a machine that provides provocative responses to ethical statements that encourage ethical thinking?
• How can learn about and read AIs (and not so AIs) tools that hidden or protected?

There were a number of great ideas including:

• What does explainability mean? Can we build an explainer?
• Can we develop guidelines for AI augmented decision making?
• Requirements of a virtual assistant that can help someone interact with the world more ethically?
• Policies around the creation of virtual persons?
• How can we characterize and assess risks of uses of AI in health?
• What do we need to preserve now to have a useful history of AI later?

Day 2, Tuesday

We then had a series of pitches on things that we could do and which would be useful.

Local Policy Maker Primer

Local policy makers don't have expertise and don't have the funds to hire people. The idea is to develop a primer for local policy makers.

Making STEM students care: a modular ethics curriculum for integration into AI classes

Developing ethics component for machine learning courses.

Accessible AI: Helping Non-technical Stakeholders Respond to AI Developments

Creating educational materials to bring non-technical statkeholder in.

Drafting Narrow Standards for Ethical AI

Would like to identify the next standards to be developed by the IEEE.

If AI is So Smart, What Are Policy Analysts For? Actually, What Are Politicians For?

Wants to imagine what an AI policy analyst would look like and then write it up as a think piece.

Enabling the Emergence of a Levelers Scenario

Imagine a scenario of the future when AI would be for the good of all. How would we get to such a scenario.

A Critical Paper on Ethics Guidelines for Trustworthy AI

Wants to write a paper focused on the recent EU documents on ethics and AI. He wants to write a reaction paper to a specific set of proposals.

Beyond Cambridge Analytica: Privacy on social media through the looking glass of AI analytics

Wants to do risk assessment of how anonymous authors can be tracked and then have inferences about them.

Turn-by-Turn Navigation Five Ways

The idea is to imagine how different priorities would mean different interfaces to mobility systems.

The Myth of Agency: a blindspot in AI ethics debates

Wants to map ideas about agency and write a short popular book on it.

Exploring Frameworks for Meaningful Human Control over Human-Machine Partnerships in Normal Accident Environments

Look at historical models and other approaches to imagine how we can have meaningful control of human-machine partnerships.

Day 3, Wednesday, July 24th

At the end of the Institute we heard from all the groups:

We first heard about Meaningful Human Control of AI and human-machine partnerships. They drew up a great list of issues to think about. Autonomous military systems is a major site for concerns. What is different now? Some of it has to do with technological shifts. Some of it has to do how systems are being developed that don't really weave in humans and other systems have humans, but they don't really have the ability to intervene like the car in Tempe. Then they talked about situations where you don't want a human involved. They chose a scenario of radiology. Their conclusion is that in high-stakes environments adaptability may be more important.

The Agency group presented their work and started with a neat AI & Agency web site. The site will be at AI and Agency dot com. It goes live on August 1st. They showed a neat mind map of the field. One of the main things they learned is how ideas need to flow not only from the humanists to the scientists, but also the other way.

The Turn-By-Turn navigation system mocked up some different mobility interfaces. They branded it mobi.ly. They focused on a design for safety. They used a design for human values method. They looked at stakeholders, the values they have, and the value tensions that come up. They showed design documents and the designs which showed which routes are the least cognitively stressful. Their process was interesting in that it showed how design can be ethical. Their design was a great example of a good use of AI. A use imagined from the start to benefit people.

I was part of a group that wrote a response to the EU Guidelines on Ethical AI.

Then a team talked about what a Policy AI might look like. They talked about use cases like providing policy advice on Huawei. They thought about how policy analysts would need AI literary to deal well with these tools.

A group worked on a potential IEEE tool for Measuring Rights Threats. They took the declaration of Human Rights and turned it into questions and then into metrics. They found some areas worked and some were harder. They also looked at one for Measuring Bias. They talked about having scores and weights that affected the score. They found it superhard to avoid slipping from measuring ethics into measuring methods. Identifying and naming areas of ethical concerns is difficult. Quantifying ethics is hard. Answers to yes or now questions are low resolution.

The AI Without Math team have mocked out a web site where they can provide training for decision makers. They showed a page on the top 11 questions to ask developers.

We then stepped back and thought about the Institute as a whole.