Designing Trustworthy AI: A Human-Machine Teaming Framework to Guide Development at AAAI Symposium

1
Designing Trustworthy AI: A Human-Machine Teaming Framework
to Guide Development
© 2019 Carnegie Mellon University
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
Designing Trustworthy AI:
A Human-Machine Teaming
Framework to Guide
Development
Carol J. Smith, @carologic
AAAI 2019 Fall Symposium on AI in Government and the
Public Sector, Washington, DC, November 7–9, 2019.
Sponsored by the Association for the Advancement
of Artificial Intelligence.

2
Related Resources
• Paper on arXiv: https://arxiv.org/abs/1910.03515 "Designing
Trustworthy AI: A Human-Machine Teaming Framework to Guide
Development"
• Checklist and Agreement - PDF on Google Drive:
https://drive.google.com/open?id=1rY8EF-
2orGquI23g7yQ5aWGJtUmzP4d6
• SEI page for Designing Trustworthy Artificial Intelligence:
https://sei.cmu.edu/research-capabilities/all-
work/display.cfm?customel_datapageid_4050=197910

3
AI’s Great Promise
Empower us with knowledge
Augment our effectiveness
We can—and must—ensure that we keep humans
safe and in control

4
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
How can AI development teams
harness the power
of AI systems
and design them
to be valuable to humans?

5
Designing Trustworthy AI
Diverse Teams
– Shared Ethics

6
Why diverse teams?
Focus more on facts
Process facts more carefully
More innovative
“They may also encourage greater scrutiny of each member’s actions,
keeping their joint cognitive resources sharp and vigilant.”
“…become more aware of their own potential biases”
Why Diverse Teams Are Smarter. Harvard Business Review. https://hbr.org/2016/11/why-diverse-teams-are-smarter

7
What is a diverse team?
Diverse with regard to gender, race, education, thinking process,
disability status, and more…
An inclusive environment is required to make diversity successful.

8
Multi-Disciplinary
• Skill set and problem framing approach
• Machine learning experts, programmers, system architects, product managers,
etc. and…
• Curiosity experts
• Focus on understanding situation, constraints, and abilities
of people who will use system and how will be used
(includes: HCI, HMI, cognitive psychologists, digital anthropologists, and UX researchers)

9
1. Well-being
2. Respect for autonomy
3. Protection of privacy and intimacy
4. Solidarity
5. Democratic participation
6. Equity
7. Diversity inclusion
8. Prudence
9. Responsibility
10. Sustainable development
Coalesce around a shared set of technical ethics
Montréal Declaration for a responsible development of artificial intelligence.
https://www.montrealdeclaration-responsibleai.com/the-declaration

10
Diversity + Ethics
Accountable
De-risked
Respectful
Secure
Honest
Usable
AI
Process
?
Montréal Declaration for a responsible development of artificial intelligence.
https://www.montrealdeclaration-responsibleai.com/the-declaration

11
HMT Framework for Designing Ethical AI Experiences:
Themes
1) Accountable to humans
2) Cognizant of speculative risks and benefits
3) Respectful and secure
4) Honest and usable

12
Goals of HMT Framework for Designing Ethical
AI Experiences
Pair with technical ethics - bridge gap between “do no harm” and reality
Reduce risk and unwanted bias
Mitigation plans
Support inspection - ethical and trustable process and system

13
Conversations and Understanding
Guide AI development teams
Inform effort
Create AI systems that are accountable, de-risked, respectful, secure,
honest, and usable.

14
Accountable to Humans
Ensure humans are always in control, able to monitor and control risk
Designate responsibility for all decisions and outcomes to humans

15
Cognizant of Speculative Risks and Benefits
Identify full range of
•Harmful, malicious use, as well as good, beneficial use
•Blind spots and unintended consequences
Create communication and mitigation plans for misuse/abuse of AI system

16
Respectful and Secure
Value humanity, ethics, equity, fairness, accessibility, diversity and
inclusion
Be robust, valid and reliable
Respect privacy and data rights
Provide understandable security

17
Honest and Usable
Value transparency with the goal of engendering trust
Explicitly state identity as an AI system

18
Designing Trustworthy AI
Checklist and Agreement
PDF on Google Drive: https://drive.google.com/open?id=1rY8EF-
2orGquI23g7yQ5aWGJtUmzP4d6

19
We are confident that we have designed our AI system
so that:
 Humans are always ultimately in control, able to monitor
and control risk
 Designate responsibility to humans for all decisions and outcomes
 Explicitly defined responsibility and who shares responsibility
 Preserve human responsibility for final decisions that affect
a person’s life, quality of life, health, or reputation
 Significant decisions made by the AI system are appealable,
able to be overridden, reversable

20
We identified the full range of risks and benefits:
 Harmful, malicious use
 Good, beneficial use
 Blind spots and unintended consequences

21
We have created plans:
 Communication plan(s) for misuse/abuse of AI system
 Mitigation plans for misuse/abuse of AI system

22
The AI system is respectful and secure:
 Integrated values of humanity, ethics, equity, fairness, accessibility,
diversity and inclusion
 Respected privacy and data rights
 Provided understandable security methods
 AI system is robust, valid and reliable

23
We value transparency with the goal of engendering
trust:
 Purpose and limitations of the AI system are explained in plain
language
 Data sources and training methods have unambiguous sources
and are verifiable
 Confidence and context are presented for humans to base decisions on
 Provided transparent justification for outcomes
 Straightforward, interpretable, monitoring systems

24
The AI system explicitly states its identity, is honest and
usable:
 Easily discern when interacting with AI system vs. a human
 Easily discern when and why the AI system is taking action and/or
making decisions
 Improvements made regularly to meet human needs and technical
standards

25
[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.[DISTRIBUTION STATEMENT A] Approved for public release and unlimited distribution.
Create AI systems that are accountable,
de-risked, respectful, secure, honest,
and usable

26
Make Trustworthy AI Systems
• Diverse team in inclusive environment
• Shared set of technology ethics
• Conduct activities to understand people’s needs and concerns
for the system
• Encourage deep conversations to align on clear expectations
and mitigation plans
• Use the HMT Framework’s Checklist and Agreement to design
ethical AI experiences

27
https://sei.cmu.edu/research-capabilities/all-work/display.cfm?customel_datapageid_4050=197910

28
Carol J. Smith
cjsmith@sei.cmu.edu
Twitter: @carologic
SEI Emerging Technology Center
Twitter: @sei_etc

Designing Trustworthy AI: A Human-Machine Teaming Framework to Guide Development at AAAI Symposium

More Related Content

What's hot

Similar to Designing Trustworthy AI: A Human-Machine Teaming Framework to Guide Development at AAAI Symposium

More from Carol Smith

Recently uploaded

Designing Trustworthy AI: A Human-Machine Teaming Framework to Guide Development at AAAI Symposium

Editor's Notes