Fairness-aware Machine Learning: Practical Challenges and Lessons Learned (WSDM 2019 Tutorial)

Fairness-aware Machine Learning:
Practical Challenges and Lessons Learned
WSDM 2019 Tutorial
February 2019
Paul Bennett (Microsoft)
Sarah Bird (Facebook/Microsoft)
Ben Hutchinson (Google)
Krishnaram Kenthapadi (LinkedIn)
Emre Kıcıman (Microsoft)
Margaret Mitchell (Google)
https://sites.google.com/view/wsdm19-fairness-tutorial

The Coded Gaze [Joy Buolamwini 2016]
• Face detection
software:
Fails for some
darker faces
• Cf. "racist soap
dispensers"
https://www.youtube.com/watch?v=KB9sI9rY3cA

Gender Shades [Joy Buolamwini & Timnit Gebru, 2018]
• Facial analysis
software:
Higher accuracy for
light skinned men
• Error rates for dark
skinned women:
20% - 34%

Algorithmic Bias
▪ Ethical challenges posed by AI
systems
▪ Inherent biases present in society
– Reflected in training data
– AI/ML models prone to
amplifying such biases
▪ ACM FAT* conference /
KDD’16 & NeurIPS’17 Tutorials

Laws against Discrimination
Immigration Reform and Control Act
Citizenship
Rehabilitation Act of 1973;
Americans with Disabilities Act
of 1990
Disability status
Civil Rights Act of 1964
Race
Age Discrimination in Employment Act of
1967
Age
Equal Pay Act of 1963;
Civil Rights Act of 1964
Sex
And more...

Fairness Privacy
Transparency Explainability

Fairness Privacy
Transparency Explainability
Related WSDM’19 sessions:
1.Tutorial: Privacy-Preserving Data Mining in Industry (Monday, 9:00 - 12:30)
2.H.V. Jagadish's invited talk: Responsible Data Science (Tuesday, 14:45 - 15:30)
3.Session 4: FATE & Privacy (Tuesday, 16:15 - 17:30)
4.Aleksandra Korolova’s invited talk: Privacy-Preserving WSDM (Wednesday, 14:45 - 15:30)

“Fairness by Design” for AI products

Outline / Learning Outcomes
• Algorithmic Bias / Discrimination
• Industry Best Practices
• Sources of Biases in ML Lifecycle
• Techniques for Fairness in ML
• Fairness Methods in Practice: Case Studies
• Key Takeaways
• Key Open Problems in Applied Fairness

Algorithmic Bias / Discrimination
and broader / related issues

Other Great Tutorials
Fairness in Machine Learning
Solon Barocas and Moritz Hardt, NeurIPS 2017
Challenges of incorporating algorithmic fairness into practice
Henriette Cramer, Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III,
Miroslav Dudík, Hanna Wallach, Sravana Reddy, Jean Garcia-Gathright, FAT* 2019
Defining and Designing Fair Algorithms
Sam Corbett-Davies, Sharad Goel, ICML 2018
The Trouble with Bias
Kate Crawford, NeurIPS 2017 Keynote

"[H]iring could become faster and less expensive, and […] lead
recruiters to more highly skilled people who are better matches for
their companies. Another potential result: a more diverse workplace.
The software relies on data to surface candidates from a wide variety of
places and match their skills to the job requirements, free of human
biases."
Miller (2015)
[Barocas & Hardt 2017]

"But software is not free of human influence. Algorithms are written
and maintained by people, and machine learning algorithms adjust
what they do based on people’s behavior. As a result […] algorithms can
reinforce human prejudices."
Miller (2015)

Do Better
Avoid Harm
[Cramer et al 2019]

More positive outcomes & avoiding harmful outcomes
of algorithms for groups of people
[Cramer et al 2019]

[Cramer et al 2019]
More positive outcomes & avoiding harmful outcomes
of automated systems for groups of people

Legally Recognized Protected Classes
Race (Civil Rights Act of 1964); Color (Civil Rights Act of
1964); Sex (Equal Pay Act of 1963; Civil Rights Act of
1964); Religion (Civil Rights Act of 1964);National origin (Civil
Rights Act of 1964); Citizenship (Immigration Reform and
Control Act); Age (Age Discrimination in Employment Act of
1967);Pregnancy (Pregnancy Discrimination Act); Familial
status (Civil Rights Act of 1968); Disability
status (Rehabilitation Act of 1973; Americans with Disabilities Act
of 1990); Veteran status (Vietnam Era Veterans' Readjustment
Assistance Act of 1974; Uniformed Services Employment and
Reemployment Rights Act); Genetic information (Genetic
Information Nondiscrimination Act) [Boracas & Hardt 2017]

Other Categories
Societal Categories
i.e., political ideology, language, income, location, topical interests,
(sub)culture, physical traits, etc.
Intersectional Subpopulations
i.e., women from tech
Application-specific subpopulations
i.e., device type

Types of Harm
Harms of allocation
withhold opportunity or resources
Harms of representation
reinforce subordination along the lines of identity, stereotypes
[Cramer et al 2019, Shapiro et al., 2017, Kate Crawford, “The Trouble With Bias” keynote N(eur)IPS’17]

Bias, Discrimination & Machine Learning
Isn’t bias a technical concept?
Selection, sampling, reporting bias, Bias of an estimator, Inductive bias
Isn’t discrimination the very point of machine learning?
Unjustified basis for differentiation

Discrimination is not a general concept
It is domain specific
Concerned with important opportunities that affect people’s life
chances
It is feature specific
Concerned with socially salient qualities that have served as the basis
for unjustified and systematically adverse treatment in the past

Regulated Domains
Credit (Equal Credit Opportunity Act)
Education (Civil Rights Act of 1964; Education Amendments of 1972)
Employment (Civil Rights Act of 1964)
Housing (Fair Housing Act)
‘Public Accommodation’ (Civil Rights Act of 1964)
Extends to marketing and advertising; not limited to final decision

Discrimination Law and Legal Terms
Treatment
Disparate Treatment, Equality of Opportunity, Procedural Fairness
Outcome
Disparate Impact, Distributive justice, Minimized inequality of outcome

[https://www.reddit.com/r/GCdebatesQT/comments/7qpbpp/food_for_thought_equality_vs_equity_vs_justice/]

Fairness is Political
Equal Treatment vs Equal Outcome

Jobs Product
Women and Men get equally good job recommendations
Both click on recommendations equally
Women and Men apply to jobs equally
Both apply to the jobs at the same rate
Both apply to the same total number of jobs
Both apply according to their proportion in the population
(in the product? In the US? In the world?)
Women and Men are hired to jobs equally
Full time vs part time? Temp vs permanent?
Women and Men are hired equally to equally good jobs
What is a good job? One that they value? Best hours? Best income?

Fairness is Political
Someone must decide
Decisions will depend on the
product, company, laws, country, etc.

Why do this?
Better product and Serving Broader Population
Responsibility and Social Impact
Legal and Policy
Competitive Advantage and Brand
[Boracas & Hardt 2017]

Industry Best Practices
for Product Conception, Design, Implementation, and Evolution

Process Best Practices
Identify product goals
Get the right people in the room
Identify stakeholders
Select a fairness approach
Analyze and evaluate your system
Mitigate issues
Monitor Continuously and Escalation Plans
Auditing and Transparency

Repeat for every new feature,
product change, etc.

Identify product goals
Be specific
What are you trying to achieve?
i.e., remove all violent content
For what population of people?
i.e., for all users, for younger users
What metrics are you tracking?
i.e., percentage of violent content removed

Jobs: Identify product goals
What are you trying to achieve?
Match people with job opportunities
For what population of people?
US users
What jobs?
Local entry-level jobs
What metrics are you tracking?
Job applications per job and per user

Get the right people in the room
Different domains require different expertise and decision makers to
be involved
Internal People
Product leaders, legal, policy, user research, design, social scientists,
domain experts, machine learning experts
External People
Academics, Consultants, Advocacy groups, Government agencies

Jobs: Get the right people in the room
Internal People
Product leaders, legal, policy, social scientists
External People
Academics, advocacy groups, government agencies

Identify stakeholders
Who has a stake in this product?
i.e., content producers, content consumers
Who might be harmed?
i.e., small local content producers
How?
i.e., underrepresentation

Jobs: Identify stakeholders
Who has a stake in this product?
business trying to hire, people seeking jobs
groups of people, society as whole?
Who might be harmed?
business trying to hire, people seeking jobs, society as whole
How?
Allocation

Select a fairness approach
What type of fairness?
Group vs individual
At what point?
Equal outcome vs equal treatment
What distributions?
US population, users of the product

Jobs: Select a fairness approach
What type of fairness?
Group: women and men
At what point?
Equal treatment
What distributions?
US population

Analyze and evaluate your system
Consider the complete system end-to-end including people, technology
and processes
Break your system into components
Analyze each component to understand the decisions made and their
impact
Determine how well it matches up to your selected fairness approach

Engineering for equity during all phases of ML design
Problem
Formation
Dataset
Construction
Algorithm
Selection
Training
Process
Testing
Process
Deployment
Feedback
Does our data include enough
minority samples? Is the data
skewed? Can we collect more data or
reweight?
Are there missing/biased features?
Was our historical data generated by
a biased processed that we reify?
Do our labels reinforce stereotypes?
Do we need to apply debiasing
algorithms to preprocess our data?
Credit: K. Browne & J. Draper

Jobs V1: Analyze and evaluate your system
Analyzed recommendations:
Women apply to many more jobs than men
Women click on many more of the recommendations than men
Men and women are seeing similar job recommendations
Women click on different types of jobs than men
There are not many men in the training data
Note: this gets worse with feedback

Jobs V2: Analyze and evaluate your system
Analyzed recommendations:
Women apply to many more jobs than men
Women click on many more of the recommendations than men
Men and women are seeing similar job recommendations
Women click on different types of jobs than men
Women and men are equally represented in the data set
We have very few of the types of jobs men click on in the inventory

Mitigate issues
Decide if you need to change your design, data, or metrics
Consider all types of interventions
Add balancing tracking metrics

Jobs: Mitigate issues
V1
Make men more represented in the data set
Track proportion of men and women in the system and in the data
V2
Interview male users and potential users, find out what they are
looking for
Reach out to local business to improve job inventory
Track different job categories and properties in inventory

Monitor Continuously and Escalation Plans
Build in monitoring and testing for all the metrics you are tracking
Things drift because the world changes, user behavior changes, etc.
Every time you deploy a model (or before), as the system runs, etc.
Develop response and escalation plans
How are you going to respond when something happens?
What blocks launch?
Who decides?

Jobs: Escalation Plans
The night before our first major launch we discover the model doesn’t
perform well for men because of issues with the dataset
We told investors that we would launch tomorrow and all of the PR and
marketing is set to go live
Do we launch? Who decides?

Auditing and Transparency
Important to also consider who else needs visibility into your process
and your system
Do you need to prove that your system meets regulations?
Do you want outside experts to certify your system?
Do users need to understand fairness in the system?

Sources of Biases in ML Lifecycle

Collaborators
Much of this section is based on survey paper and
tutorial series written by Alexandra Olteanu,
Carlos Castillo, Fernando Diaz, Emre Kıcıman

Data bias: a systematic
distortion in data that
compromises its use
for a task.

Note: Bias must be considered relative to task
62
Gender discrimination is
illegal
Gender-specific medical
diagnosis is desirable

What does data bias look like?
Measure systematic distortions along 5 data properties
1. Population Biases
2. Behavioral Biases
3. Content Production Biases
4. Linking Biases
5. Temporal Biases

Measure distortions along 5 data properties
Differences in demographics or other user characteristics between a user
population represented in a dataset or platform and a target population
4. Linking Biases
5. Temporal Biases

Example:
Different user
demographics on
different social
platforms
See [Hargittai’07] for statistics about social media use
among young adults according to gender, race and
ethnicity, and parental educational background.
65
Figure from http://www.pewinternet.org/2016/11/11/social-media-update-2016/

Systematic distortions must be evaluated in a
task dependent way
Gender Shades
E.g., for many tasks, populations
should match target population,
to improve external validity
But for some other tasks,
subpopulations require
approximately equal
representation to achieve task
parity
http://gendershades.org/

Differences in user behavior across platforms or contexts, or across
users represented in different datasets
4. Linking Biases
5. Temporal Biases

Behavioral Biases from Functional Issues
Platform functionality and algorithms influence human behaviors
and our observations of human behaviors
[Miller et al. ICWSM’16]
Figure from: http://grouplens.org/blog/investigating-the-potential-for-miscommunication-using-emoji/

Cultural elements and social contexts are
reflected in social datasets Figure from
[Hannak et al. CSCW 2017]
69

Societal biases embedded in behavior can be
amplified by algorithms
Users pick
biased options
Biased
actions are
used as
feedback
System learns to mimic
biased options
System presents
options,
influencing user
choice

Lexical, syntactic, semantic, and structural differences in the
contents generated by users
4. Linking Biases
5. Temporal Biases

Behavioral Biases from Normative Issues
Community norms and societal biases influence observed behavior
and vary across online and offline communities and contexts
What kind of pictures would you share
on Facebook, but not on LinkedIn?
Are individuals comfortable
contradicting popular opinions?
E.g., after singer Prince died, most
SNs showed public mourning. But
not anonymous site PostSecret
The same mechanism can embed
different meanings in different
contexts [Tufekci ICWSM’14]
[the meaning of retweets
or likes] “could range
from affirmation to
denunciation to sarcasm
to approval to disgust”

Privacy concerns affect what content users share, and,
thus, the type of patterns we observe.
Foursquare/Image from [Lindqvist et al. CHI’11]
The awareness of
being observed by
other impacts user
behavior: Privacy
and safety concerns
73

As other media, social
media contains
misinformation and
disinformation
Misinformation is false information,
unintentionally spread
Disinformation is false information,
deliberately spread
Figures from [Kumar et al. 2016]
Hoaxes on Wikipedia: (left) impact as number
of views per day for hoaxes surviving at least 7
days, and (right) time until a hoax gets detected
and flagged.
74

4. Linking Biases
Differences in the attributes of networks obtained from user
connections, interactions, or activity
5. Temporal Biases

Behavior-based and connection-based social links are
different
76
Figure from [Wilson et al. EuroSys’09]

4. Linking Biases
5. Temporal Biases
Differences in populations and behaviors over time

Different demographics can
exhibit different growth rates
across and within social
platforms
TaskRabbit and Fiverr are online freelance
marketplaces.
Figure from [Hannak et al. CSCW 2017]
78

E.g., Change in Features over Time
Introducing a new feature or
changing an existing feature
impacts usage patterns on the
platform.

Biases can creep in at data collection as well
• Common data collection issues:
• Acquisition – Platform restrictions on data gathering
(sampling, APIs, …)
• Querying – Bias from limited expressiveness, keywords, geo-
and other
• Filtering – Outliers, stop words

Design Data Model Application
Best Practices for Bias Avoidance/Mitigation

Consider
team composition
for diversity of thought,
background and
experiences

Understand the task,
stakeholders, and
potential for errors and
harm

Check data sets
Consider data provenance
What is the data intended to
represent?
Verify through qualitative,
experimental, survey and
other methods

Check models and validate results
Why is the model making decision?
What mechanisms would explain
results? Is supporting evidence
consistent?
Twyman’s law: The more unusual
the result, more likely it’s an error

Post-Deployment
Ensure optimization and guardrail metrics
consistent w/responsible practices and avoid
harms
Continual monitoring, including customer
feedback
Have a plan to identify and respond to
failures and harms as they occur

Google's Responsible Fairness Practices
https://ai.google/education/responsible-ai-practices?category=fairness
Summary:
• Design your product using concrete goals for fairness and inclusion.
• Engage with social scientists and other relevant experts.
• Set fairness goals
• Check system for unfair biases.
• Include diverse testers and adversarial/stress testing.
• Consider feedback loops
• Analyze performance.
• Evaluate user experience in real-world scenarios.
• Use representative datasets to train and test your model.

Techniques for Fairness in ML
1. Product Introspection
2. Practical Testing ← Main focus today
3. Training Data
4. Modeling
5. UI/Product Design

Product Introspection (1):
Make Your Key Choices Explicit [Mitchell et al., 2018]
Goals Decision Prediction
Profit from loans Whether to lend Loan will be repaid
Justice, Public safety Whether to detain Crime committed if not detained
• Goals are ideally measurable
• What are your non-goals?
• Which decisions are you not considering?
• What is the relationship between Prediction
and Decision?

Group Exercise!?
1. Form small groups (2-4)
for group exercise
2. Introduce yourselves

Shad(e)vice™ LyricGram™
You have a music website for online
discussion of song lyrics. In response to
seeing increasing levels of online abuse,
you design an automated comment
moderation system.
You run an online sunglasses shop.
You are designing a new feature that
lets users upload photos in order to get
automated suggestions based on their
facial features.

moderation system.
facial features.
Immediate goal: sell sunglasses to users
Long term goal: customer loyalty
Decision: recommend sunglasses?
Input: image

moderation system.
Immediate goal: prevent distress
Long term goal: increase engagement
Decision: block comment?
Input: text
facial features.
Input: image

moderation system.
Input: text
facial features.
Input: image
Q1: What prediction(s) will your ML system(s) make?
Q2: How will you use these predictions to make decisions?

Input: text
Predictions???
Input: image
Predictions???

Input: text
Predictions???
Input: image
Predictions:
- CropFace(image)
- Purchase(face, sunglasses)
- FuturePurchases(face, sunglasses)

Input: text
Predictions:
- Language(text)
- IsAbusive(text, language)
Input: image
Predictions:
- CropFace(image)

Identify Potential Harms
• What are the potential harms?
• Applicants who would have repaid are not
given loans
• Convicts who would not commit a crime
are locked up.
• Are there also longer term harms?
• Applicants are given loans, then go on to
default, harming their credit score
• Are some harms especially bad?

Identify Potential Harms
Types of Harms
• Representational
• Allocative
• Exclusion
e.g., image classification
• Disadvantage
e.g., poor recommendations
• Opportunity Denial
e.g., loans

Seek out Diverse Perspectives
• Fairness Experts
• User Researchers
• Privacy Experts
• Legal
• Social Science Backgrounds
• Diverse Identities
• Gender
• Sexual Orientation
• Race
• Nationality
• Religion

Strengthen your Ethical Thinking Skills
https://aiethics.princeton.edu/case-studies/case-study-pdfs/
https://www.scu.edu/ethics-in-technology-practice/case-studies/
https://ethicalos.org/

Predictions:
- Language(text)
- Abusive(text, language)
User Harms: ???
Most impacted users: ???
Predictions:
- CropFace(image)
User Harms: ???
Most impacted users: ?
Q3: What are potential user harms?
Q4: Which users are most likely to be affected?

Predictions:
- Language(text)
User Harms: ???
Predictions:
- CropFace(image)
User Harms:
- Failure to detect face
- Poor sunglass recommendations
Most impacted users:
- Dark skin
- Cultural minorities

Predictions:
- Language(text)
User Harms:
- Failure to protect from abuse
- Censoring of non-abusive comments
- Targets of hate speech
- People using LGBT terms positively
- Users discussing songs with sexual/violent
lyrics
Predictions:
- CropFace(image)
User Harms:
- Dark skin

Launch with Confidence: Testing for Bias
• How will you know if users are being
harmed?
• How will you know if harms are unfairly
distributed?
• Detailed testing practices are often not
covered in academic papers
• Discussing testing requirements is a
useful focal point for cross-functional
teams

Model Predictions
Evaluate for Inclusion - Confusion Matrix

Model Predictions
Positive Negative

Model Predictions
Positive Negative
● Exists
● Predicted
True Positives
● Doesn’t exist
● Not predicted
True Negatives

Model Predictions
Positive Negative
● Exists
● Predicted
True Positives
● Exists
● Not predicted
False Negatives
● Doesn’t exist
● Predicted
False Positives
● Doesn’t exist
● Not predicted
True Negatives

Efficient Testing for Bias
• Development teams are under multiple
constraints
• Time
• Money
• Human resources
• Access to data
• How can we efficiently test for bias?
• Prioritization
• Strategic testing

Choose your evaluation metrics in light
of acceptable tradeoffs between
False Positives and False Negatives

Privacy in Images
False Positive: Something that doesn’t
need to be blurred gets blurred.
Can be a bummer.
False Negative: Something that
needs to be blurred is not blurred.
Identity theft.
False Positives Might be Better than False Negatives

Spam Filtering
False Negative: Email that is SPAM is
not caught, so you see it in your inbox.
Usually just a bit annoying.
False Positive: Email flagged as SPAM
is removed from your inbox.
If it’s from a friend or loved one, it’s a
loss!
False Negatives Might Be Better than False Positives

Types of Practical
Fairness Testing
1. Targeted Tests
2. Quick Tests
3. Comprehensive Tests
4. Ecologically Valid Tests
5. Adversarial Testing

1. Targeted Tests
Based on prior experience/knowledge
• Computer Vision
⇒ Test for dark skin
• Natural Language Processing
⇒ Test for gender stereotypes
Cf. smoke tests
(non-exhaustive tests that check that
most important functions work)

Targeted Testing of a Gender Classifier
[Joy Buolamwini & Timnit Gebru, 2018]
• Facial recognition
software:
Higher accuracy for
light skinned men
• Error rates for dark
skinned women:
20% - 34%

2. Quick
Tests
• "Cheap"
• Useful throughout product cycle
• Spot check extreme cases
• Low coverage but high informativity
• Need to be designed thoughtfully, e.g.
• World knowledge
• Prior product failures

Quick Tests
for Gender in
Translate

Quick Counterfactual Testing: What If Tool

3. Comprehensive Tests
Include sufficient data for each subgroup
• May include relevant combinations of attributes
• Sometimes synthetic data is appropriate
Particularly important if model will be used in larger
system
Cf. Unit tests
(verify correct outputs for wide range of correct inputs)

Comprehensive Testing of a Toxic Language Detector
[Dixon et al., 2018]

AUC Metrics for
Comprehensive Testing
• Subgroup AUC:
• Subgroup Positives vs
Subgroup Negatives
• "BPSN" AUC:
• Background Positives vs
Subgroup Negatives
• "BNSP" AUC:
• Background Negatives vs
Subgroup Positives

Comprehensive Testing of a Toxicity Detector
https://github.com/conversationai/perspectiveapi/blob/master/model_cards/English/toxicity.md

4. Ecologically Valid
Testing
Data is drawn from a distribution representative of the
deployment distribution
• Goal is NOT to be representative of the training
distribution
• (When appropriate) Condition on labels & certainty
Example usage scenarios :
• Continuous monitoring
• You have historical product usage data
• You can estimate user distribution reasonably well

Ecologically Valid Testing:
Distributions Matter
What is being compared?
Over what data?

Challenges with Ecologically Valid Testing
• Post-deployment distributions may not be known
• Product may not be launched yet!
• Sensitive attributes often not available in deployment
• User distributions may change
• We may want user distributions to change
• e.g., broaden user base

5. Adversarial
Tests
Search for rare but extreme harms
• “Poison needle in haystack”
• Requires knowledge of society
Typical usage scenario:
• Close to launch

Hypothetical Example of Adversarial Testing
• Emoji autosuggest: are happy emoji suggested for sad sentences?
My dog has gone to heaven
Suggest:
Input:
😊

Summary of Practical
Fairness Testing
1. Targeted Tests: domain specific (image, language, etc)
2. Quick Tests: cheap tests throughout dev cycle
3. Comprehensive Tests: thorough
4. Ecologically Valid Tests: real-world data
5. Adversarial Testing: find poison needles

Fairness Testing Practices
are Good ML Practices
• Confidence in your product's fairness
requires fairness testing
• Fairness testing has a role throughout
the product iteration lifecycle
• Contextual concerns should be used to
prioritize fairness testing

Predictions:
- Language(text)
User Harms: ???
- Failure to detect from abuse
- Groups which are targets of hate speech
- Groups writing comments
- confusable for abuse
- in multiple languages
Predictions:
- CropFace(image)
User Harms:
- Dark skin
Q5: How could you test for fairness?

Shad(e)vice™
Predictions:
- CropFace(image)
User Harms:
- Dark skin
• targeted testing:
• faces with darker skin tones
• comprehensive testing:
• diverse gender expression
(male/female/androgynous)
• diverse headwear (caps, hijabs, turbans, ...)
• ecologically valid testing:
• compare sales conversions for users of
different skin tones and locations
• adversarial testing:
• does system recommend sunglasses which
are offensive or have cultural, religious or
political associations?

LyricGram™
Predictions:
- Language(text)
User Harms: ???
- Targets of hate speech
- People using LGBT terms positively
- Users discussing songs with
sexual/violent lyrics
• targeted testing:
• comments about gangster rap lyrics
• comments using LGBT identity terms
• comprehensive testing:
• comments using many identity terms
• comments in many languages
• ecologically valid testing:
• compare error rates for different song
genres
• compare error rates for users in different
locations
• adversarial testing:
• comments with offensive slang which
model may not have seen before

Fairness-aware Data Collection
[Holstein et al., 2019]
• ML literature generally assumes data is fixed
• Often the solution is more and/or better training data
But: need to be Thoughtful!
When might more Data not Help?
• If your data sampling techniques are biased
• Fundamental problems in data quality [Eckhouse et al., 2018]
• What does your data really represent? E.g. crimes vs arrests
• Recall: Product Introspection: How do Predictions relate to Decisions?

Get to Know Your Training Data: Facets Dive

Datasheets for Datasets [Gebru et al., 2018]

Fairness-Aware Data Collection Techniques
1. Address population biases
• Target under-represented (with respect to the user population) groups
2. Address representation issues
• Oversample from minority groups
• Sufficient data from each group may be required to avoid model treating them as
"outliers"
3. Data augmentation: synthesize data for minority groups
• E.g. from observed "he is a doctor" → synthesize "she is a doctor"
4. Fairness-aware active learning
• Collect more data for group with highest error rates

Predictions:
- Language(text)
User Harms:
- Failure to protectfrom abuse
Predictions:
- CropFace(image)
User Harms:
- Dark skin
Q6: How could you improve your training data?

Shad(e)vice™
Predictions:
- CropFace(image)
User Harms:
- Dark skin
• Include more faces from under-
represented groups
• Include historical data from
offline sales (rather than just
online sales)
• Include more recent data (more
predictive of fashion trends)

LyricGram™
Predictions:
- Language(text)
User Harms:
• Include more comments data
related to under-represented
music genres
• Include comments data
unrelated to music
• especially from minority language
variants
• Include more recent data
(discussing new songs)

Sometimes data biases are
unavoidable
Solution: ML Techniques

Practical Concerns with Fair Machine Learning
•Is the training process stable?
•Can we guarantee that fairness policies
will be satisfied?
• Cf. Legal requirements in education,
employment, finance

Machine Learning Techniques: Adversarial Training?
P(Label=1) P(Group)
Negative
Gradient
Fairly well-studied with some
nice theoretical guarantees.
But can be difficult to train.
Features, Label, Group

Machine Learning: Correlation Loss
[Beutel et al., 2018]
Motivation: Overcome training instability with adversarial training
Key idea: include fairness objective in the loss function

Predicted P(Target) distribution
for “Blue” and “Red” examples
(Illustrative Example)
min Loss(Label, Pred)
Pred = P(Label=1)
Machine Learning Techniques: Correlation Loss

+ Abs(Corr(Pred, Group))|Label=0
Pred = P(Label=1)
Predicted P(Target) distribution
for “Blue” and “Red” examples
(Illustrative Example)

● Computed per batch
● Easy to use
● More stable than
adversarial training.
+ Abs(Corr(Pred, Group))|Label=0
Pred = P(Label=1)

Machine Learning: Constrained Optimization
[Cotter et al., 2018]
Motivation: Can we ensure that fairness policies are satisfied?
• Fairness goals are explicitly stated as constraints on predictions, e.g.
• FPR on group 1 <= 0.8 * FPR on group 2
• Machine learner optimizes objective function subject to the
constraints

Model Cards for Model Reporting[Mitchell et al., 2018]

Further Machine Learning Techniques
Many more approaches are linked to from the tutorial website.
https://sites.google.com/corp/view/wsdm19-fairness-tutorial

Fairness in UI/Product Design
1. Robust UIs handle ML failures gracefully
2. UIs should empower users

Predictions:
- Language(text)
User Harms: ???
Predictions:
- CropFace(image)
User Harms:
- Dark skin
Q7: How could UI Help to Mitigate Unfairness?

Shad(e)vice™
Predictions:
- CropFace(image)
User Harms:
- Dark skin
• Be graceful when face detection
fails
• Still give some suggestions!
• Use additional signals rather than
just face
• Let users crop image to face
themselves
• Give users some control in
guiding predictions

LyricGram™
Predictions:
- Language(text)
User Harms:
• UI might distinguish
quoted lyrics from user
comments about those lyrics
• Ability to flag comments
for review
• Ability to appeal
automated censoring decisions

Thanks to
Alex Beutel (Research Scientist, fairness in ML),
Allison Woodruff (UX Research, privacy, fairness and ethics),
Andrew Zaldivar (Developer Advocate, ethics and fairness in AI),
Hallie Benjamin (Senior Strategist, ethics and fairness in ML),
Jamaal Barnes (Program Manager, fairness in ML),
Josh Lovejoy (UX Designer, People and AI Research; now Microsoft),
Margaret Mitchell (Research Scientist, ethics and fairness in AI),
Rebecca White (Program Manager, fairness in ML)
and others!

Fairness Methods in Practice
(Case Studies)

[Play Google Assistant video from desktop]
Key points:
• Think about user harms
• How does your product make people feel?
• Adversarial ("stress") testing for all Google Assistant launches
• People might say racist, sexist, homphobic stuff
• Diverse testers
• Think about expanding who your users could and should be
• Consider the diversity of your users

Google Camera
Key points:
• Check for unconscious bias
• Comprehensive testing:
"make sure this works
for everybody"

This is a “Shirley Card”
Named after a Kodak studio model
named Shirley Page, they were the
primary method for calibrating color
when processing film.
SKIN TONE IN PHOTOGRAPHY
SOURCES
Color film was built for white people. Here's what it did to dark skin. (Vox)
How Kodak's Shirley Cards Set Photography's Skin-Tone Standard, NPR

Until about 1990, virtually all
Shirley Cards featured Caucasian
women.
SOURCES
Colour Balance, Image Technologies, and Cognitive Equity, Roth
How Photography Was Optimized for White Skin Color (Priceonomics)

As a result, photos featuring
people with light skin looked fairly
accurate.
SOURCES
Film Kodachrome
Year 1970
Credit Darren Davis,
Flickr

Photos featuring people with
darker skin, not so much...
SOURCES
Film Kodachrome
Year 1958
Credit Peter Roome,
Flickr

Google Clips
"We created controlled datasets by
sampling subjects from different genders
and skin tones in a balanced manner, while
keeping variables like content type, duration,
and environmental conditions constant. We
then used this dataset to test that our
algorithms had similar performance when
applied to different groups."
https://ai.googleblog.com/2018/05/automat
ic-photography-with-google-clips.html

Geena Davis Inclusion Quotient
[with Geena Davis Institute on Gender in Media]

(Historical)
Gender
Pronouns in
Translate

1. Detect Gender-Neutral Queries
Train a text classifier to detect when a Turkish query is gender-neutral.
• trained on thousands of human-rated Turkish examples

2. Generate Gender-Specific Translations
• Training: Modify training data to add an additional input token specifying the
required gender:
• (<2MALE> O bir doktor, He is a doctor)
• (<2FEMALE> O bir doktor, She is a doctor)
• Deployment: If step (1) predicted query is gender-neutral, add male and female
tokens to query
• O bir doktor -> {<2MALE> O bir doktor, <2FEMALE> O bir doktor}

3. Check for Accuracy
Verify:
1. If the requested feminine translation is feminine.
2. If the requested masculine translation is masculine.
3. If the feminine and masculine translations are exactly equivalent with the
exception of gender-related changes.

Result: Reduced Gender Bias in Translate

Adversarial Testing
for Smart Compose
in Gmail

Responsible AI
and Conversational
Agents

Background: Conversational Agents
• Social bots
• Informational bots
• Task-oriented bots
Eliza, Joseph Weizenbaum (MIT) 1964

Questions to ask during
AI Design
Who is affected by AI?
How might AI cause harms?

Stakeholders: Who might be affected?
1. Humans speaking with the agent
• Emotional harms, misinformation, threaten task completion
2. The agent “owner”
• Harm practices and reputation of the owner
3. Third-party individuals and groups
• People mentioned in conversations!
4. Audiences listening to the conversation
• This may include general public!

Functional Harms
• Misrepresentation of capabilities
• Misinforming user about task status
• Misunderstanding user and doing the
wrong task
• Revealing private information
inappropriately
Yes, I can do that!
Can you order
sushi for me?
Great, one
California roll
please
I don’t
understand.
?!?!

How might AI cause harms? Functional Harms
Functional Harms
wrong task
inappropriately
In just 1 minute
When will my
order arrive?
Where’s my
order?
Arriving in 1
minute
?!?!

Functional Harms
• Misunderstanding user and doing
the wrong task
inappropriately
Ordering egg yolks
Tell me a joke
?!?!

Functional Harms
wrong task
inappropriately
Bob Smith’s CC
number is …
What’s Bob
Smith’s number?
?!?!

Functional Harms
wrong task
inappropriately
These harms are even more problematic when
they systematically occur for some groups of
people but not others

Social Harms: Harms to Individuals
• Inciting/encouraging harmful behavior
• Self/harm, suicide
• Violence or harassment against others
• Discouraging good behavior, e.g., visiting doctors
• Providing wrong information
• Medical, financial, legal advice
• Verbal harassment
• Bullying, sexual harassment

Social Harms: Harms to Communities
• Promoting violence, war, ethnic cleansing, …
• Including promoting related organizations and philosophies
• Engaging in hate speech, disparagement, mocking, …
• Including inadvertent, or Inappropriate imitation (dialect, accent,…)
• Disruption to social processes
• Election disruption, fake news, false disaster response, …

Why is this hard?
Language is ambiguous, complex,
with social context
Examples of complex failures:
• Failure to deflect/terminate
contentious topics
• Refusing to discuss when
disapproval would be better
• Polite agreement with
unrecognized bias
Yes, I can do that!
Let’s talk about
<something evil>

Why is this hard?
with social context
contentious topics
unrecognized bias
Sounds ok.
Men are better at
<whatever> than
women

Why is this hard?
with social context
contentious topics
unrecognized bias
I don’t like talking
about religion
I was bullied at
school because
I’m muslim

Implications for data collection
Common data sources
• Hand-written rules
• Existing conversational data (e.g., social media)
• New online conversations (e.g., from new customer interactions)
Cleaning training data
• For anonymization
• E.g., remove individual names. But keep famous names (fictional characters, celebrities, politicians, …)
• Ensure adheres to social norms
• Not enough to filter individual words: Filter “I hate [X]”, and you’ll miss “I’m not a fan of [X].
• Remember meanings change with context
• Differentiate between bot input and bot output in training data
• Remove offensive text from bot output training
• But don’t remove from bot inputs  allow learning of good responses to bad inputs

Responsible bots: 10 guidelines for
developers of conversational AI
1. Articulate the purpose of your bot
2. Be transparent that you use bots
3. Elevate to a human when needed
4. Design bot to respect cultural norms
5. Ensure bot is reliable (metrics, feedback)
6. Ensure your bot treats people fairly
7. Ensure your bot respects privacy
8. Ensure your bot handles data securely
9. Ensure your bot is accessible
10.Accept responsibility
https://www.microsoft.com/en-us/research/publication/responsible-bots/

Key take-away points
• Many stakeholders affected by conversational agent AIs
• Not only people directly interacting with AI, but also indirectly affected
• Many potential functional, social harms to individuals, communities
• Functional harms exacerbated when systematically biased against groups
• Challenges include complexity and ambiguity of natural language
• Avoiding these harms requires careful consideration across the entire
AI lifecycle.

Acknowledgments
• Chris Brockett, Bill Dolan, Michel Galley, Ece Kamar

Deep Dive: Talent Search
Fairness in AI @ LinkedIn

Create economic opportunity for every
member of the global workforce
LinkedIn’s Vision
Connect the world's professionals to make
them more productive and successful
LinkedIn’s Mission

610M
Members
30M
Companies
20M
Jobs
50K
Skills
90K
Schools
100B+
Updates viewed
LinkedIn Economic Graph

AI @LinkedIn
25 B
ML A/B experiments
per week
data processed
offline per day
2002.15 PB
data processed
nearline per day
2 PB
Scale
graph edges with 1B
nodes
53 B
parameters in ML
models

Guiding Principle:
“Diversity by Design”

“Diversity by Design” in LinkedIn’s Talent Solutions
Insights to
Identify Diverse
Talent Pools
Representative
Talent Search
Results
Diversity
Learning
Curriculum

Inclusive Job Descriptions / Recruiter Outreach

Representative Ranking for Talent Search
S. C. Geyik,
K. Kenthapadi,
Building
Representative Talent
Search at LinkedIn,
LinkedIn engineering
blog post, October’18.

Intuition for Measuring Representativeness
• Ideal: same distribution on gender/age/… for
• Top ranked results and
• Qualified candidates for a search request
• LinkedIn members matching the search criteria
• Same proportion of members with each given attribute value across
both these sets
• “Equal opportunity” definition [Hardt et al, NIPS’16]

Measuring (Lack of) Representativeness
• Skew@k
• (Logarithmic) ratio of the proportion of candidates having a given attribute value
among the top k ranked results to the corresponding proportion among the set of
qualified candidates
• Minimum Discrete Skew: Minimum over all attribute values genders (e.g., the
most underrepresented gender’s skew value).
• Skew = 0 if we have ⌊pq,r,v * k⌋ candidates from value v in the top k results

Reranking Algorithm for Representativeness
• Determine the target proportions within the attribute of interest,
corresponding to a search request
• Compute a fairness-aware ranking of size k

Target Proportions within the Attribute of Interest
• Compute the proportions of the values of the attribute (e.g., gender,
gender-age combination) amongst the set of qualified candidates
• “Qualified candidates” = Set of candidates that match the search query
criteria
• Retrieved by LinkedIn’s Galene search engine
• Target proportions could also be obtained based on legal mandate /
voluntary commitment

Fairness-aware Reranking Algorithm
• Partition the set of potential candidates into different buckets for
each attribute value
• Rank the candidates in each bucket according to the scores assigned
by the machine-learned model
• Merge the ranked lists, balancing the representation requirements
and the selection of highest scored candidates

Validating Our Approach
• Gender Representativeness
• Over 95% of all searches are representative compared to the qualified
population of the search
• Business Metrics
• A/B test over LinkedIn Recruiter users for two weeks
• No significant change in business metrics (e.g., # InMails sent or accepted)
• Ramped to 100% of LinkedIn Recruiter users worldwide

Key takeaway points
• Post-processing approach desirable
• Agnostic to the specifics of each model
• Scalable across different model choices for our application
• Robust to application-specific business logic
• Easier to incorporate as part of existing systems
• Build a stand-alone service or component for post-processing
• No significant modifications to the existing components
• Complementary to efforts to reduce bias from training data & during model
training

Acknowledgements
•Team:
• AI/ML: Sahin Cem Geyik, Stuart Ambler, Krishnaram Kenthapadi
• Application Engineering: Gurwinder Gulati, Chenhui Zhai
• Analytics: Patrick Driscoll, Divyakumar Menghani
• Product: Rachel Kumar
•Acknowledgements (in alphabetical order)
• Deepak Agarwal, Erik Buchanan, Patrick Cheung, Gil Cottle, Nadia
Fawaz, Rob Hallman, Joshua Hartman, Sara Harrington, Heloise Logan,
Stephen Lynch, Lei Ni, Igor Perisic, Ram Swaminathan, Ketan Thakkar,
Janardhanan Vembunarayanan, Hinkmond Wong, Lin Yang, Liang
Zhang, Yani Zhang

Reflections
• Lessons from fairness challenges  Need
“Fairness by Design” approach when building AI
products
• Case studies on fairness-aware ML in practice
• Collaboration/consensus across key stakeholders
(product, legal, PR, engineering, AI, …)

Good ML
Practices Go a
Long Way
• Lots of low hanging fruit in terms of improving
fairness simply by using machine learning best
practices
• Representative data
• Introspection tools
• Visualization tools
• Testing
• Fairness improvements often lead to overall
improvements
• It’s a common misconception that it’s
always a tradeoff

Breadth and
Depth
Required
• Looking End-to-End is critical
• Need to be aware of bias and potential
problems at every stage of product and
ML pipelines (from design, data
gathering, … to deployment and
monitoring)
• Details Matter
• Slight changes in features or labeler criteria
can change the outcome
• Must have experts who understand the
effects of decisions
• Many details are not technical such as how
labelers are hired

The Real
World is What
Matters
• Decisions should be made considering the real
world goals and outcomes
• You must have people involved that understand
these real world effects
• Social scientist, Lawyers, domain experts…
• Hire experts (even ones that don’t code)
• You need different types of testing depending
on the application
• We need more research focused on people,
applications, and real world effects
• A lot of the current research is not that
useful in practice
• We need more social science + machine
learning research

Key Open Problems in
Applied Fairness

Key Open
Problems in
Applied
Fairness
• What if you don’t have the sensitive attributes?
• When should you use what approach? For
example, Equal treatment vs equal outcome?
• How to identify harms?
• Process for framing AI problems: Will the
chosen metrics lead to desired results?
• How to tell if data generation and collection
method is appropriate for a task? (e.g., causal
structure analysis?)
• Processes for mitigating harms and
misbehaviors quickly

Related Tutorials / Resources
• Sara Hajian, Francesco Bonchi, and Carlos Castillo, Algorithmic bias: From discrimination
discovery to fairness-aware data mining, KDD Tutorial, 2016.
• Solon Barocas and Moritz Hardt, Fairness in machine learning, NeurIPS Tutorial, 2017.
• Arvind Narayanan, 21 fairness definitions and their politics, FAT* Tutorial, 2018.
• Sam Corbett-Davies and Sharad Goel, Defining and Designing Fair Algorithms, Tutorials at
EC 2018 and ICML 2018.
• Ben Hutchinson and Margaret Mitchell, Translation Tutorial: A History of Quantitative
Fairness in Testing, FAT* Tutorial, 2019.
• Henriette Cramer, Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miroslav
Dudík, Hanna Wallach, Sravana Reddy, and Jean Garcia-Gathright, Translation Tutorial:
Challenges of incorporating algorithmic fairness into industry practice, FAT* Tutorial,
2019.
• ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*)

Thanks! Questions?
•Tutorial website:
https://sites.google.com/view/wsdm19-fairness-
tutorial
•Feedback most welcome 
• slbird@microsoft.com, benhutch@google.com,
kkenthapadi@linkedin.com, emrek@microsoft.com,
mmitchellai@google.com

Fairness-aware Machine Learning: Practical Challenges and Lessons Learned (WSDM 2019 Tutorial)

More Related Content

What's hot

Similar to Fairness-aware Machine Learning: Practical Challenges and Lessons Learned (WSDM 2019 Tutorial)

More from Krishnaram Kenthapadi

Recently uploaded

Fairness-aware Machine Learning: Practical Challenges and Lessons Learned (WSDM 2019 Tutorial)