Incident 78: Meet the Secret Algorithm That's Keeping Students Out of College

Description: In response to the Covid-19 pandemic, the International Baccalaureate final exams were replaced by a calculated score, prompting complaints of unfairness from teachers and students.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History

Entities

View all entities
Alleged: International Baccalaurette developed and deployed an AI system, which harmed International Baccalaureate students.

Incident Stats

Incident ID
78
Report Count
1
Incident Date
2020-07-06
Editors
Sean McGregor, Khoa Lam
Applied Taxonomies
CSETv0, CSETv1, GMF, MIT

CSETv1 Taxonomy Classifications

Taxonomy Details

Incident Number

78

AI Tangible Harm Level Notes

The harm was caused by a statistical algorithm that did not meet our definition of AI. Harm did occur, but it was intangible (opportunity loss) instead of tangible.

CSETv0 Taxonomy Classifications

Taxonomy Details

Problem Nature

Unknown/unclear

Physical System

Software only

Level of Autonomy

Low

Nature of End User

Expert

Public Sector Deployment

No

Data Inputs

Prior test and exam grades, school attended

MIT Taxonomy Classifications

Machine-Classified
Taxonomy Details

Risk Subdomain

7.3. Lack of capability or robustness

Risk Domain

  1. AI system safety, failures, and limitations

Entity

AI

Timing

Post-deployment

Intent

Unintentional

Incident Reports

Reports Timeline

Incident OccurrenceMeet the Secret Algorithm That's Keeping Students Out of College
wired.com · 2020

EIGHTEEN-YEAR-OLD ANAHITA NAGPAL fears her plans to start training this fall to be a doctor have been derailed by a statistical model.

Nagpal, who lives in Göttingen, Germany, had been offered a premed place and scholarship at NYU. Her acce…

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents

Racist AI behaviour is not a new problem

· 4 reports
NY City School Teacher Evaluation Algorithm Contested

NY City School Teacher Evaluation Algorithm Contested

· 7 reports
Kronos Scheduling Algorithm Allegedly Caused Financial Issues for Starbucks Employees

Kronos Scheduling Algorithm Allegedly Caused Financial Issues for Starbucks Employees

· 10 reports
Previous IncidentNext Incident