Over 2,000 academic researchers in the AI/ML field have signed this open letter opposing Springer’s decision to include a predictive policing paper in a book. I think this is a good sign. There’s power in numbers, and if the authority with which these researchers speak isn’t good enough, then maybe their sheer mass will be.
What was most interesting to me were the assertions made in the open letter. Here are the main points, in their words.
Data generated by the criminal justice system cannot be used to “identify criminals” or predict criminal behavior. Ever.
This is their first assertion. Basically, the output of a human system should not serve as the training data for a machine learning system. Not only will it simply echo the prejudices in the human-built criminal justice system, it will amplify its power while distorting the inherent false correlations. This is because “real-world performance is almost always lower than advertised test performance for a variety of reasons.”
Technical measures of “fairness” distract from fundamental issues regarding an algorithm’s validity.
Using quantitative measures of fairness, while attractive, is not a viable solution to the problem. It only sprinkles some fairy dust of false legitimacy while totally failing to address the underlying concerns. This, in my opinion, is a more important assertion to communicate to the general public. The cargo cult of “data-driven” is tough to root out, especially when it’s a mode of thought that has proven to be successful in many fields. In some cases, data simply does not tell the whole story. In the authors’ own words: “This self-critique must be integrated as a core design parameter, not a last-minute patch. The field of machine learning is in dire need of a critical reflexive practice.”
Conclusion: Crime-prediction technology reproduces injustices and causes real harm
I would even expand this to say that any technology that has consequences on an individual’s life, when applied at scale, has the potential to reproduce injustice and cause real harm. When technology is wielded by something as consequential as a government, it requires deep and careful consideration.
In addition the clear and concise letter, the extensive set of footnotes is an absolute treasure trove of references and reading materials in this newly bustling field of AI ethics. It contains references not just to books and academic papers, but also blog posts and other material that will keep me occupied for days. I plan to highlight the best thoughts and ideas I come across.
For more info, look at this Verge piece about the open letter.