NSW Police targeting shows the ethical dangers of secret algorithms

Children as young as 10 have been added to the NSW Police Suspect Targeting Management Plan (STMP), singled out for stop-and-search and move-on directions whenever police encounter them. But how and why that happens is a secret.

This lack of transparency means individuals are left wondering whether their race or family history has been a factor.

It's a reasonable fear, given the evidence that algorithms can learn gender and racial biases from our own biased language, or that algorithms developed for caucasian skin have trouble with people of colour.

"[Algorithms] are, in fact, reflections of how the world works, and as such they are riddled with human biases and flawed logics. We have seen algorithms that were racist, sexist, heteronormative, violent, competitive, law breaking, and aggressively American -- and those are just the ones we know about," said professor Genevieve Bell, in the last of her four Boyer Lectures earlier this month.

STMP could well be encoding existing policing biases.

According to the Youth Justice Coalition (YJC), the data they've managed to obtain shows that STMP "disproportionately targets young people, particularly Aboriginal and Torres Strait Islander people".

"Young people experience the STMP as a pattern of oppressive, unjust policing," wrote YJC researchers Dr Vicki Sentas and Camilla Pandolfini, in their report Policing Young People in NSW published on Wednesday.

"[They] experience a pattern of constant harassment by police including use of stop and search powers by police, presentations at the young person's home akin to bail checks (although the young person may not be on bail at the time), and move on directions," the researchers wrote.

"In some cases, the STMP is being used by police as a substitute for having 'reasonable grounds to suspect' the young person has committed, or is about to commit, an offence."

One case study from the report is about Dean, a 16-year-old male. His lawyer describes his offending as "minor and not extensive", relating mostly to graffiti, plus a caution for cannabis possession and for a minor trespass offence.

Dean has been stopped and searched 23 times in 10 months. Police records show justifications including "persons of interest who wear Nautica clothing are 'known to commit criminal damage offences' (graffiti); that young people who get on the last carriage of a train and wear Nautica are known to commit criminal damage; and that Dean was with a group of young people."

Young people have found themselves subject to an STMP even though they have only minor, non-violent prior convictions -- or even no prior convictions at all, merely extensive contact with police, through family connections, for example.

In the 10 police Local Area Commands (LACs) for which the researchers obtained data, 44 percent of those on the STMP were identified as Aboriginal. The highest proportion was in Redfern LAC, where 60 percent of those targeted were Aboriginal, even though "only 2 percent of the Redfern population is Aboriginal".

The report documents an incident when a 13-year-old Aboriginal boy was approached by police when he was waiting for a bus. "They asked for Bill's name and then said words to the effect of 'We remember you from [another locality], you're a thieving little dog'," the researchers wrote. Not an uncommon incident, it would seem.

While issues of police behaviour are beyond ZDNet's remit, key questions remain.

Has STMP simply recreated existing biases, because it's based on data from policing processes which may already have implicit biases? Is the model used to predict the potential for future criminal acts valid? Does putting someone on the STMP run the risk of increasing the likelihood of unfavourable police encounters?

And the key ones ...

Why is this process, not only the algorithms but even just the broad conceptual model, a secret? Why are individuals not told whether they're on the STMP or not? Why can't they know how to get off the list? Why is there no right of appeal?

The YJC report makes it clear that NSW Police sees STMP as a magic justification box. Their comments this week reinforce that impression.

"The Suspect Target Management Plan (STMP) is a framework which targets recidivist criminal offenders, regardless of age, to prevent them from committing crimes and disrupt their capability to commit crime. A thorough risk management framework is used to ensure the NSW Police Force is targeting the right people at the right times to reduce violence and crime in the community," they told pop culture site Junkee.

"On all occasions, the STMP undergoes a quality assurance process by a senior police officer to ensure the validity of the process. While deliberately engaged by police, STMP nominees are treated with respect and tolerance, but they are reminded that the community will not tolerate criminal behaviour."

Globally, though, there's a rapidly-increasing focus on the ethics and morality of our data-driven world.

"In the EU, they have recently rebooted the European Group on Ethics in Science and New Technologies, and there have been ongoing debates around data privacy and ownership, the right to be forgotten and algorithmic transparency," Bell said.

The EU's General Data Protection Regulation (GDRP), which comes into effect on 25 May 2018, includes what may well amount to a right to explanation -- that is to say, the right to know how an automated system has reached its decision -- although the actual strength of that right has yet to be tested in court.

Meanwhile, the YJC has called for the the STMP policy and operational arrangements to be made public; for individuals to be formally notified when they're placed under the STMP, and why; to provide data to the highly-regarded NSW Bureau of Crime Statistics and Research (BOCSAR); and to have BOCSAR evaluate whether STMP is in fact reducing youth crime.

"[Our] research has been limited by the lack of publicly available information on the STMP and the absence, to date, of scrutiny and oversight of the program," wrote the YJC researchers.

That needs to change.

Related Coverage

Walmart turns to robots to help solve out-of-stock issues

The world's largest retailer has had a well-documented struggle with empty shelves in its stores.

Apple COO: Smartphone is a 'major platform' for future of AI (TechRepublic)

Apple's Jeff Williams spoke on the impact of artificial intelligence, and how it could change major industries as well as the daily lives of consumers.

Intel aims to be inside your artificial intelligence stack

Intel's master plan is to do for AI what it has done for server processors. The goal is to create a general purpose AI stack so enterprises can broadly adopt machine learning.

Machine learning-as-a-service the way of the future: ServiceNow

With "as-a-service" offerings, all organisations will need to do is ensure their data is rich and contextual enough to reap the rewards of machine learning, ServiceNow has said.

Cray supercomputing comes to Microsoft Azure to boost AI workloads in the cloud (TechRepublic)

The strategic alliance will allow Microsoft Azure users to leverage Cray supercomputing systems to run advanced analytics, modeling workloads, and more at scale.