In my colleague Katie Nagy de Nagybaczon's recent post, she highlighted that inherent bias from human data can cause bias in AI systems, and touched on the idea that lacking transparency with AI systems may hamper mass adoption.
By coincidence, in two days I’ve had two discussions with data scientists on these very points. As AI proliferates, I share three related issues which help to explain why understanding bias is just the first of many AI themes which those providing professional or data-driven services will need to get to grips with.
Firstly, when speaking to those actively developing AI-focused solutions, it becomes clear that companies in this space are in an AI arms race -- the impulse to innovate quickly and to be seen gaining momentum is strong. This means that necessarily slower conversations (those that may be less inspiring, but still need to be had) can take a back seat. These conversations can include those around compliance, safeguarding, or record-keeping. Professional advisors will increasingly be asked to give an opinion on suitability of new AI services, so will need to develop frameworks which can properly evaluate offerings in ways which balance client pressures to innovate against the need to manage risk and efficiency.
Secondly, we will need to get better at keeping track of a different type of ever-evolving system. Many software solutions, including tools used in the trade mark world benefit from regular updates, delivering impressive accuracy improvements. However, with the drive to improve, the data feeding the AI system changes frequently, in turn changing the output. Whilst frequent improvements are welcome, this process can make it difficult to review historic results in their context. In the absence of clear explanations around how computers produce results it can also be challenging (or sometimes even impossible) to consider the appropriateness, or impartiality, of a particular strategy. As Google puts it - we don't really understand how neural networks, for example, work:
"...neural networks can learn how to recognize images far more accurately than any program we directly write, but we don’t really know how exactly they decide whether a dog in a picture is a Retriever, a Beagle, or a German Shepherd."
This means a human cannot always to check the efficacy, bias, or general appropriateness of the methodologies adopted by AI systems. It will be difficult to evaluate individual cases and to learn from mistakes if we cannot work out if a failure was one of the system or it’s operator. Identifying how a system was working several versions ago (or hundreds of versions ago) is likely to be a further challenge. Since much legal work (and most legal actions) involve managing and apportioning risk, this lack of transparency is a likely source of concern for general counsel (both in-house and at law firms, alike).
Thirdly, using appropriately "cleaned" input data is important to ensure accuracy (and indeed to counteract bias, as Katie's article states). Data scientists I’ve spoken to have certainly long been cognisant of the issue of bias within AI systems, yet it still pervades even market-leaders' efforts. This issue is exemplified by Amazon's recently shut down recruitment tool which applied machine learning to Amazon's historic recruitment data and consequently took on the historic bias towards men which has pervaded Amazon's historic recruitment practices.
The clean-up task is far more difficult than it sounds. We often do not always understand our own basic biases, let alone those which may be specific to our respective industries, so data scientists will need to augment their teams with experts in a range of fields to properly assess and then to clean input data.
Once that data is analysed, the next issue is that we do not always understand the way AI systems are interpreting that data. This phenomenon is explained in one amusing anecdote from a data scientist:
One particular AI system was asked to filter photographs to only those with open curtains . The system worked flawlessly on test data. However, once rolled out, its accuracy was poor. On review, scientists identified that instead of basing its reasoning on the state of the windows in each room, it was instead simply selecting images with beds in, because all rooms with closed curtains in the learning data set were bedrooms.
Furthermore, many next-generation AI software applications promise to learn from users' interactions with output data, supposedly allowing real-time feedback to deliver better tailored results to the individual user. However, if we return to the example of Amazon's AI nightmare, surely allowing humans to override systematically de-biased results (and allowing human choices to influence future results) risks re-introducing certain biases via the backdoor?
AI technologies certainly present exciting opportunities and will revolutionise certain types of data analysis. But the very same features which make them revolutionary also open them up to misunderstanding and risk. A different set of skills and a team with the right mix of competencies will be needed to fully utilise, understand, and manage these innovations. Thus, with great innovation, comes great (user) responsibility.
The challenge for many AI systems is that they are a "black box", with only those that have developed the algorithm understanding how they work. How would we know whether the unconscious biases of the developers have been (unintentionally) built into the systems they design? Calls for transparency in machine learning are increasing but research in this area is still in the early stages.