Frith Tweedie, Digital Law Leader at EY Law New Zealand, discusses the pros and cons of computer algorithms which are increasingly governing our everyday life through artificial intelligence. She explores how we can ensure transparency and accountability around the use of our personal information in algorithms and how we can avoid biased outputs.
 |
Introduction
The algorithms fundamental to artificial intelligence have huge potential to benefit business, government and society in many ways. But how can we ensure transparency and accountability around the use of our personal information in algorithms and how can we avoid biased outputs?
Google search. Facebook’s News Feed. “You might also enjoy” suggestions on Amazon.
These are all are driven by powerful computer algorithms. And those algorithms – sets of rules that tell a computer what to do and how to do it – are increasingly applied to many different aspects of our lives.
But are we handing over too much power “to the machines”? How can we ensure we understand how our personal information is being used in automated decision making processes and that appropriate decisions are being made? And what is the key to help unlock and understand otherwise opaque algorithmic processes?
HOW AND WHERE ARE ALGORITHMS USED?
Algorithms are one of the foundational elements of computer science – self-contained, step-by-step sets of operations usually performed by a computer. Put simply, an algorithm is a series of instructions, like a recipe or a flowchart. They can be used for calculation, to identify patterns in data and to automate decision making.
Algorithms are ubiquitous in modern life. Every day we use them to give us access to vast amounts of information on the internet with quick, relevant and tailored results. Algorithms and AI can also make help sense of vast amounts of data to drive improved health outcomes, automate tedious or dangerous work, support decision making with speed and accuracy, reduce business costs and optimise business processes.
Gartner forecasts that global business value derived from AI will total US$1.2 trillion in 2018, an increase of 70 per cent from 2017. AI-derived business value is forecast to reach $3.9 trillion in 2022, according to figures released in April by the analyst firm.
WHEN ALGORITHMS GO BAD
Despite the obvious benefits, questions are increasingly being asked about the far-reaching impact of algorithms on our lives. More decisions using our personal information are influenced by algorithms, including what jobs we are offered, whether we get a mortgage or credit card, what medical treatment is recommended and whether the police are likely to regard us as potential criminals.
Is our increasing dependence on algorithms making us increasingly vulnerable? How many of us really understand how or why certain decisions are made? How can we know when to correct errors and contest decisions if we don’t know why an algorithm produced the result it did? And as reliance on AI – and in particular machine learning – grows, how can we be sure that the right decisions are being made based on our personal information?
A range of studies and examples indicate that these concerns are not unfounded and that the following risks must be addressed.
1. Bias and discrimination
-
-
- An investigation into machine bias found that the algorithms used by criminal justice systems across the United States to predict future criminals were biased against black people.
- A Carnegie Mellon University study showed that Google ads for high-paying jobs were shown more often to men than to women.
-
2. Poor quality data
-
-
- “Garbage in garbage out” or “GIGO” is a term well known in computer science. A system is only as good as the data it learns from. Microsoft issued a public apology after its Tay chatbot turned into a holocaust-denying racist following corruption by Twitter trolls. And Google’s problem of search keywords like “gorilla” and “chimp” returning images of African-Americans still isn’t fixed.
-
3. Inadequate training of AI systems leading to incorrect outputs
-
-
- A report found that IBM Watson had made multiple “unsafe and incorrect treatment recommendations” to cancer doctors as a result of incomplete training that used synthetic data instead of patient data taken from real cancer cases. In one reported instance, a doctor at Jupiter Medical Center in Florida using IBM Watson for Oncology reportedly went as far as to call the system “a piece of sh*t”.
-
4. Security threats
-
-
- Attackers may exploit vulnerabilities in AI systems.
- The “Malicious use of artificial intelligence” report notes that “As AI capabilities become more powerful and widespread, we expect the growing use of AI systems to lead to the expansion of existing threats, the introduction of new threats and a change to the typical character of threats” including cybercrime, political disruption and even physical attacks.
-
5. Proprietary technologies
Organisations may be reluctant to share the internal workings of their algorithms for fear of disclosing trade secrets and sources of competitive advantage. That makes it hard to monitor and challenge decisions and other outputs, as illustrated when a US appeal court was asked to decide whether sentencing judges should be able to take into account the results of a set of algorithms designed to predict an individual’s risk of recidivism. The accused argued that the proprietary and confidential nature of the algorithms meant that the details of how it worked, and what information went into it, were kept secret from both the sentencing judge and the accused.
Regulation
General Data Protection Regulation
The General Data Protection Regulation (GDPR) addresses algorithmic accountability through its focus on “automated individual decision-making” (making a decision solely by automated means without any human involvement) and “profiling” (automated processing of personal data to evaluate certain things about an individual).
Article 21 gives people the right to object to automated processing of their personal data, including profiling. Article 22 states that people will have the right not to be subject to a decision based solely on automated processing or profiling.
Data Protection Impact Assessments (DPIAs) are likely to be necessary for high-risk processing activities like automated processing. DPIAs are designed to demonstrate that risks have been identified and assessed and consideration has been given to how they will be addressed.
The GDPR also requires individuals to be given specific information about the processing, what steps are being taken to prevent errors, bias and discrimination and how individuals can challenge and request a review of the decision. That includes providing “meaningful information” about the logic involved in the decision-making process, as well as the significance and the envisaged consequences for the individual.
While the GDPR’s approach is a great starting point for achieving algorithmic transparency, it won’t necessarily assist in all contexts. For example, some predictive policing tools do not necessarily profile on an individual basis, focusing instead on locations and statistics to try to understand and predict crime trends across geographical areas. Such tools can have a discriminatory impact even without relying on personal information.
Privacy Act 1993
The Privacy Act does not specifically address algorithmic transparency but its core principles can help to encourage greater transparency.
Under New Zealand law, individuals have the right to know what personal information agencies are collecting about them and why, and to access and correct information held about them. Reasonable steps must be taken to check the accuracy of information before it is used and the information must be appropriately stored, protected and disposed of once no longer needed.
But in a fast-changing world, is local privacy legislation keeping up? New Zealand has an opportunity to help address new risks arising out of emerging technology like AI in its Privacy Bill. Unfortunately the current draft does not follow the GDPR’s lead by introducing mandatory Privacy by Design or DPIAs, much less restrictions around automated decision making and profiling. However, the Privacy Commissioner’s submissions on the Bill recommended it include additional provisions to address automated decision-making and require algorithmic transparency in appropriate cases.
A public and private sector issue
Government review
It will be interesting to see whether the Privacy Commissioner’s recommendations are implemented, particularly in light of the government project to review public sector algorithm use that is currently under way. Launched in May 2018, the objective of the project is to ensure New Zealanders are informed and have confidence in how the government uses algorithms. The initial focus will be on operational algorithms that inform decisions directly impacting individuals or groups. Good practice, and opportunities to support agencies that use algorithms in decision-making will also be looked at.
The Principles for Safe and Effective use of Data and Analytics will underpin the analysis. A report with the findings of the first phase of the review is expected to be published in shortly.
Private sector challenges
Commercial, operational and reputation risks can arise from bad data and poorly trained algorithms, often aggravated by inadequate data governance. Poor credit risk decisions, lost revenue, failed marketing campaigns and customer churn are all real challenges in a range of industries.
Questions persist as to whether New Zealand boards are equipped to understand and oversee the use of AI technologies. Both boards and senior management need to be ready to ask questions about how to optimise potential AI benefits while also identifying and managing potential risks.
WHAT CAN BE DONE?
Algorithmic transparency
The principle of “algorithmic transparency” requires creators of AI to report and justify algorithmic decision making and to mitigate any negative social impacts or potential harms.
But algorithmic transparency can be challenging to implement because of commercial secrecy and the often impenetrable “black box” nature of forms of AI like machine learning and deep learning.
-
- “Machine learning” is a technique that allows algorithms to extract correlations from data with minimal supervision. Machine learning is designed to mimic our own decision-making – if you could get access to those algorithms, it would be possible to understand their reasoning.
- “Deep Learning” is a subset of machine learning that is much harder for humans to decipher, relying on “deep neural networks”, or computer systems modelled on the human brain and nervous system with multiple layers. Even engineers who build systems that seem relatively simple on the surface, such as apps and websites that use Deep Learning to serve ads or recommend songs, cannot always fully explain their behaviour, since you cannot just look inside a deep neural network to see how it works.
So if computer engineers cannot explain the behaviour of their own creations, what hope is there for the rest of us? How comfortable should we be leaving key public and private decisions in the hands of a limited number of data scientists? The risk is that these “black boxes” will operate outside the scope of meaningful scrutiny and accountability. If it’s not clear how decisions are made, then how can they be monitored and verified and how can we predict when failures might occur?
While there is no commonly agreed answer to these questions, data governance, algorithmic impact assessments and ongoing monitoring measures may help address some of those issues.
Data governance: As privacy professionals will be well aware, having a full picture of your data enables better management of that data and any associated risks. Comprehensive data strategies that focus on technology, data availability and acquisition, data labelling, and data governance will help manage GIGO (Garbage In, Garbage Out) and data quality risks. In addition, a set of tailored data and AI ethical principles can help organisations be clear on key privacy, data ethics and other considerations relevant to their work.
Privacy by DesignandAlgorithmic Impact Assessments: Applying principles familiar from a Privacy by Design approach will help encourage the development of processes that address the algorithmic life cycle from data selection, to algorithm design, integration, and live production use. Privacy, security and bias problems will be much easier to address if identified at an early stage.
Much like Privacy Impact Assessments, Algorithmic Impact Assessments (AIAs) involve a self-assessment of existing and proposed automated decision systems, evaluating potential impacts on things like fairness, justice, bias, or other concerns across affected communities.
The AI Now Institute issued a report earlier this year on AIAs that aims to provide a practical framework for the public sector. That framework is designed to “support affected communities and stakeholders as they seek to assess the claims made about these systems, and to determine where – or if – their use is acceptable”.
The report acknowledges that implementing AIAs will not solve all of the problems raised by automated decision systems. But it argues AIAs provide an important mechanism to inform the public and to engage policymakers and researchers in productive conversation.
Monitoring and testing: Organisations that establish processes for assessing and overseeing algorithm data inputs, workings, and outputs, including objective reviews of algorithms by internal and external parties, will be far better placed to identify and manage issues. Ethics boards can use the organisation’s previously defined ethical principles to review algorithmic-based projects and processes.
CONCLUSION
The proliferation of powerful algorithms impacting all aspects of our lives will only increase. While there’s no doubt this will introduce a wide range of potential efficiencies and benefits, we also need to be careful not to overlook the potential risks, particularly to parts of society that have traditionally been marginalised.
An increased awareness of algorithmic risks among researchers, businesses, governments and consumers is required, as well as tools to challenge automated decisions where appropriate. Legislation can assist, as can structured, auditable approaches to identifying and managing risk such as Algorithmic Impact Assessments.
But we also need to recognise that AI is still a nascent and immature field. Mistakes will be made, so we must ensure we learn from those mistakes. And we cannot forget that ultimately algorithms, machine learning and the rest are simply tools programmed by humans. It is our responsibility to use them to enhance humanity, not to amplify existing problems.
Frith Tweedie has more than 16 years’ experience advising on privacy, technology, IP, online/e-commerce, consumer protection and entertainment law issues. She has extensive experience advising on privacy issues arising in a digital context, including the privacy implications of data analytics, data sharing, cloud storage and data monetisation initiatives.
Prior to joining EY Law to lead the Digital Law team, she was responsible for establishing a large privacy programme at a major telecommunications company in response to the European General Data Protection Regulation (“GDPR”). Frith also worked closely with a major bank’s digital teams on numerous data-driven initiatives requiring extensive privacy input. In her current role, Frith works closely with EY management consultants advising on and implementing a range of technology and digital solutions. Contact Frith at frith.tweedie@nz.ey.com
You can also connect with EY New Zealand via Facebook, Twitter and LinkedIn
