Lessons from the Flight Deck

How the Use of Automation in Aviation can Inform Artificial Intelligence in Financial Services

📅 August 13, 2024

Defining Key Terms

While the definitions of “artificial intelligence” and “automation” have yet to achieve global consistency – and in some situations the use of these words is blurred – for this article we will use the definitions below. We also define “machine learning,” which is closely related and particularly relevant in the financial crime compliance setting.

Artificial intelligence (AI) is a machine-based system that can “for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments,” according to the National Artificial Intelligence Act 2020. It aims to replicate humans on a cognitive level, enabling machines to perform tasks that usually require human intelligence such as understanding language, recognizing patterns, solving problems and – most importantly – learning from experience. AI can adapt and learn over time.
Machine learning (ML) is one type of AI. It uses algorithms and models to automatically learn insights and recognize patterns from data, applying that learning to make better decisions. Rather than explicit instructions, it relies on patterns and inference drawn from large datasets.
Automation executes predefined tasks exactly and reliably. It is well-suited to repetitive actions that are tedious and error-prone for humans. Automation does not learn or adapt unless it is re-programmed.

These three concepts are distinct but interrelated. Many businesses have implemented automation and are well-progressed with machine learning, although few have fully reached the level of artificial intelligence. Although less sophisticated, the successes and failures of automation are instructive on the business benefits, implications, and regulatory intentions as companies progress with the implementation of AI.

Successes and Failures

“Automated systems have been successfully used for many years, and have contributed significantly to improvements in safety, operational efficiency, and precise flight path management… [However] pilots sometimes over-rely on automated systems – in effect, delegating authority to those systems…”

– Federal Aviation Administration Flight Deck Automation Working Group report on Operational Use of Flight Management Systems

As we seek to improve efficiency and reliability throughout our life and work, automation – and its advancement into machine learning and artificial intelligence – provides significant benefits. For example, machines are:

Better suited than humans to repetitive tasks
Have the capability to analyze and identify patterns and anomalies in vast datasets
Can maintain and apply complex rulesets with great accuracy

However, we as humans remain responsible for the technology we create, and for its decisions.

In the aviation industry, failures to adequately manage automation can have life-critical impacts. Examples are the fatal outcomes on flights Air France 447, Lion Air 610, and Ethiopian Airlines 302, where automation produced outcomes unexpected by the human air crew, ultimately leading to a loss of the aircraft and the lives of those onboard.

In the financial services environment, the effects of flawed automation or AI may have significant, though more subtle, implications. The lessons learned on the flight deck can inform the implementation of advanced technologies in banking too.

“Other factors that affect the pilots’ decisions include the high reliability of the systems [resulting in] insufficient cross verification… [and] operational policies that direct the pilots to use automated systems over manual flying. Such policies are said to be preferred by some operators because the automated systems can perform more precisely while reducing pilot workload.”

– Federal Aviation Administration Flight Deck Automation Working Group report on Operational Use of Flight Management Systems

Perhaps counter-intuitively, the reliability and accuracy of machines can discourage humans from scrutinizing or challenging their outputs. We are then more likely to experience “automation surprises”, such as technology behaving in ways we do not understand or expect. The erosion of human situational awareness may lie hidden until a significant technological failure occurs.

Responsible principles for the use of artificial intelligence have been created in several jurisdictions as well as globally. For example, the United Nations Educational, Scientific, and Cultural Organization (UNESCO) published its Recommendations on the Ethics of Artificial Intelligence in 2022. The Recommendations recognize the “profound and dynamic positive and negative impacts of artificial intelligence (AI) on societies, environment, ecosystems and human lives… [but that they] can deepen existing divides and inequalities in the world, within and between countries, and that justice, trust and fairness must be upheld…”

Financial services regulators have made it clear that they consider management of AI-related risks to be a board level responsibility and that they will take action where companies do not meet their requirements.

Regulatory Responses in the United States and European Union

In the U.S., the Consumer Financial Protection Bureau issued guidance in 2023 about the legal requirements that lenders must meet when using artificial intelligence. “Lenders must use specific and accurate reasons when taking adverse actions against consumers… This requirement remains even if those companies use complex algorithms and black-box credit models that make it difficult to identify those reasons.”

U.S. Treasury has also explored opportunities and risks related to the use of AI, including concerns related to bias and discrimination in financial services and challenges with “explainability”. Explainability refers to the ability of human users to understand a model’s outputs and decisions, or how the model establishes relationships based on its inputs. Proposed amendments to the U.S. AML Act would require FinCEN to issue rules specifying the standards applicable for testing methods applicable to innovative approaches such as machine learning.

In the European Union, the Artificial Intelligence Act was approved by the European Council on May 21^st, 2024, and it comes into force later this year. The Act bans some types of AI systems as creating an “unacceptable risk”, such as government-run social scoring. High-risk applications are subject to specific legal requirements. These include those used for automated processing of personal data to assess aspects of a person’s life such as their economic situation. It is likely that many applications of AI within banks, such as for use in credit scoring, will meet the definition of high risk and require additional controls.

In the United Kingdom, the Financial Conduct Authority took action against Amigo Loans Ltd for poorly implemented automation. While automation is not as advanced as AI, the action taken indicates regulatory intent and provides lessons for financial firms.

Amigo Loans: Automation Without Sufficient Human Oversight

Amigo Loans Ltd (Amigo), a UK company, received a penalty from the UK Financial Conduct Authority was assessed as being subject to a fine of $94 million (£72.9 million) for failing to properly assess the affordability of its lending by relying heavily on a complex IT system with a high degree of automation. Amigo provided guarantor loans aimed at consumers who may have been unable to access finance from traditional lenders due to their circumstances or credit history. Both borrowers and guarantors needed to pass Amigo’s affordability checks for a loan to be approved.

The FCA found that:

Amigo’s assessment of whether a customer could afford to borrow relied heavily on a complex IT system with a high degree of automation
Design issues and insufficient controls meant the system processed loans which were potentially unaffordable for the customer
Although the system raised manual flags for review in some circumstances, staff did not sufficiently consider or question information provided by the customer
The company had not adequately considered regulatory requirements, or internal and external review findings identifying weaknesses in its approach to affordability and creditworthiness.

In addition to issues with its use of technology, Amigo had failed to maintain adequate records of its historic business processes and had deleted email accounts of former staff, both of which hampered the FCA’s investigation.

The FCA held back on imposing the $94 million fine because the company demonstrated it would cause serious financial hardship and would have threatened Amigo’s ability to pay redress to its customers.

Data and AI

“At the heart of AI is data. The more complete the data on which an AI is trained, the more valuable will be the output. But just like human beings, if an AI has nothing on which to base its predictions, its output will be worthless.” – Forbes Technology Council

In the example above, Amigo’s technology solution used automation – a set of pre-defined rules – rather than having any AI capability to learn itself. As AI, including machine learning, introduces more “distance” between the human-directed inputs and the AI outputs, being able to demonstrate explainability of decisions is even more critical to meeting ethical and regulatory obligations.

Artificial intelligence offers advantages: machine learning algorithms can be designed to focus only on the variables that improve predictive accuracy rather than on the subjective factors influencing human decision-making.

However, AI algorithms and training datasets need to be carefully constructed to achieve these potential benefits. For example, if demographic categories are missing or under-represented in training datasets then models can fail to scale correctly and produce more errors. A well-known example of this is AI developed in 2014 by Amazon to rate job applicants. The technology was found to favor male applicants for software and technical positions: the models had been trained to score applications on patterns from the previous 10 years – almost all of them from men, reflecting a gender-skewed professional field. The initiative was cancelled in 2017.

One of the benefits of machine learning is that it can identify correlations and patterns in datasets that may not be detected by human analysts, such as indicators of fraud in customer and transaction data. However, datasets and algorithms should consider both “positive” and “negative” weightings to produce accurate results.

Case Study

For example, when a customer opens a new account at a bank, a Social Security Number (SSN) without credit history may be given a “positive weighting” as an indicator of fraud, meaning an investigator would consider it suspicious.

However, the time the person has been present in the United States should also be considered. If the time a customer has resided in the country is short, say a few weeks, this should contribute a “negative weighting” to an assessment of suspicious activity. This may indicate a person who has recently relocated and has received their SSN but not yet built up a credit history. Data on length of presence is available to the bank in the documents it collects during onboarding, such as identification documents (including their visa and its validity date) and address history (if the customer has previous held accounts in a different country).

Financial institutions have a responsibility to supervise their AI to ensure it is producing accurate results, particularly for less common (under-represented) client groups.

Financial institutions who use automation and AI responsibly will benefit from increased efficiency and accuracy. Those who do not will find themselves subject to regulatory scrutiny and investigation.

Actions a Compliance Officer Can Take

While the responsibility for implementation AI and automation extends from the board level of a business, in relation to financial crime compliance specifically, there are several actions a Chief Compliance Officer can take to fulfil their responsibilities when using automation and AI in compliance:

Use artificial intelligence types and solutions which are suitable for their intended use, taking into consideration whether automation, machine learning, or artificial intelligence (such as supervised or unsupervised) is most appropriate
Ensure datasets used to train AI models are fit for purpose, including being complete, accurate, relevant, and representative
Maintain oversight of the operation of AI systems, during testing and ongoing operation to ensure they are working appropriately, and report and act on any issues identified
Maintain records on the operation and monitoring of the system to ensure the firm can respond to questions about explainability from regulators or consumers
Ensure teams involved have sufficient understanding of how to use AI responsibly, including developers, testers, end users, and assurance functions
Ensure processes include human escalation points and reviews to ensure consumer protection and explainability requirements are being met
Monitor regulation and regulatory enforcement actions to ensure compliance, identify lessons learned, and apply these within their department
Participate in firm-wide AI initiatives and working groups to ensure the use of AI in compliance aligns with corporate requirements

Upcoming Webinar

Join the Institute for Financial Integrity for an in-depth webinar designed to equip financial institutions and professionals with essential strategies for leveraging Artificial Intelligence (AI) in compliance practices.

The integration of AI into financial compliance is becoming more crucial for managing the escalating complexity and sophistication of financial crimes. As the financial industry continues to evolve, AI presents opportunities to enhance financial crime compliance, reduce operational costs, and improve the accuracy and effectiveness of compliance programs.

Led by industry experts, this webinar will explore how AI technologies can be integrated into traditional compliance frameworks, address the unique challenges presented by the digital transformation of financial services, and prepare financial institutions for the future of compliance.

Explore the Agenda and Register

Recommended Blogs

Elevating Defenses

August 22, 2024

Dive into the cutting-edge strategies that financial institutions are implementing to combat the evolving threat of AI-driven fraud. From deploying advanced detection technologies to fostering robust networks of allies, learn how the financial sector is bolstering its defenses against sophisticated scams like deepfakes and biometric fraud.

Deepfakes and Dollars

August 20, 2024

While AI has the potential for transformative impact in the financial sector, it also introduces new challenges. From biometric mimicry to the alarming rise of deepfakes, this blog highlights emerging threats and challenges in countering AI-driven fraud.

Understanding and Addressing Fraud and Corruption Risks

July 2, 2024

In our recent webinar, industry experts delved into the crucial topic of fraud and corruption risks. From seasoned professionals to anti-corruption activists, the webinar featured a panel of distinguished speakers: Steve Burgess, Nikki Kenyon, Chris Williams, and James Wasserstrom. This blog post aims to summarize the key points discussed during the session, equipping readers with the knowledge necessary to navigate these risks effectively.

Corruption in Construction

June 21, 2024

Corruption in the construction sector can include everything from fraud, to extortion, embezzlement, and other abuses. Corruption doesn’t just undermine good governance; it endangers lives and threatens the reputation of any financial institution involved in construction projects that are rife with fraud and abuse. How can government organizations and financial institutions mitigate corruption risks when funding or supporting infrastructure projects? How can they detect suspicious transactions?

Corruption Kills

June 13, 2024

Construction projects are especially vulnerable to corruption because of their complexity, high price tags, and the number of intermediaries and other parties involved. Corruption in the construction sector does not merely result in financial losses, environmental destruction, and inequality, but it can also result in loss of life.

Tech vs. Corruption

June 6, 2024

As corruption continues to pose significant challenges across various sectors, including government agencies, traditional banking institutions, and designated non-financial businesses and professionals (DNFBPs), innovative technological solutions are proving to be essential in combating these issues. This article highlights how blockchain, AI, and data analytics are revolutionizing the fight against corruption.

Upholding Integrity — Obligations in Combating Corruption and Bribery

May 29, 2024

Bribery and corruption represent a critical nexus where ethics, governance, and societal values intersect. These issues have long been a stain on both public and private sectors, creating distrust and distorting fair competition. International bodies and conventions typically define public sector corruption as the abuse of public office or a position of public trust for private gain. Explore a recent case study and how organizations can protect against corruption.

AI in Crypto Security: Navigating the Dual Edges of Innovation and Vulnerability

May 14, 2024

As cryptocurrencies gain mainstream acceptance, they also become a new frontier for financial crimes, including money laundering, fraud, and the financing of terrorism. Amidst this burgeoning digital economy, AI emerges as both a beacon of hope to help prevent crimes and a potential threat, giving illicit actors additional resources to move proceeds of crime. The dual nature of AI in combating or facilitating crypto-related financial crimes presents a complex puzzle: Is AI a harbinger of risk or the ultimate solution to mitigate it in this digital age of ambiguity?

Golden Visas & Global Graft – How Criminal Actors Exploit Citizenship by Investment Programs

April 3, 2024

While special naturalization programs offered in certain jurisdictions to foreign investors intend to attract foreign direct investment and economic and infrastructure development, these programs can be abused by criminals who seek to launder and conceal proceeds of crime or commit new offences, including financial crimes.