AI in Financial Services: Compliance Risks Firms Should Consider

Artificial intelligence (“AI”) has shifted from being, until relatively recently, a theoretical possibility to now a practical reality that is deeply ingrained throughout almost every industry in Australia. In searching for efficiency, speed and enhanced capability, organisations nationwide are increasingly experimenting with, deploying and/or scaling, AI‑driven systems.

In relation to Australia’s financial services (“AFS”) sector, AI does offer clearly identifiable benefits for clients, providers of financial product advice and product providers alike. However, these benefits do not come without risks and, for those who use AI software in the provision of their services (e.g. AFS licensees including financial advisers and product providers), the lure of “operational efficiencies” and “maximising revenue” can be prioritised over the assessment of the risks associated with the use of a particular AI software program or relying too heavily on AI software for assistance. The Australian Securities and Investments Commission (“ASIC”) has consistently affirmed to the financial services industry that the adoption of AI must be matched by safe, ethical and accountable governance practices.

Setting the scene

The release of ASIC’s first dedicated report addressing the use of AI in the financial services industry was published in October 2024, entitled “Beware the gap: Governance arrangements in the face of AI innovation” (“Report 798”).

In Report 798, ASIC examined how 23 AFS licensees were using AI in their businesses. These AFS licensees reported to ASIC that, in total, there were 624 “AI use cases” (i.e. a clear practical way that AI was being used), each of which directly or indirectly affected their clients. The 23 AFS licensees operated across various industries including banking, credit, insurance and financial advice. In Report 798, ASIC observed that, generally:

• most AFS licensees’ use of AI tended to be cautious, particularly in relation to decision-making and customer interactions
• AI was generally used to augment, rather than replace, human decision-making
• there was limited direct interaction between AI systems and the AFS licensees’ clients.

However, since ASIC published Report 798, ASIC’s Chair, Mr Joseph Longo, warned that governance and risk management arrangements, being inherently slower to change, may struggle to keep pace with accelerating AI adoption across the financial services industry.

Longo further cautioned that without appropriate AI governance frameworks in place, “we risk seeing misinformation, unintended discrimination or bias, manipulation of consumer sentiment and data security and privacy failures, all of which has the potential to cause consumer harm and damage to market confidence.”

What Report 798 means for AFS licensees’ regulatory obligations

While specific use cases of AI may be novel and specific to each business’ needs, Report 798 highlighted the divergence in how frequently and extensively different AFS licensees used AI. As financial services laws (e.g.  Corporations Acts 2001 (Cth) (“the Corporations Act”) and ASIC Act 2001 (Cth)) are “technology neutral”, ASIC’s regulatory position remains unchanged. However, from Report 798 we can see that ASIC, as the regulatory authority for the financial services industry in Australia, has certain expectations that it expects AFS licensees to meet if they either currently use or intend to use AI in providing financial services to their clients.

In Report 798, ASIC reiterated that AFS licensees must continue to:

1. maintain measures to comply with all of their obligations, including the thirteen general obligations under section 912A(1) of the Corporations Act
2. ensure that company directors and officers discharge their duties with the reasonable degree of care and diligence required at law
3. avoid engaging in unconscionable conduct
4. refrain from making false or misleading representations
5. remain fully responsible for outsourced functions, including AI-enabled processes delivered by third-party vendors
6. ensure they have adequate technological and human resources to supervise, operate and monitor AI systems effectively.

Compliance risks

With the scene now set, the question becomes what specific risks arise when financial services firms integrate AI into their operations? Based on ASIC’s observations in Report 798 and broader industry analysis, three areas of heightened compliance risk stand out.

1. “Black box” and opaque AI models

One of the clearest concerns ASIC raised in Report 789 was in relation to AI systems whose internal logic is not transparent, particularly when those AI systems have direct impacts on an AFS licensee’s end client. For example, in Report 798, ASIC highlighted an example involving an AI model being used to predict consumer credit default risk by generating a risk score. However, the model was described as a “black box” because the AFS licensee could not explain or identify what the variables were that influenced the client’s risk score or how those variables were applied to affect the customer’s risk score.

Why does this matter?

This matters because, by using an AI system that is “opaque”, in that its internal logic (i.e. the method by which the AI system makes its decisions in relation to the information that it is processing) is either unknown or poorly understood by the AFS licensee, the AFS licensee may inadvertently fail to meet their general obligation to provide their services efficiently, honestly and fairly to their clients. This is because, where an AFS licensee cannot explain the decisions made by their AI system (e.g. why a client received a particular product recommendation, how a credit score or risk rating was generated, or whether a model has inadvertently embedded bias against certain groups), it will not know whether or not it has acted unfairly towards the client.

Report 798 found that of the 624 use cases analysed, there is a growing shift towards more complex and opaque techniques, particularly those used to process and analyse large volumes of images, audios and text data. These techniques accounted for 32% of all use cases under development.

ASIC emphasised that the adoption of opaque techniques and generative AI (“Gen-AI”) introduces several risks that can amplify consumer harm.

2. AI‑driven disclosure and communication risks

Report 798 identified that, in the limited instances where Gen-AI was used to interact with consumers, it was typically confined to generating first draft of documents, such as correspondence or marketing material, analysing calls, and summarising call transcripts and consumer correspondence. In other words, generative output was largely used to support human processes, with some AFS licensees purposely deciding that a human would be involved in, and accountable for, each decision where AI was involved, rather than replace them.

However, Report 798 also highlighted an emerging tension between ASIC and AFS licensees regarding the appropriate level of transparency and contestability in AI-enabled interactions. Specifically, questions revolved around:

1. how much AI had to be involved in an interaction or decision before it should be disclosed to an AFS licensee’s clients
2. whether consumers would find disclosure useful
3. whether it was necessary to introduce transparency now, given some models had been in use for a longer period of time.

ASIC, in an effort to steer the industry toward a more conservative and consumer-centric approach, recommended that AFSLs refer to the Australian AI Ethics Principles as good practice for AI policies and procedures.

In effect, where in doubt, ASIC’s view is that AFS licensees should provide their clients with meaningful disclosure if they are using AI systems and software in connection with their provision of financial services and should implement robust internal governance processes and procedures to effectively monitor and control the risks that are associated with the use of those AI systems.

3. Chatbots (consumer-facing AI assistants)

Since the release of ChatGPT in November 2022, the number of Gen-AI use cases has expanded exponentially. While many of these applications exist in standalone platforms or as separate products, there are several that allow for “chatbots” to be implemented onto an AFS licensee’s website, which can interact directly with customers who visit the AFS licensee’s website.

However, at the time of Report 798, ASIC found that the deployment of chatbots within the wider financial services industry was limited. Most chatbots that were being used by several of the AFS licensees reviewed by ASIC were being used either for internal support or for answering simple customer questions using pre-scripted responses, and so posed a lower risk to the business.

Risks unique to chatbots

Despite their potential benefits, Gen-AI can make mistakes. These mistakes can include “hallucinations”, which means that where the Gen-AI system cannot locate a specific answer it takes a “best guess”, and, in doing so, may artificially create information to fill in the gaps in its own information systems. This introduces several compliance risks that require careful management:

1. Provision of unlicensed financial advice: An AFS licensee may only be authorised to provide general financial advice and not personal advice. Say, the AFS licensee decides to implement a chatbot on its website that interacts with its clients who visit their website. While we (i.e. people) may be able to distinguish between general advice and personal advice, a chatbot (even a sophisticated one) will not likely be able to do so. As a result, users may be able to, with a sufficient amount of prompting, force the chatbot into inadvertently giving personal advice by producing a personalised product recommendation.
2. Insufficient escalation pathways: Chatbots may not be able to identify vulnerable customers, complex situations or queries requiring human judgment. Without clear, well-tested escalation processes, customers may receive inappropriate information or insufficient information, heightening the risk of poor or unfair outcomes.
3. Failure to maintain adequate records: If a client’s interaction with a chatbot is not properly recorded and retained, it may pose a problem for an AFS licensee’s ability to demonstrate compliance with its legal obligations. For example, if a client raises a complaint with a chatbot, that complaint might not be appropriately escalated internally resulting in an AFS licensee not responding to the client in the required timeframe.

Implementing pragmatic AI practices

As AI adoption accelerates throughout the financial services sector, compliance functions must stay ahead of the curve. ASIC does not expect firms to resist AI, but rather embed governance mechanisms that evolve in parallel with innovation and maintain the same standards of fairness, transparency and accountability that apply to all other traditional systems and processes used in the provision of financial services to their clients.

TIP 1: Implement an AI Policy – as AI integration and reliance increases, AFS licence holders should develop and maintain a dedicated AI policy. Their AI policy should outline the processes and procedures that are in place to ensure that any AI systems or software that is used in the provision of financial services to clients is monitored, controlled and reviewed in an appropriate manner by staff who have the requisite skills and experience.

The Australian Government’s Department of Industry Science and Resources has published a sample template here.

TIP 2: Establish an AI-specific post-implementation review program – firms should undertake periodic audits of model outputs, chatbot transcripts and AI‑generated disclosures. These reviews support ASIC’s expectations for strengthened governance and help ensure that misstatements, emerging bias or incorrect decision logic are identified and addressed promptly.

TIP 3: Maintain comprehensive model explainability documentation – when deploying or scaling AI systems, AFS licensees should maintain detailed records of training data sources, underlying assumptions, known limitations and human‑oversight controls. Housing this documentation within the firm’s record‑keeping systems strengthens internal supervision and enables AFS licensees to demonstrate compliance if ASIC seeks information about how an AI‑generated outcome was reached.

The Australian Government has published a Voluntary AI Safety Standard to help organisations develop and deploy AI systems safely and reliably.

Have any further questions?

Author: Glenjon Aligiannis (Senior Associate) and Luka Razlog (Graduate)