Leveraging MLOps Principles to Comply with CECL
Table of Contents
Savvy risk managers have reimagined their model risk management frameworks by incorporating model operations (MLOps) powered by technology and automation. Case in point are the early adopters of the current expected credit loss (CECL) model, who realized that the reimagined frameworks fit well within the ambit of the new accounting standard and benefitted greatly from its implementation.
The Federal Reserve System’s SR 11-7 supervisory guidance (2011) provides an effective model risk management framework for financial institutions (FIs) to avoid model risk scenarios that can lead to monetary losses, reputational damage, or poor business and strategic decision making. These risks grow multifold when advance statistical or ML techniques are leveraged for modeling complex impairment models such as CECL.
Fortunately, SR 11-7 guidelines are still applicable to some of the new aspects from the accounting standard CECL (Financial Accounting Standards Board, 2016). For instance, any FI under CECL regulation must explain and justify their entire CECL process, including (but not limited to) model development, validation and governance. Hence, regulators encourage the usage of a systematic process (such as MLOps) to track and handle the end-to-end CECL model lifecycle through the design, development, implementation and ongoing maintenance phases.
Considering this regulatory guidance, organizations preparing to adopt CECL ahead of its January 2023 implementation date should recognize how modern MLOps can help them comply with the new accounting standard with improved trust and efficiency.
What is CECL?
Meeting the complex demands of CECL adoption by manual processes is quite difficult but the broader SR 11-7 guidelines will eventually help both FIs and non-FIs develop effective CECL processes to limit model risk.
The CECL model under Accounting Standards Update (ASU) 2016-13 aims to simplify U.S. Generally Accepted Accounting Principles (GAAP) and provides for a forward-looking recognition of credit losses (i.e., expected losses) instead of accounting for incurred losses.
The objectives of the CECL model are to:
-
Streamline credit impairment rules, which extend across many standards, down to a single standard that covers most credit transactions, including securities held at amortized cost and loans
-
Facilitate timely recognition of credit losses by using an expected loss model instead of an incurred loss model
-
Require FIs to build expected credit loss estimates based upon historical information, current conditions, and reasonable and supportable forecasts
-
Simplify estimate calculation methodology requirement to the size and complexity of the organization
While the above CECL guidelines have a greater impact on banks, most non-banks have financial instruments or other assets that are subject to CECL as well. In the absence of any specific methodology requirements, these bank and non-bank entities have an opportunity to put together a solution that serves their institutional needs while appeasing regulators. However, the implementation and management of some of the critical elements of CECL can be overwhelming, especially given the fast-evolving implementation and management technology space, which is still standardizing the best way to architect end-to-end solutions.
Critical CECL Components
Compared to the existing Allowance for Loan and Lease Losses (ALLL) requirements, CECL requires more complex modeling inputs, assumptions, analysis, and other necessary compliance processes, such as:
-
Alternative benchmark models
-
Documentation
-
Independent model validation and model governance
-
Model back-testing and performance
-
Model tuning
-
Ongoing model performance monitoring
Since CECL is a forward-looking/expected future loss model that doesn’t come with specifications on modeling techniques (e.g., rule-based, statistical or machine learning), institutions are free to choose a loss estimation methodology, which can range from basic to highly complex. Some methods (e.g., Average Charge-Off, Vintage Analysis, Discounted Cash Flow (DCF), or Static Pool Analysis) rely on oversimplified assumptions that lead to less accurate estimates and therefore ineffectively predict expected losses.
On the other hand, the more complex methodologies (e.g., Probability of Default (PD), Loss Given Default (LGD), and Exposure at Default (EAD)) require the development and implementation of several interdependent supervised machine learning models. After being trained with historical loan level portfolio data, these models are then used to score new data.
What is MLOps?
MLOps is a set of best practices for different components of an ML workflow that allows organizations to experiment with different settings and keep what works for them. MLOps helps organizations and business leaders generate long-term value and reduce model lifecycle management risks associated with data science, machine learning and AI initiatives.
By setting a clear, consistent methodology for model operation, and more specifically ML operation, organizations can:
-
Proactively address common business concerns (such as regulatory compliance)
-
Acquire, clean and version large amounts of data
-
Enable reproducible models by tracking data, models, code and model versioning
-
Package and deliver models in repeatable configurations to support reusability
-
Enable greater collaboration between data scientists and engineers to build production-class services
Connecting the Two: How MLOps Can Help CECL Adoption
Both CECL and MLOps are principle-driven and not prescriptive in nature. An MLOps philosophy includes a process for streamlining model training, packaging, validation, deployment and monitoring in such a way that suits an organization’s size and needs. MLOps can help resolve CECL-related issues, from business implications, data management and credit modeling to risk, governance and technology.
Five Areas MLOps Can Help Resolve CECL Compliance Issues
While FASB’s implementation terms appear daunting, MLOps can help resolve CECL-related issues from business implications, data management and credit modeling to risk, governance and technology. The issues presented below are typical of a machine learning model life cycle inside an organization where many different professionals with varying skill sets attempt to use entirely different tools.
In addition, MRM professionals themselves often use different tools and their work can become painstakingly manual as they try to piece together a model lifecycle across an organization. MLOps standardization can help improve these processes and offer efficiencies for auditing and risk management, which is critical, particularly in industries where model audits are labor intensive but play a crucial role.
Laying an MLOps foundation allows data, development and production teams to work collaboratively and leverage automation to deploy, monitor and govern machine learning services and initiatives within an organization. By facilitating necessary compliance processes such as CECL, MLOps can add value through increased efficiencies, decreased operational risk and reduced costs.
BDO’s Valuation & Capital Market Analysis practice can help businesses evaluate and adjust their MRM processes to meet their evolving needs.
SHARE