Model risk management (MRM) is a framework of systemic oversight of the models a financial institution or organization relies on for financial reporting, decision-making, and other critical purposes. As financial regulators have noted, this oversight is important because of the "possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused.” Consider the potential consequences of using incorrect or misapplied models to conduct stress testing, assess capital adequacy, estimate credit losses, or monitor and manage high-risk customers.
An effective MRM function:
- creates and maintains an inventory of models used
- ranks those models by categorical risk
- ensures that models receive an appropriate and timely effective challenge within a system of strong internal control.
“Appropriate” and “timely” are relative concepts, so for low-risk models, an analysis might occur every three years. According to Abrigo advisors, current expected credit loss (CECL) models are typically high-risk, and the material estimates the models produce are too important not to warrant an annual examination at a minimum.
Model validation is a series of tests of the critical elements of a model. Testing is performed by a separate party with no involvement in the development or day-to-day use of the model to avoid conflicts of interest.
Model validation results in identifying the weaknesses and limitations of a model, which may then be taken into consideration by bank or credit union management as the model outputs are utilized. You might often make a qualitative adjustment after identifying a limitation in your model—perhaps your model doesn’t address peaks and valleys very well, and you need to be sure they are accurately reflected. Periodic review of models is a best practice—as is getting a validation every time your financial institution or its markets undergo a change that could affect a model’s output.
An effective validation framework includes four core elements:
- Evaluation of conceptual soundness, including developmental evidence
- Evaluation of model inputs, which are your data and assumptions
- Evaluation of the key calculations of the model
- Evaluation of outcomes analysis, including back-testing and benchmarking
These elements apply equally to model validations performed in-house or by a vendor. They are outlined in the Federal Reserve’s supervisory letter SR11-7.
