SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS

of external data in the model development, the outsourcing of validation task and the validation in the context of data scarcity. 1 Regulation (EU) No 1093/2010 establishing a European Supervisory Authority (European Banking Authority) 2 https://www.eba.europa.eu/eba-publishes-report-on-progress-made-on-its-roadmap-to-repair-irb-models SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 4 List of figures and boxes FIGURES Figure 1: Structure of the validation handbook 11 Figure 2 : Schematic view of the development sample versus Validation samples 37 Figure 3: Schematic view of best practice of uses of validation samples between the CRCU and the validation function 70 CONTEXT BOXES Context box 1: assessment of the process-related aspects of rating system changes by IA 26 Context box 2: accuracy in the development sample and conservatism in the application portfolio of the rating assignment 29 Context box 3: the different forms of human judgment 34 Context box 4: specific cases with conservative requirements in the CRR 44 Context box 5: EBA Supervisory Benchmarking 49 Context box 6: the data quality framework and its application in the IRB framework 60 Context box 7: the review of estimates 74 FOCUS BOXES Focus box 1: validation cycle in relation to the interaction with the CA 16 Focus box 2: validation sample and validation data set 21 Focus box 3: quantitative thresholds and comparative analyses 22 Focus box 4: accuracy of the rating assignment in the context of model (re-)development 35 Focus box 5: accuracy of the rating assignment in the context of model (re-) development 40 Focus box 6: specific challenger analyses for the modelling choices under the slotting approach 58 Focus box 7: IT implementation and assignment of parameter estimates 64 Focus box 8: samples used during the validation of new or changed aspects of a rating system 66 Focus box 9: specific additional analyses expected during the full validation 79 INTERACTION BOXES Interaction box 1: structural independence of the validation function vis-a-vis the CRCU 17 Interaction box 2: assessment of the rating system 24 Interaction box 3: assessment of the definition of default 28 Interaction box 4: assessment of the human judgment in the assignment process 31 Interaction box 5: statistical assessment of the model between CRCU and validation function 36 Interaction box 6: assessment of the CRM 52 Interaction box 7: other validation function in FCP 55 Interaction box 8: assessment of the data quality between the different functions 61 Interaction box 9: assessment of the IT implementation of the rating system 64 Interaction box 10: interaction between the CRCU and the validation function during the first validation activities 67 Interaction box 11: statistical assessment of the model between CRCU and validation function during the first validation 70 Interaction box 12: interaction between the CRCU and the validation function during the regular validation 76 SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 5 Abbreviations CA Competent authority CDR Commission Delegated Regulation CF Credit conversion factor CRCU Credit risk control unit CRD Capital Requirements Directive (Directive 2013/36/EU) CRM Credit risk mitigation CRR Capital Requirements Regulation (Regulation (EU) 2013/575) DR Default rate EBA European Banking Authority EL Expected loss EU European Union FCP Funded credit protection GL Guidelines GL on CRM Guidelines on Credit Risk Mitigation for institutions applying the IRB Approach with own estimates of LGDs (EBA/GL/2020/05) GL on downturn LGD estimation Guidelines for the estimation of LGD appropriate for an economic downturn (‘Downturn LGD estimation’) (EBA/GL/2019/03) GL on Internal governance Guidelines on internal governance under Directive 2013/36/EU (EBA/GL/2021/05) GL on outsourcing Guidelines on outsourcing arrangements (EBA/GL/2019/02) GL on PD and LGD estimation Guidelines on PD estimation, LGD estimation and the treatment of defaulted exposures (EBA/GL/2017/16) IRB Internal ratings based IA Internal Audit LGD Loss given default MoC Margin of conservatism OOS Out-of-sample OOT Out-of-time PD Probability of default RDS Reference data set RTS Regulatory Technical Standards CDR on assessment methodology Commission Delegated Regulation (EU) 2022/439 SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 6 CDR on model changes Commission Delegated Regulation (EU) No 529/2014 CDR on slotting approach Commission Delegated Regulation (EU) 2021/598 RW Risk weight RWEA Risk-weighted exposure amounts Slotting approach Supervisory Slotting Criteria Approach UFCP Unfunded credit protection SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 7 1. Introduction: overview of the validation handbook 1.1 Legal status of the supervisory handbook 1. Background. In the context of the ‘Internal Ratings Based’ Approach, the EBA has previously published under the roadmap to repair IRB models a number of requirements, aiming at reducing the RWEA unjustified variability stemming from different supervisory and bank- specific practices. The validation of IRB rating systems is an essential step to ensure a robust measurement of credit risk within the IRB Approach and, consequently, competent authorities usually have a (published or unpublished) set of expectations for the internal validation of IRB rating systems as part of their supervisory practice. This supervisory set of expectations is in particular reflected in the supervisory approval process and the on-going supervision of rating systems used under the IRB Approach. In the past, the EBA has identified heterogeneous supervisory practices in this regard. With the validation handbook, the EBA aims to achieve a harmonised supervisory understanding and harmonised supervisory practices by providing an outline of best supervisory practices on expectations than can be put on the institution’s implementation of the requirements for validation of IRB rating systems. 2. Mandate. The task of the EBA to develop and maintain a supervisory handbook derives from Article 8(1)(aa) of the EBA Regulation3 which stipulates that the EBA shall ‘develop and maintain an up-to-date Union supervisory handbook on the supervision of financial institutions in the Union which is to set out supervisory best practices and high-quality methodologies and processes and takes into account, inter alia, changing business practices and business models and the size of financial institutions and of markets’. In addition, Article 29(2), second subparagraph, of the EBA regulation specifies that ’For the purpose of establishing a common supervisory culture, the Authority shall develop and maintain an up-to-date Union supervisory handbook on the supervision of financial institutions in the Union, which duly takes into account the nature, scale and complexity of risks, business practices, business models and the size of financial institutions and of markets.’ Therefore, the supervisory handbook should cover all matters which are within EBA's remit with the aim to set out best supervisory practices rather than provide further specifications for the application of the legislation. 3. Content of the handbook. The validation handbook details the best supervisory practices, which the CAs are expected to give consideration to when performing their supervisory activities and developing their own expectations on the validation of IRB rating systems. More precisely, with the validation handbook, the EBA aims to promote convergence of CA’s approaches and practices by providing good and best practices observed both within institutions, i.e. in terms of validation framework implemented, as well as within supervisors, 3 Regulation (EU) No 1093/2010 establishing a European Supervisory Authority (European Banking Authority) SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 8 i.e. in terms of best supervisory practices and ‘expectations’ derived from existing regulatory requirements. These expectations should be understood as a best supervisory practice of the expectation to a sound IRB validation. 4. The supervisory handbook vis-à-vis EBA GL. Both EBA GL and the supervisory handbook are of non-binding nature, of general application and acts of Union law whose validity can be determined only by the Union courts, in a preliminary ruling.4 However, unlike guidelines, the supervisory handbook is not addressed directly to financial institutions, but to CAs, and neither limit their judgment-led supervision, nor the supervisory assessment of the individual cases. As no ‘comply or explain’ mechanism is applicable to the handbook, any departure from it can be justified merely on the needs of judgment-led supervision. In terms of harmonisation effects, the handbook will predominantly be used as a benchmark of convergence during peer and other reviews. 1.2 Specificities of the validation in the regulatory framework 5. General definitions. The general definition of the term ‘model validation’ is known in various fields such as computer science, engineering and finance; as different as these disciplines are, model validation always refers to one of the key assessments undertaken to verify that a model is working as expected. In general, ‘model risk’ can be described as the potential for adverse consequences of decisions based on incorrect or misused model results and reports. Against this background, point 11 of Article 3(1) of the Capital Requirements Directive (Directive 2013/36/EU - CRD) defines this risk as the risk of a potential loss ‘an institution may incur, as a consequence of decisions that could be principally based on the output of internal models, due to errors in the development, implementation or use of such models’. Thus, the main task of the model validation process is to prevent models from producing inadequate results, by effectively challenging them and by assessing and identifying possible assumptions, limitations and shortcomings. 6. Previous work on the validation in context of the Internal Ratings-Based Approach (IRB Approach). The scope and objectives of validation have already been described by various other initiatives. This handbook is bringing together the perspectives provided by these various initiatives, also leveraging on good and best practices observed by CA. Prior to this handbook, the ‘CEBS Guidelines on the implementation, validation and assessment of Advanced Measurement and Internal Ratings Based Approaches’5 (CEBS Guidelines 10) provided clarification on the validation activities for IRB Approach in the European Union (EU). This guidance was based on the Newsletter 4 on validation6 from the Basel Committee on Banking 4 Therefore, the handbook provides general supervisory ‘expectations’, for the consideration of CAs, which are linked to already existing regulatory requirements. It also contains a set of what is considered as ‘best practices’ based on supervisory experience, as well as ‘good practices’ observed in institutions. 5 https://www.eba.europa.eu/sites/default/documents/files/documents/10180/16094/525151b9-ea22-42b2-bd28- 00e35a0add7e/GL10.pdf?retry=1 6 https://www.bis.org/publ/bcbs_nl4.pdf SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 9 Supervision and listed six principles that validation of the IRB Approach should follow.7 In the United States of America, the Board of Governors of the Federal Reserve System issued an ‘SR letter’ dealing with the ‘supervisory guidance on model risk management’.8 This letter provides guidance on effective model risk management, where model validation plays a critical role. It builds on a previous bulletin issued by the Office of the Comptroller of the Currency in 2000, which outlines key model validation principles and expectations on sound model validation processes which was updated in 2011 and included in the 2011 Federal Reserve System newsletter. 7. The IRB validation beyond a model validation. In all these previous publications, there is a consensus that the validation of IRB rating systems goes beyond the pure concept of model validation. The validation of IRB rating systems is not limited to the proper functioning of a model from a statistical perspective, but also includes the assessment of the data quality, the structure of the rating system and its correct application. It includes the set of policies, processes and procedures put in place to assess the accuracy and performance of the rating systems9 on the institution-specific portfolios and to verify that the models10 used by the institutions work properly. 8. The IRB validation in the EU framework. This understanding is reflected in the EU regulation,11 and further elaborated in Chapter 3 of the CDR on assessment methodology12 which clarifies the interaction between the validation of IRB rating systems and the internal governance, as well as risk oversight in general. As such, the internal validation framework is not only limited to the tasks and organisation of the validation function: the regulation includes minimum requirements on the senior management and management body,13 the internal reporting,14 the 7 Principle 1: Validation is fundamentally about assessing the predictive ability of a bank’s risk estimates and the use of ratings in credit processes; Principle 2: The bank has primary responsibility for validation; Principle 3: Validation is an iterative process; Principle 4: There is no single validation method; Principle 5: Validation should encompass both quantitative and qualitative elements; Principle 6: Validation processes and outcomes should be subject to independent review. 8 https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf 9 A ‘rating system’ is defined in the Article 142(1), point (1) of the Capital Requirements Regulation (Regulation (EU) No 575/2013 -CRR) as ‘the methods, processes, controls, data collection and IT systems that support the assessment of credit risk, the assignment of exposures to rating grades or pools, and the quantification of default and loss estimates that have been developed for a certain type of exposures’. 10 ‘PD model’ and ‘LGD model’ are defined in section 2.4 of the Guidelines on PD estimation, LGD estimation and the treatment of defaulted exposures (EBA/GL/2017/16 - Guidelines on PD and LGD estimation). While not provided in these guidelines, the definition of a CF model can be inferred using Article 4(1), point (56) of the CRR, as ‘All data and methods used as part of a rating system within the meaning of Article 142(1), point (1) of the CRR, which relate to the differentiation and quantification of own estimates of CF which are used to assess the level of currently undrawn amount of a commitment that could be drawn and that would therefore be outstanding at default, to the currently undrawn amount of the commitment, for each facility covered by that model.’ 11 Such as Articles 144(1)(f) and 185 of the CRR. 12 These requirements only apply indirectly to institutions as the CDR on assessment methodology provides the scope of assessment criteria and the methods to be applied by CAs. 13 Article 14 of the CDR on assessment methodology 14 Article 15 of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 10 interaction with CRCU15 and the internal audit (IA).16 In addition, while the validation function is not separately mentioned in the Guidelines on internal governance,17 it can be viewed as an internal control function (section 19) given the tasks it performs. 9. The IRB validation through multiple layers of defence. The scope of activities to be performed in the context of the validation of IRB rating systems has led to a specific set of governance and organisational requirements. In particular, the assessment of the model performance is performed by several functions, each of them with its own perspective. In this respect, the CRCU has an ‘active participation in the design or selection, implementation and validation of models used in the rating process’18 and as such is the first function to analyse and validate the model. However, the EU regulation requires in addition institutions to set up a specific independent validation function with its own responsibilities. Following the background and rationale of the Final draft regulatory technical standard on assessment methodology for IRB,19 the independence of the validation function from the CRCU is essential ’in order to allow for an objective assessment of the rating systems, a limited incentive to disguise the model deficiencies and weaknesses, as well as a fresh view on the rating systems by people not involved in the development process’. 15 Article 16 of the CDR on assessment methodology 16 Article 17 of the CDR on assessment methodology 17 EBA/GL/2021/05 18 Article 190(2)(f) of the CRR 19 EBA/RTS/2016/03 SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 11 1.3 Structure of the supervisory handbook Figure 1: Structure of the validation handbook 10. Structure of the handbook (illustrated in Figure 1). A general description of the requirements applicable to the validation function and on the tasks to be performed is given in the sections [2 General ] and [3 Validation content] respectively. In practice, the description of the validation tasks is organised in two sections. The tasks related to the model performance assessment are developed in section [3.1 Assessment of the core model performance] and the ones dealing with the modelling environment (i.e. the data quality and the IT implementation of the rating system) are further described in section [3.2 Assessment of the modelling environment]. The presentation of the core performance assessment is split in three parts: SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 12 a. Performance of the rating system - Risk differentiation: Consistency and comprehensiveness of the rating assignment 1. Documentation for consistency 2. Comprehensiveness and conservatism for non- standard ratings Accuracy of the rating assignment Discriminatory power Homogeneity & Heterogeneity Input data 1. Data quality 2. Completeness of the RDS 3. Data preparation (including estimations) 4. Representativeness Methodological choices 1. Risk drivers 2. Functional forms and human judgment 3. Definition of grades or pools Statistical tests 1. Scope and level of application 2. Various economic conditions Validation challengers 1. Impact of overrides 2. Number of overrides 3. Stability of the ratings 4. Monotonicity of the DR 5. External data sources 6. Concentration in rating grades b. Performance of the rating system - Risk quantification Input data 1. Data quality 2. Completeness of the RDS 3. Data preparation (review of the exclusions and realised LGD floored at 0%) 4. Representativeness (challenge adjustments) Methodological choices PD LGD Conservatism Downturn (DT) 1. General calibration methodology 2. Average DR (Overlapping windows only relevant for PD) 3. LRA (including for LGD treatment of Incomplete work-out) 4. Calibration segment and type 5. Appropriate adjustments 1. 180(1)(c) and 181(1)(c)(d) of the CRR 2. Quantification for each MoC category 3. Aggregation of MoC categories 1. Period for the Economic DT 2. DT LGD Validation challengers 1. Compare DR with PD and similar analysis for LGD and CF – 185(b) CRR 2. Other quantitative validation tools (best estimates) – 185(c) CRR 3. External data sources SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 13 c. Performance of the rating system - Other specific points Defaulted exposures 1. RDS: reference dates, realised LGDs and data requirements 2. ELBE: MoC, economic conditions and SCRA 3. LGD in default: relation with LGD non defaulted and ELBE CRM 1. RDS: source and allocation of cash flows, recoveries from collateral 2. Level of validation 3. Meaningful recognition (no double counting) FCP UFCP 1. On-balance sheet netting and master netting agreement 2. Adverse dependency 1. Choice of the approach 2. Recognition of multiple CRM Use of multiple CRM Slotting approach 1. Assessment of the assignment process 2. Assessment of the input data 3. Assessment of the modelling choices 4. Quantitative and challenger analyses 11. Difference between the first and on-going validation. While the section [3 Validation content] describes the areas to be assessed by the validation in a general manner, the actual tasks to be performed may differ depending on the position in the validation cycle (as further described in focus box [1]): a. The first validation of a rating system is part of the institution’s assessment conducted before submitting the application to the CA. Section [4 First validation activities] provides guidance with respect to the specific aspects of these validation activities regarding a first validation. b. On the contrary, on-going validation activities are to be conducted once the rating system has been approved by the CA in accordance with Article 143 of the CRR. Details on on-going validation activities are described in Section [5 On-going validation activities]. 12. Focus sections. This handbook also covers some specific areas where the validation function may face specific challenges: a. Some models are developed on a broader scope than the scope of application. The specificities of the validation of this type of models are further discussed in the section [6.1 Focus 1: validation in the context of the use of external data] b. Some of the operational tasks of the validation function can be outsourced both internally or externally. However, the responsibility for these tasks remains within the validation function. This particular point is further described in the section [6.2 Focus 2: validation in the context of outsourcing of validation tasks]; c. The validation of the so-called low-default portfolio, or more generally low-data portfolio, can result in challenges for the use of some validation tools which require a minimum number of observations to be conclusive. Further thoughts on the SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 14 validation of these models are given in the section [6.3 Focus 3: validation in the context of data scarcity]. 13. Validation of CF estimates. This handbook contains less guidance on the validation of own estimates of CF, given that these parameters were not explicitly part of the EBA IRB repair program. However and as a general remark, it is considered as best practice to have similar validation techniques in place as for the LGD risk parameter, especially when it comes to the validation of the downturn component (as mentioned in paragraph [44]) and the treatment of extreme realised values (as mentioned in paragraph [36.a]). SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 15 2. General principles for the validation framework 2.1 Scope and objectives of the validation 14. Scope of the validation. To satisfy the requirements of Part III, Title II, Chapter 3 of the CRR, the validation activities should be conducted at each level where a CA has granted an approval for a rating system (or is expected to do so in the context of an initial validation of a new rating system). Therefore, in the case where a rating system has received the approval on a consolidated as well as sub-consolidated and/or individual basis, the internal validation should be performed at all of these levels. While more than one validation function may be involved in the validation of a rating system, in particular in the context of outsourcing and/or when a rating system is used by different legal entities, the responsibility of the validation tasks and the validation objectives mentioned in paragraph [17] should be retained by the validation function of the entity at the level of which the rating system has been approved. As mentioned in paragraphs [18] and [25], the validation function’s resources and framework are expected to be commensurate with the complexity and materiality of the rating system. 15. Involvement of several entities in the validation through outsourcing. The degree to which one validation function can leverage on the validation activities of another validation function is further described in section [6.2 Focus 2: validation in the context of outsourcing of validation tasks], and in particular in paragraph [135]. 16. Involvement of several entities in the validation through a common rating system. In the case where a rating system is used at different levels of a group, the validation functions of the involved entities are expected to share their findings. A good practice observed in institutions is to form an opinion on a single shared set of possible recommendations on the corrective actions against any identified model deficiency or under-estimation of risk parameters. In particular, the validation functions may come to an agreement on whether a deficiency identified at a certain level is an indication of a general deficiency of the rating system at group level, taking into account the assessment of paragraph [17], along with a common understanding of the possibilities on how these might be remediated in line with paragraph [19]. In any case, institutions are required to ensure the sufficient capitalisation at all relevant levels (consolidated, sub-consolidated and/or individual basis), taking into account the assessment of the validation functions. 17. Objective of the validation.20 The validation function is expected to form an opinion on whether the final rating system developed by the CRCU meets the regulatory requirements and the internal expectations on the quality of the IRB models. It is expected to then communicate 20 Article 185(a) of the CRR, Article 11 of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 16 its opinion to the CRCU, the senior management and the management body, as part of the corporate governance as mentioned in paragraph [19].21 To this end, the validation function is expected to provide: a. A list of all the deficiencies identified, along with an assessment of their materiality and severity (e.g. via quantitative impact), such that it can be used by the CRCU to prepare a prioritisation plan for their resolution; b. An assessment of the consequences of the combination of these deficiencies on the overall performance of the rating system, along with the consequences in terms of usability of the rating system for regulatory purposes; c. An assessment of the level of confidence in the results of its assessments, in particular when lack of data can be considered as an impediment to the robustness of the statistical tests. This objective and independent assessment is in particular essential in order to ensure an effective interaction with the CA, as further explained in Focus Box [1]. FOCUS BOX 1: VALIDATION CYCLE IN RELATION TO THE INTERACTION WITH THE CA Validation activities are continuously performed during the full life cycle of an institution’s rating system. These activities aim at giving confidence to the CA that the IRB rating system is working in an appropriate way, either with a view of a first approval, or for its continued use. The first validation and related validation activities take place during or subsequently to the model development in order to assess the regulatory compliance and performance of the rating system, in view of providing an approval under Article 143 of the CRR. The first validation aims at ensuring the appropriateness of the rating system once being used for own funds requirements and internal risk management. At the same time, it also ensures that the newly developed rating system is ready for a supervisory (e.g. on-site) assessment (with necessary changes to the rating system implemented by CRCU following the validation function’s assessment).22 As such, an important focus point in the first validation is the methodological choices taken by the CRCU. The first validation of the rating system can be used as a starting point for the on-going validation and the related validation activities that are required to be conducted after regulatory approval was granted. The on-going validation activities aim at ensuring an effective challenge for the adequate model performance and appropriateness of the rating system for IRB purposes on an on-going basis, 21 Article 189 of the CRR. For the rest of this handbook, this identification and communication of the deficiencies will be referred to as ‘form an opinion on’. 22 Article 144 (1)(f) of the CRR SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 17 subsequently to a first validation. As such, the outcome of the on-going validation will typically be taken into account in the on-going supervisory assessment performed by the CA. These on- going validation activities differ from the first validation as they benefit from additional data and observations and have at their disposal previous conclusions from the first validation. As a consequence, for some specific tasks, the assessment from the validation function can be based largely on its previous conclusions. 2.2 The validation function as a second layer of defence, between CRCU and Internal Audit 18. Independence of the validation function vis-à-vis the CRCU. The validation function assesses the final rating system developed by the CRCU as a second layer of defence, i.e. it challenges in an independent manner the model design and methodological choices used by the CRCU during the model development. Thus, the independence of the validation function is crucial to prevent any conflict of interest, as well as ensure no subordination in relation to the CRCU. This independence is ensured by two means: a. The structural independence, ensured via the organisational setup (see Interaction box [1]). In this regard, it is expected that large and complex institutions apply the setup which provides the highest level of independence of the validation function (Point 1 of Interaction box [1]).23 However, as further described in paragraph [23.d], the validation function can leverage to some extent on the work performed by the CRCU. b. The sufficient resource allocation. In this regard, it is expected that the number, seniority and expertise of the validation staff is commensurate with the complexity and materiality of the rating systems under the scope of validation of the validation function,24 such that the validation function can still effectively challenge the work of the CRCU. INTERACTION BOX 1: STRUCTURAL INDEPENDENCE OF THE VALIDATION FUNCTION VIS-A-VIS THE CRCU Article 10 of the CDR on assessment methodology provides three different types of setups within the institution’s organisational structure which can be allowed, depending on the nature, size and scale of the institution and the complexity of the risks inherent to its business model: 1. The validation function is in a unit separated from the CRCU and both units report to different members of the senior management; 23 Article 10(5) of the CDR on assessment methodology 24 Articles 10(2)(a), 10(3)(a), 10(4)(a) and 12(b) of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 18 2. The validation function is in a unit separated from the CRCU, but both units report to the same member of the senior management; 3. The validation function is not in a unit separated from the CRCU, i.e. no separated validation unit exists, but the staff performing the validation function is different from the staff responsible for the design and development of the rating system, and from the staff responsible for the credit risk control function. 19. Communication of the findings and recommendations. The management body owns the responsibility for all material aspects of the rating and estimation processes and senior management shall have a good understanding of the rating system designs and operations and shall ensure, on an on-going basis, that rating systems are operating properly.25 In this context, the outcome of the validation function’s analyses, in the form of findings and recommendations, is expected to allow senior management to understand the identified model deficiencies and be in a position to decide on a remediation action plan as well as to have a good understanding of how these deficiencies are addressed in the risk estimates. Effectively, a key product of the validation function is the validation report, as further described in paragraph [24].26 In addition, while the validation function should perform its assessments independently from the CRCU (i.e. independently identify and report deficiencies and shortcomings) it is nonetheless expected to have a good understanding on the issues detected and on the possibilities on how these might be remediated. In any case, the findings and recommendations of the validation function should not displace the responsibility for the design or selection, implementation, oversight and performance of the rating systems, which should remain within CRCU, and the validation function should always remain critical on any changes implemented on the rating system.27 20. The IA as a separate function. When it comes to the IA function, in accordance with Article 191 of the CRR, the ‘Internal audit or another comparable independent auditing unit shall review at least annually the institution’s rating systems and its operations’.28 This requirement is complemented in the CDR on assessment methodology,29 which de facto sets the IA and validation functions as independent functions within institution’s governance structure, which constitute different levels of defence and should not be merged into a single function. Hence, the roles and responsibilities of the IA and the validation function should be clearly defined, such that all relevant tasks necessary for the evaluation of the rating system are performed, as discussed in paragraph [22], and cannot be transferred between each other. This requires the existence of an effective separation between the staff of the IA and the validation function. 25 Article 189 of the CRR 26 Article 13 of the CDR on assessment methodology 27 For example, the validation function could recommend redeveloping part of or in full the rating system, or recommend a recalibration or the introduction of additional MoC. 28 In the rest of the handbook, the IA refers to either the internal audit or another comparable independent auditing unit. 29 Article 17(1)(a)(iv) of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 19 21. The role of IA in the assessment of the validation function. In addition, according to paragraph 155 of the Guidelines on internal governance30 ‘The risk management framework should be subject to independent internal review, e.g. performed by the internal audit function’. Consequently, as the validation function is part of an institution’s risk management framework, the IA function should have an independent opinion on the institution’s validation function, which encompasses: a. The independence of the validation function, mentioned in paragraph [18]: i.e. the setup of the validation function (e.g. whether it has a sufficient number of resources) and its independence (in relation to the CRCU as well as to the personnel and management function responsible for originating or renewing exposures);31 b. The institution’s validation policy, mentioned in paragraph [23], i.e. the scope and suitability of the validation tasks in terms of assessment of the rating systems, including the documentation to be produced. This is of particular relevance in the context of a change in the validation methodology or processes, as this may entail a categorisation of the rating system change as material,32 and in this case the assessment of the IA is part of the application package sent to CAs;33 c. The adherence of the validation function to the validation policy during the performance of the validation tasks; d. The comprehensiveness and clarity of the conclusions of the validation function and the related documentation produced, including the validation report mentioned in paragraph [24]; e. The appropriateness and timeliness of the follow up of the validation function’s findings mentioned in paragraph [19] by the institution34 (and, where relevant, of the findings raised by the CA). 22. Interaction between the IA and the validation function in the assessment of rating systems. On top of the assessment of the validation function, the IA should review the adherence to all requirements applicable to the institution’s rating systems.35 This assessment encompasses: a. A high-level perspective of the institution’s rating systems, which includes in particular an overview of the rating systems and related risks to ensure the adequacy of own funds requirements (this includes the assessment of model risk, including the 30 EBA/GL/2021/05 31 Article 10(1)(a) of the CDR on assessment methodology 32 Annex I, Part II, section 1 point 4 of the CDR on model changes 33 Article 8(1)(e) of the Commission Delegated Regulation (EU) No 529/2014 (CDR on model changes): report of the institutions’ independent review 34 Article 13(c) of the CDR on IRB assessment methodology 35 Article 191 of the CRR SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 20 review of the classification of the materiality and complexity of the rating systems as further discussed in Context Box [1], as well as the corporate governance36 and use test37 fulfilment); b. An overview of all the operations related to rating systems, which includes in particular an annual review of the performance of each rating system. However, to avoid duplication of tasks between these two functions and ensure an appropriate challenge by the IA, the independent opinion that the IA forms on the institution’s rating systems can take into consideration the analyses performed by the validation function, where appropriate. In any case, all the necessary tasks to form an opinion on the institution’s rating systems should be performed and the IA should be responsible of the assessment of their completeness (i.e. absence of gaps due to the distribution of tasks between the different internal control functions); c. A detailed assessment of the elements not assessed in depth by the validation function. Some of these elements are further clarified in ‘interaction boxes’ in other parts of the handbook. In practical terms, some elements may not be reviewed in depth by the validation function before the assessment of the IA as part of the overview mentioned in paragraph [22.b]. For example, the validation function may not be responsible for the detailed review of the proper implementation of each rating system, which includes in particular the integrity of the rating system and rating grades assignment process, or the correct calculation of own funds requirements (e.g. allocation of each exposure to the proper exposure class, correct application of PD and LGD input floors, calculation of the maturity, IT implementation of the rating system). 2.3 Validation policy and validation report 23. Validation policy.38 The validation policy documents the validation framework, i.e. the roles, responsibilities, processes and content of the validation activities that are expected to be performed in a sufficiently precise manner such that a third party is able to gain a good understanding of the tasks the validation function will perform. A good practice observed in institutions is to document the validation policy in a single document. In particular, the validation policy is expected to include: a. A description of how the validation function forms its opinion on the accuracy and consistency of the rating system as a whole.39 This implies that the validation policy is expected to describe the aggregation methodology to be used across the different 36 Article 189 of the CRR, Chapter 3 - section 3 of the CDR on assessment methodology 37 Articles 144(1)(b), 145, 171(1)(c), 172(1)(a), 172(1)(c), 172(2) and 175(3) of the CRR, Chapter 4 of the CDR on assessment methodology 38 Articles 9(3)(c) and 12 of the CDR on assessment methodology, section 4.2.2 of the Guidelines on PD estimation, LGD estimation and the treatment of defaulted exposures (EBA/GL/2017/16 - GL on PD and LGD estimation) 39 Article 185(a) of the CRR SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 21 analyses, in particular where quantitative tests are performed as further described in paragraph [25.b]; b. A description of the data collection and selection process underlying the construction of all the data sets used for the purpose of validation. This ‘validation data set’ contains, but is not limited to, the validation samples as described in Focus Box [2], in addition to the reference data set (RDS) used by the CRCU during the estimation of the risk parameters or the review of estimates; FOCUS BOX 2: VALIDATION SAMPLES AND VALIDATION DATA SET The validation samples used for the performance assessment, i.e. for running the quantitative tests as further described in paragraph [25.b] validation challengers, contain all information to allow for all relevant types of validation analyses. This includes observations covering a time period that is as long as possible, a different level of consolidation (at sub-consolidated and/or individual level, where relevant) and at the level of scope of application of the individual PD and LGD models. In addition to the validation samples, other data is expected to be used by the validation function, wherever benefit can be gained from this additional information. The set of all data sets used by the validation function for the validation of a rating system constitutes the ‘validation data set’. This includes in particular the development data or other data used by the CRCU, or data gathered from independent data sources (e.g. for benchmarking purposes). c. The list of the analyses to be performed (as further described in paragraph [25]), along with a description of their purposes and possible limitations (i.e. underlying assumptions and theory), their scope of application (i.e. data sets on which they are applied and compulsory or discretionary use), their envisaged frequency (including if relevant for first, regular and/or full validation), the methodology to derive the scores mentioned in paragraph [17.b] where relevant, and the associated findings and recommendations mentioned in paragraph [19]. As further developed in the section [4 First validation activities] and section [5 On-going validation ], the validation framework is expected to take into account the specificities of the model life cycle. For the quantitative tests as further described in paragraph [25.b], the documentation is expected to include a high-level description of the expected data preparation process, the computations to be performed, the fixed targets and tolerance thresholds (see Focus Box [3]), as well as the potential (neither exhaustive nor mandatory) qualitative analyses to be conducted to complement the assessment; SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 22 FOCUS BOX 3: QUANTITATIVE THRESHOLDS AND COMPARATIVE ANALYSES In practical terms, while absolute thresholds can generally be considered as adequate backstops, it is usually helpful to complement them by an ad-hoc comparative analysis (such as with results of previous years as mentioned above). For the on-going validations, the comparison between the latest results of the validation and the ones observed in the previous years (and in particular the ones observed during the first validation) can be used to detect a trend (in particular in the case of a deterioration) in the model performance. In the context of the validation of rating system changes, typically the model performance of the new model has improved compared to the current version. Hence, the results of the key performance metrics of the new model can reasonably be expected to be better than the ones performed using the current model in place. Consequently, if the model performance of the new model has not improved compared to the current version, the validation function is expected to conduct a more in-depth analysis to fully understand the reasons. In the context of newly introduced rating systems, a relative comparison may be harder to find. While absolute thresholds may be deemed sufficient, other alternatives are however possible. One possible way is to compare the results of key performance metrics with the ones calculated for other rating systems (e.g. other exposure classes), for instance in the context of a roll out where relevant. Another possibility might be to use the differentiation provided by the Standardised Approach (via the different allocation of exposures into different risk weight (RW) buckets), the loss given default (LGD) and conversion factor (CF) regulatory values in the case of the introduction of the use of own LGD or CF estimates for non-retail exposures or the supervisory slotting criteria approach (slotting approach) for specialised lending exposures, and ensure that the risk differentiation provided by the IRB parameters leads to better results (this approach produces less relevant results for retail exposures). d. The conditions under which the validation function may leverage on the work performed by the CRCU (e.g. by reviewing the work performed by CRCU instead of performing its own calculation). As further developed in the section [4 First validation activities] and section [5 On-going validation ], the degree of leverage on the work performed by the CRCU may be different depending on the position in the model life cycle; e. The main content, frequency and recipients of the validation reports. 24. Validation report.40 A key component of the communication of the validation opinion on the rating system is the validation report. Its structure is left to the own judgment of the validation function in order to optimise the communication of its opinion, and as such is not expected to 40 Articles 9(2)(c), (d), 9(3)(d), (e) and 13 of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 23 be harmonised between institutions (and is in particular not required to follow the structure of this validation handbook), nor necessarily between different rating systems for a given institution. In practical terms, the validation report is expected to be understandable by a knowledgeable third party and describes how the validation policy has been applied to a particular rating system (the validation report can refer to the validation policy while providing the validation result for a particular rating system), and as such details: a. The rating system version that was subject to validation and, where relevant, a description of the on-going model development activities, i.e. performed since the last validation or planned in a short-term horizon. In this context, the validation report is expected to provide the opinion of the validation function on the rating system changes, including their materiality assessment, since the last validation report, and the changes themselves; b. All the relevant tests performed to challenge the rating system, along with a description of the validation data preparation and the related data quality of validation samples used mentioned in paragraph [23.b], including their sizes (e.g. in terms of number of exposures and number of years), and a comparison vis-a-vis the application portfolio and overlap with the RDS (e.g. development sample or calibration sample); c. The outcomes of the validation analyses, which are expected to be verifiable by other internal functions or external parties (e.g. the CRCU, the IA or the CA). In this regard, the validation function is expected to express a clear opinion on the performance of the rating system as mentioned in paragraph [17], and to determine a categorisation of the findings and the relevant recommendations in accordance with their materiality. In this context, a good practice observed in institutions is to communicate these results in the form of scores (e.g. traffic light approach); d. A good practice observed in institutions is that the report includes a comparison between the latest results of the validation and the ones observed in the previous years (as a result of the analyses performed as per paragraph [25]), as well as the highlighting of the previously identified deficiencies, along with their severity, and a description of how they have been addressed. 2.4 Validation tasks 25. Type of analyses to assess the accuracy and consistency of rating systems. Institutions shall have robust systems in place to validate the accuracy and consistency of rating systems, processes and the estimation of all relevant risk parameters.41 The validation methods and procedures should be appropriate to the nature, degree of complexity and range of application of the rating systems and the data availability, and as such all relevant validation techniques 41 Article 185(a) of the CRR, Article 12(a) of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 24 are expected to be used. In particular, institutions are expected to include quantitative as well as qualitative methods that are commensurate with the complexity and materiality of the rating system (in terms of volume and riskiness).42 In particular, the areas described in section [3 Validation content] do not prevent institutions from developing additional tools and methods. In practice, for the assessment of the model performance described in section[3.1 Assessment of the core model performance], the type of analyses performed by the validation function are mainly, on the one hand, an assessment of the work done by the CRCU, and in particular of the different (modelling) choices taken during the development of the rating system, and, on the other hand, the development of own empirical validation challengers, in particular using new set of data not used during the development of the rating system: a. Assessment of CRCU’s work and related documentation.43 To the end of gaining a good understanding of the CRCU work, the validation function is expected to review and challenge the steps performed during model development and risk quantification (or potentially the review of estimates during the on-going validation), respectively, as well as the decisions made during these processes. In this context, as a good practice observed in institutions, the validation function reproduces the documented steps of the risk parameters’ estimation and / or reviews directly the code, and in any case is expected to check if the methodological and technical documentation of the rating system related to the validation function’s scope of assessment is usable and understandable by any third parties.44 The usability implies that a third party is able to replicate the estimation of risk parameters and arrive at the same results, such that the validation function can independently assess the rating system. As a matter of fact, in addition to being used by the CA for the authorisation of the rating system, the documentation is expected to be used by the validation function to assess the model specifications and monitor their changes (see Interaction Box [2]), on the top of using metrics measuring the pure performance of the model. INTERACTION BOX 2: ASSESSMENT OF THE RATING SYSTEM DOCUMENTATION While the validation function assesses the content of the documentation, in particular during its assessment of the rating system developed by the CRCU, it is not necessarily expected to perform a regulatory compliance check based on the documentation it receives.45 It is considered as best practice for institutions to have at least the documentation of the technical IRB modelling aspects of compliance with applicable laws, rules, regulations and standards checked by the validation function. 42 Article 12(b) of the CDR on assessment methodology 43 Article 11(2)(a) of the CDR on assessment methodology 44 Articles 144(1)(e), 171(1)(b), 175 of the CRR, Article 3(1)(d) of the CDR on assessment methodology 45 E.g. assessment of the completeness of the documentation on the design and operation details of rating systems as per Article 31 of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 25 When it comes to the review of the non-technical aspects of the documentation, different practices have been observed (e.g. documentation proper track-change history, proper labelling). It is therefore not necessarily the validation function which is responsible for these checks, but in any case, it should be clear to all involved internal control functions46 and properly documented who is responsible for ensuring compliance with which aspects of applicable laws, rules, regulations and standards, as discussed in paragraph [20]. However, as further mentioned in paragraph [79], the validation function is expected to review the documentation submitted to CAs in the context of a material change of a rating system (i.e. the application package that documents the change of the rating system).47 b. Assessment via validation challengers. As mentioned in paragraph [23.a], the validation function is expected to form an opinion on the accuracy and consistency of the rating system as a whole.48 This implies that, in the case where statistical tests are performed at a more granular level (e.g. for specific years or specific grades or pools), or when multiple tests or metrics are calculated, the validation function is expected to develop an aggregation methodology to deliver an overall assessment of the performance of the rating system as a whole. During this aggregation, specific attention is expected to be retained on the results of the assessment where a high share of obligors and exposure values are observed (in the development and application portfolio). 26. Consistency of the validation tasks in the performance assessment. The assessment of the performance of the rating systems should be performed ‘consistently and meaningfully’,49 and as such the institution should define and implement validation methods and procedures that are consistent across rating systems as well as over time.50 It is considered as best practice ensure to the changes in the validation policy are recorded and highlighted, both for the methods and the data used (data source and periods covered). Nevertheless, institutions are expected to seek for state-of-the art validation techniques as well as to develop a targeted validation framework (e.g. processes, tests or frequencies) for a specific type of portfolios or rating systems with specific challenges, including using data appropriate to the portfolio.51 Particular validation challenges are further discussed in the section [6 Focus on specific validation challenges]. 27. Assessment of the materiality of a model change or extension. In the context of the development activities that occurred since the last validation, the validation function should review the materiality of all rating system changes and extensions, and their combined 46 Paragraph 169 of the GL on internal governance 47 Article 8 of the CDR on model changes 48 Article 185(a) of the CRR 49 Article 185(a) of the CRR 50 Article 185(d) of the CRR, Article 12(d) of the CDR on assessment methodology 51 Article 185(c) of the CRR SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 26 effects.52 This includes a qualitative assessment, using the relevant annexes of the CDR on model changes, and the quantitative assessment, using the thresholds defined in Article 4(1)(c) of the CDR on model changes. This quantitative assessment requires the institution to be in a position to estimate the own funds requirements resulting from its updated risk parameter estimates, as well as to be in a position to estimate the own funds requirements resulting from the last version of the model before the implementation of the change. CONTEXT BOX 1: ASSESSMENT OF THE PROCESS-RELATED ASPECTS OF RATING SYSTEM CHANGES BY IA The IA is expected to review the process-related aspects of model changes as regard to their identification, notification and classification (i.e. change in the interpretation and implementation of the requirements from the CDR on model changes). In any case, the IA is expected to check the process ensuring that one material extension or change is not split into several changes or extensions of lower materiality.53 In any case, the register of rating systems should cover all changes and extensions, and include their impact as part of the description of the change category assigned in accordance with the CDR on model changes.54 52 Article 11(2)(d) of the CDR on assessment methodology 53 Article 3(3) of the CDR on model changes 54 Article 32(2)(c) and (d) of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 27 3. Validation content 28. Purpose of this section. This section recalls the main dimensions on which the validation function is expected to have an opinion, without listing the exact validation tasks and analyses to be conducted to form such opinions. In any case, as mentioned in paragraph [25], this section does not prevent institutions from developing additional tools and methods. This is because, as stated in paragraph [26], the validation framework to be defined by each institution (e.g. processes, tests or frequencies) is expected to be tailored to the specificities of a given type of exposures or rating system.55 3.1 Assessment of the core model performance 29. Dimensions of the assessment of the core model performance. One of the objectives of the validation function is to assess the core performance of the rating system.56 As such, this assessment can be broken down using the structure of the CRR, i.e. distinguishing between the performance in terms of risk differentiation and risk quantification:57 a. Risk differentiation: The model should allow for a meaningful differentiation of risk58 to ensure grouping of sufficiently homogenous exposures into the same grade or pool. To this end, the validation of a model is expected to evaluate in particular its discriminatory power, as well as the homogeneity within and heterogeneity across grades or pools.59 b. Risk quantification: The estimates should meet all regulatory requirements.60 To this end, the validation of the risk parameter estimates should include a comparison of realised DR with estimated PDs for each grade or pool, and analogous analysis for LGDs and CFs where institutions received permission to use own estimates for those risk parameters,61 taking into account the rating philosophy.62 For LGD and CF estimates, this should include an assessment of their appropriateness for an 55 Article 142(1) point (1) and (2) of the CRR 56 Article 11(2)(c) of the CDR on assessment methodology 57 Articles 144(1)(a) and 185(a) of the CRR 58 Article 170(1) and (3) of the CRR 59 Articles 170(1)(a),(d) and 170(3)(c) of the CRR, Articles 36 of the CDR on assessment methodology and paragraphs 69 and 130 of the GL on PD and LGD estimation. 60 Articles 178 to 184 of the CRR 61 Article 185(b) of the CRR. For the rest of the handbook, unless specified otherwise, the requirements on the LGD and CF models apply only to institutions which have received permission to use own estimates of those risk parameters for the respective type of exposures (i.e. within the respective rating system). 62 Article 12(f) of the CDR on assessment methodology, paragraph 66(c) of the GL on PD and LGD estimation SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 28 economic downturn where those estimates are more conservative than the long-run average.63 30. Calculation of IRB metrics. The validation function is expected to assess the performance of the rating system using regulatory definitions. To this end, the IRB metrics used to assess the core model performance are expected to be calculated according to the regulatory requirements (and it is expected that the related data requirements are fulfilled).64 This includes the calculation of the realised DR,65 the economic loss and realised LGD66 as well as the realised CF,67 while the definition of default may be treated separately (see Interaction Box [3]). In practice, the validation function is expected to form an opinion on the compliance of the IRB metrics used by CRCU as part of the model development, risk quantification and the review of estimates. When the validation function does calculate these IRB metrics by itself to run the statistical tests and validation challengers, it is expected to compare its own IRB metrics as well as the results of the analyses with those that are derived by the CRCU. INTERACTION BOX 3: ASSESSMENT OF THE DEFINITION OF DEFAULT The assessment of the definition of default may involve multiple analyses, which are not necessarily conducted by the same function: 1. The validation function may not necessarily perform the assessment of the compliance of the internal criteria used for the identification of defaulted exposures with the regulation68. Nevertheless, the responsibility of the review for these assessments should be clear to all involved internal control functions; 2. The validation function may not necessarily perform the assessment of the correct implementation of the default definition, and in particular the documentation, the implementation in the IT systems69 and the identification and monitoring of the technical past due situations,70 Nevertheless, the responsibility of the review for these assessments should be clear to all involved internal control functions; 63 Articles 181(1)(b) and 182(1)(b) of the CRR, Commission Delegated Regulation (EU) 2021/930, GL on downturn LGD estimation and GL on LGD estimation. 64 Sections 5.3.1, 6.1.2 and 7.1.3 of the GL on PD and LGD estimation 65 Article 4(1) point (78) of the CRR, Article 46(1) of the CDR on assessment methodology, section 5.3.2 of the GL on PD and LGD estimation, Q&A_2019/4599 66 Articles 5 point (2), 175(3), 181(1)(i), 182(1)(c) of the CRR, Article 48(h), 49 and 54(d) of the CDR on assessment methodology, sections 6.3.1 and 7.3.1 of the GL on PD and LGD estimation 67 Article 182(1)(c) of the CRR, Article 48(h) and 54(d) of the CDR on assessment methodology and section 6.3.1.2 of the GL on PD and LGD estimation 68 Articles 175(3) and 178 of the CRR, Commission Delegated Regulation (EU) No 2018/171, Chapter 6 of the CDR on assessment methodology and the Guidelines on the application of the definition of default (EBA/GL/2016/07) 69 Chapter 10 of the GL on the definition of default 70 Paragraphs 23 and 24 of the GL on the definition of default SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 29 3. The definition of default may trigger some issues in the model development or risk quantification (e.g. representativeness, reconstruction of historical time series), which are on the other side expected to be assessed by the validation function, including the related appropriate adjustments and margin of conservatism (MoC). As such, the validation function is expected to review the documentation related to the definition of the default in order to understand the changes impacting the RDS, as well as in the context of a material rating system change as part of the review of the documentation submitted to the CA (see Interaction Box [2]). 3.1.1 Risk differentiation 31. Dimensions of the assessment of the risk differentiation. The validation function is expected to form an opinion on two dimensions (see Context Box [2]) to assess the risk differentiation performance of a model: a. The consistency and comprehensiveness of the rating assignment process; b. The accuracy of the rating assignment in the model development. 32. Outcome of the assessment of the model. In any case, the validation function is expected to be confident enough that all the deficiencies observed in the risk differentiation are sufficiently 71 Article 171(2) of the CRR, section 8.1 of the GL on PD and LGD estimation 72 Article 171(2) of the CRR, paragraph 74 of the GL on PD and LGD estimation 73 To note, this conservative rating assignment is different to the conservatism added to the estimates (the latter being so called ‘margin of conservatism’). CONTEXT BOX 2: ACCURACY IN THE DEVELOPMENT SAMPLE AND CONSERVATISM IN THE APPLICATION PORTFOLIO OF THE RATING ASSIGNMENT [Q&A 2021/5761, Q&A 2019/5029] To ensure that RWEA are calculated in a conservative way, IRB models generally need to be applied in a conservative manner, i.e. the rating assignment process itself is required to be conservative when relying on insufficient information.71 This requirement is frequently implemented by e.g. using conservative assumption(s) in case of a lack of information or missing risk drivers. In contrast to the application, the model development should ensure that the risk quantification is based on an accurate rating assignment.72 As such, a conservative rating assignment73 could lead to a biased default rate calculation and risk parameter estimation when used subsequently for the calibration sample. This aspect is further developed in the Focus Boxes [3] and [4]. SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 30 limited according to its criteria defined in the validation policy (including the fixed targets and tolerance thresholds mentioned in paragraph [23.c] and the Focus Box [2]).74 33. Assessment of the consistency and comprehensiveness of the rating assignment process. In order to validate the on-going rating assignment process, the validation function is expected to review the framework used for the rating assignment process, such that: a. The rating assignment process is adequately documented and understandable by a third party, such that it can be performed in a consistent manner (see Interaction Box [4]), both in terms of definition of the scope of application of the rating system as well as in terms of definition of rating criteria (including the assignment to a ranking method and to a calibration segment);75 b. The assignment process is performed in a comprehensive manner.76 In practice, the validation function is expected to analyse the policy for the treatment of those cases where the obligor or facility could not be assigned to an obligor grade or pool based on the ‘standard‘ rating assignment and assess the materiality of these cases in the application portfolio (along with the resolution plan proposed by the CRCU).77 This includes missing ratings and cases where the assignment was based on outdated or missing data, or where the assignment could not be renewed in time (outdated ratings). A good practice observed in institutions is to make the assessment of materiality in terms of exposure value and RWEA, as well as in terms of number of obligors or facilities to monitor the magnitude of the deficiencies. 74 As explained in the Context Box [2], deficiencies or lack in performance in the risk differentiation can generally not be covered directly by additional conservatism. It is nevertheless likely that a lack of homogeneity within grades or pools will indirectly lead to a higher uncertainty at grades or pools level in the risk quantification process and will increase the MoC added to the best estimates. 75 Articles 142(1)(1) and (2) and 171(1) of the CRR, Articles 24(1)(a), (c), (d), 31(2)(b), 32(2)(a) of the CDR on assessment methodology, section 4.1 of the GL on PD and LGD estimation. 76 Articles 24(1) and 25(2) of the CDR on assessment methodology 77 Article 171(2), 172(2) and 173(1) of the CRR, Article25(3) of the CDR on assessment methodology and section 8.1 and paragraph 75 of the GL on PD and LGD estimation. N.B.: these cases are not expected to be dealt with the standardised approach, but instead rather through a conservative IRB treatment (i.e. via a downgrade of the exposure). SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 31 34. Dimensions of assessment of the accuracy of the rating assignment. IRB models are in practice based on either statistical models or other mechanical methods, in order to assign exposures to obligors or facilities grades or pools. In this context, the validation function should assess the input data, challenge all methodological choices used during the risk differentiation, and perform statistical tests on the model performance, in order to form an opinion on two dimensions of the accuracy of the rating assignment, namely: 78 Article 173 of the CRR 79 Articles 173(1)(b) and 173(2) of the CRR, Article 25(2) of the CDR on assessment methodology INTERACTION BOX 4: ASSESSMENT OF THE HUMAN JUDGMENT IN THE ASSIGNMENT PROCESS The validation function is expected to review the incorporation of subjective data in the model (i.e. the specification of the model) for the assignment of the exposures to grades or pools. This assessment is expected to be conducted along two dimensions: 1. The clarity of the definitions, processes and criteria defined by the CRCU to ensure the consistency of the rating assignment, as described in paragraph [33.a]; 2. The integration of the human judgment in the overall rating assignment, as mentioned in paragraph [36.b]. On the other hand, the validation function may not necessarily perform the evaluation of the actual implementation (e.g. integrity of the assignment process).78 Nevertheless, the responsibility of the review for these assessments should be clear to all involved internal control functions. This includes: 1. Governance aspects, such as the assessment of the independence of staff and management responsible for the final approval of the assignment from the ones responsible for the origination or renewal of exposures; 2. Application aspects, such as the evaluation of the consistency between the framework and the actual implementation of the human judgment (i.e. the application of the model specifications). In addition, the IA may assess the rating assignment review and in particular the frequency and adequacy of the assignment process (in order to ensure that all outdated ratings are properly identified as such).79 SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 32 a. The discriminatory power of the model, i.e. its capacity to efficiently discriminate riskier obligors or facilities from less risky ones, based on the difference in the level of default (for the PD), loss given default (for the LGD) and conversion (for the CF) risk;80 b. The homogeneity within each grade or pool, in terms of default, loss given default and conversion risk, and the heterogeneity between grades or pools, in terms of distributions’ overlaps of default, loss given default and conversion risk between all grades or pools.81 35. Assessment of the input data. The assessment of the input data is expected to include: a. An opinion on the data quality of the full RDS, as part of the validation function activities described in section [3.2.1]; b. A review of the completeness of the RDS,82 in terms of scope (obligors, facilities and default identification) and information (values of the risk drivers at the relevant dates, data necessary for calculating realised DR, realised LGD and realised CF and any other relevant data used in the risk differentiation);83 c. A review of all the procedures applied to the data used for the development of the model, including data collection, data cleansing, data processing (e.g. normalisation, treatment of collinearity) and data estimation (e.g. cash flow projections used for specialised lending). A good practice observed in institutions is to complement the review of the framework used for these estimated input data with back-testing comparisons between these estimations (including the projections which go beyond the one-year time-horizon) and the subsequently realised values (out-of-time (OOT) validation tests); d. The analysis of the representativeness of the development sample vis-à-vis the application portfolio.84 The validation function is expected to include in its assessment the dimensions mentioned in the GL on PD and LGD estimation: the scope of application, the definition of default, the distribution of relevant risk characteristics as well as the lending standards and recovery policies.85 This assessment should be done in accordance with the process and methods defined in the validation policy.86 80 Article 170(1) and (3) of the CRR 81 Articles 170(1)(a), (d) and 170(3)(c) of the CRR, Articles 36 of the CDR on assessment methodology and paragraphs 69 and 130 of the GL on PD and LGD estimation 82 Article 174(b) of the CRR 83 Sections 5.2.1, 6.1.2 and 6.2.1 of the GL on PD and LGD estimation 84 Article 174(c) of the CRR 85 Article 37(2) of the CDR on assessment methodology and section 4.2.3 of the GL on PD and LGD estimation 86 Section 4.2.2 of the GL on PD and LGD estimation SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 33 It is expected to subsequently evaluate the measures taken by the CRCU to deal with deficiencies in these areas. 36. Assessment of the modelling choices and specifications. The assessment of the modelling choices and specifications87 should ensure that the chosen input variables form a reasonable and effective basis for the resulting predictions and that the model does not have any material bias.88 As such, the validation function is expected to have a good understanding of the documentation and the features of the model, including its scope of application, limitations and weaknesses, main and alternative assumptions or approaches to those finally chosen, in order to effectively challenge them.89 In this context, the validation function is expected to assess: a. The selection process and related outcomes of risk drivers and rating criteria in terms of predictive power, such that all relevant information is taken into account. 90 In practice, the choices are expected to be consistent with the results of the statistical methods further described in paragraph [37]91 and with business expectations.92 This analysis is expected to include a review of: - The minimum list of potential risk drivers to be considered mentioned in the regulation;93 - Extremely high values of realised LGDs, as it could require the identification of specific risk drivers;94 - Where external ratings are used as primary risk driver, whether all relevant information has been considered;95 - The use of third-party ratings,96 by challenging the appropriateness of the use of rating transfers, the related use of overrides (as further discussed below) or the related use of risk drivers; - The framework for the treatment of connected clients, ensuring in addition that cases where the obligors are assigned to a better grade 87 Article 174(d) of the CRR 88 Article 174(a) of the CRR 89 Article 175(4)(a) of the CRR, Article 38 of the CDR on assessment methodology 90 Article 171(2) of the CRR, Article 24(1)(e) and (f) of the CDR on assessment methodology 91 Article 33(1)(c) of the CDR on assessment methodology 92 Article 171(1)(c) of the CRR, Article 33(1)(b) of the CDR on assessment methodology 93 Article 170(4) of the CRR, Article 33(2) of the CDR on assessment methodology, sections 5.2.2 and 6.2.1 of the GL on PD and LGD estimation. 94 Paragraph 162 of the GL on PD and LGD estimation 95 Article 171(2) of the CRR 96 Section 5.2.3 of the GL on PD and LGD estimation SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 34 than their parent entities are intended to be exceptional and duly justified.97 b. Any functional form or ‘hyperparameters’98 used in the model development to aggregate all the risk drivers. This includes how the statistical model and human judgement (see Context Box [3]) are combined to derive the final assignment of exposures to grades or pools.99 This assessment is expected to comprise an evaluation of the theoretical framework (such that only additional information not considered in the statistical model is incorporated via human judgment in a consistent way – see Interaction Box [4])100 and the comprehensiveness of its documentation.101 CONTEXT BOX 3: THE DIFFERENT FORMS OF HUMAN JUDGMENT The human judgement refers to three particular notions: 1. The human judgement applied in the development of the model;102 2. The human judgement applied in the process of assignment of exposures to grades or pools, in the form of subjective input data (such as qualitative variables based on an expert-based assessment);103 3. The human judgement in the form of overrides, either of input or outputs, of the assignment process of exposures to grades or pools.104 c. How obligor and facility grades or pools are defined, such that the methodology used to define grades or pools ensures the homogeneity of obligors and exposures assigned to the same grade or pool over time.105 In particular, the validation function is expected to assess whether: 97 Article 172(1)(d) of the CRR and Article 24(3) of the CDR on assessment methodology 98 See [Discussion Paper on Machine Learning] pages 16 and 17 99 Article 174(e) and Article 31(5)(d) of the CDR on assessment methodology 100 Article 39(b) and (d) of the CDR on assessment methodology 101 Article 39(c)(i) and (d) of the CDR on assessment methodology 102 Article 39(a) and (d) of the CDR on assessment methodology, paragraph 35(a) the GL on PD and LGD estimation 103 Paragraph 201(a) the GL on PD and LGD estimation 104 Article 24(2) and 39(b), (c) and (d) of the CDR on assessment methodology, paragraph 201(a) and (b) of the GL on PD and LGD estimation 105 Article 170(1) and (3)(c) of the CRR and Article 36 of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 35 - The definition of grades or pools is sufficiently clear, and the rating scale is not too granular in order to allow for a consistent assignment of obligors or facilities posing similar risks to the same grade or pool;106 - The number of rating grades meets the regulatory requirements in terms of minimum number: the obligor rating scale for exposures to corporates, institutions and central governments and central banks shall have a minimum of seven grades for non-defaulted obligors and one for defaulted obligors and at least four grades for non-defaulted obligors and at least one grade for defaulted obligors for specialised lending exposures treated under the slotting approach;107 - The number of rating grades is not excessive: the number of exposures in a given grade or pool is sufficient to allow for meaningful quantification and validation of the default or loss characteristics at the grade or pool level;108 as such, a high number of rating grades can be an indication of a lack of heterogeneity between grades or pools. FOCUS BOX 4: ACCURACY OF THE RATING ASSIGNMENT IN THE CONTEXT OF MODEL (RE-) DEVELOPMENT [Q&A 2021/5761 AND Q&A 2019/5029] In the context of a model development, the rating assignment of exposures in past years may involve some operational challenges, in particular when the model cannot retrospectively be run in a fully automated manner (e.g. use of human judgement in the form of qualitative variables or overrides). In this context, the validation function is expected to assess the assumptions and limitations of the approach chosen by the CRCU to determine the rating assignment of past exposures, and get an opinion of their impact on the risk quantification (in terms of bias or additional uncertainty). In practice, this may involve: 1. If an institution chooses to build the risk quantification on retrospectively calculated ratings (based consistently on the model to be calibrated), the validation function is expected to assess whether these retrospectively performed rating assignments were incorporating conservative adjustments in the ratings. For example, information on overrides or other forms of human judgement may not be available and be derived via assumptions from the overrides performed as part of the old model; 106 Article 171(1)(c) of the CRR 107 Articles 170(1) and 170(2) of the CRR, Article 34(1)(a) of the CDR on assessment methodology. To note, this assessment is also related to the validation challenger provided in paragraph [38.f]. 108 Article 170(3)(b) of the CRR, Article 34(1)(c) and (d) of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 36 2. If an institution chooses to base the risk quantification on historically performed ratings (even potentially stemming from different versions of a rating model), the validation function is expected to check if appropriate measures were taken in model development, or risk quantification respectively, to remove conservative assumptions from historical ratings. This analysis should then be considered for the assessment of the appropriateness of the MoC as further described in the Focus Box [4] in the section risk quantification. 37. Quantitative validation challenger analyses. The empirical assessment of the model performance is expected to be based on rigorous statistical tests (see Interaction Box [5]). These tests are expected to be documented in the validation policy, as mentioned in paragraph [23], be sound and adequate (including using IRB metrics as defined in the regulation as mentioned in paragraph [30]) and consider all available data (see Focus Box [1]). In addition, they are expected to: a. Cover the dimensions mentioned in paragraph [34]. As such, they are expected to be conducted at all relevant levels. In this context, it is considered as best practice to complement the empirical assessment based on the final rating by a deep dive analysis based on the intermediate steps of the model (e.g. qualitative or quantitative sub modules, evaluation before and after overrides) for the dimension in paragraph [34.a]. For the evaluation of the selection of risk drivers and final ranking, the evaluation is expected to be performed in particular at calibration segment level (when used). For the evaluation of the homogeneity and heterogeneity, the evaluation is expected to be performed within (homogeneity) and across (heterogeneity) grades; b. Allow for an evaluation of the performance of the model under various economic conditions.109 INTERACTION BOX 5: STATISTICAL ASSESSMENT OF THE MODEL BETWEEN CRCU AND VALIDATION FUNCTION Institutions shall establish a rigorous statistical process including out-of-time (OOT) and out-of- sample (OOS) performance tests for validating the model.110 In particular, institutions need to develop robust models to allow for stable model use across time (and thus to a certain extent across changing environment or economic conditions). As such, these tests are based on two different samples, as illustrated in a schematic view in Figure 2. 109 Article 35 of the CDR on assessment methodology 110 Article 175(4)(b) of the CRR SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 37 Figure 2 : Schematic view of the development sample versus Validation samples These tests are expected to be used by the CRCU as part of the model development. However, as further developed in the sections [4 First validation activities] and [5 On-going validation ], the validation function is expected to or may have to perform additional tests to form its opinion on the performance of the model. NB: as these tests are related to the development of the model (risk differentiation), the above requirements are without prejudice to the need to use all relevant data when it comes to the risk quantification.111 38. Validation challengers. In addition to these statistical tools, the validation function is expected to assess the following: a. The impact of overrides on the performance of the rating assignment process. To this end, the validation function is expected to assess the performance of the model before and after overrides; 112 b. The number of overrides applied on the model outcomes. This could indicate a weakness in the rating model in terms of effectiveness to consider all relevant information.113 To this end, the validation function is expected to assess their materiality (in terms of number of obligors or facilities, their exposure value and their related RWEA) for the application portfolio, and review the threshold set as maximum acceptable rate of overrides for the model;114 111 Articles 179, 180, 181 and 182 of the CRR 112 Article 172(3) of the CRR, Article 24(2)(d) of the CDR on assessment methodology, paragraphs 206 and 207 of the GL on PD and LGD estimation 113 Articles 170(4) and 172(1) of the CRR, Article 24(2)(d) and (e) of the CDR on assessment methodology 114 Paragraph 205 of the GL on PD and LGD estimation SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 38 c. The stability of the ratings assigned to individual obligors or facilities (using for instance migration matrices) in relation to the economic cycle, to understand the core feature of the model with respect to the rating philosophy.115 The outcome of this analysis is expected to be compared to the expected outcome due to the rating philosophy.116 In case of material deviation, this could be an indication of a deficiency in the model, such as missing risk drivers or inadequate definition for grades or pools (i.e. lack of homogeneity or heterogeneity). In addition, the validation function is expected to be aware of the rating philosophy and rating stability properties of the model, and their adequacy for the respective scope of application, the risk quantification methodologies used117 and their impact on the stability of risk parameters.118 The result of this analysis is expected to be considered for back-testing purposes, as mentioned in paragraphs [46] and [47]; d. The relationship between obligor grades in terms of the level of default risk.119 In particular, this assessment can be done by analysing the monotonicity of the one-year DR or long-run average DR. The validation is expected to have a good understanding of the reasons for the non-monotonicity and is expected to take into account this analysis in particular when assessing the discriminatory power of the model. A similar analysis can be conducted for the realised LGD or realised CF in the case where rating grades are used; e. Other relevant external data sources, where available.120 For this purpose, where a sufficient number of external ratings is available, it is considered as best practice to use them as a challenger. As such, the comparison with the ranking derived from the external ratings is expected to be used as a tool to search for potential weaknesses in terms of effectiveness to consider all relevant information; f. The potential concentration in rating grades, which if unwarranted, could be an indication of a lack of homogeneity within grades or pools and therefore of missing risk drivers.121 In addition, concentration in rating grades can give rise to data scarcity related issues, which are further discussed in section [6.3 Focus 3: validation in the context of data scarcity]. 115 Article 12(f) of the CDR on assessment methodology, section 5.2.4 of the GL on PD and LGD estimation 116 In case of the application of paragraph 90 of the GL on PD and LGD estimation, the calibration can have an impact on the rating assignment and should therefore be considered. 117 Paragraph 66(c) of the GL on PD and LGD estimation 118 Article 11(2)(c) of the CDR on assessment methodology 119 Article 170(1)(c) of the CRR 120 Article 185(c) of the CRR 121 Articles 170(1)(d), 170(1)(f) and 170(3)(b) of the CRR, Article 34(1)(b) of the CDR on assessment methodology, paragraph 87(c) of the GL on PD and LGD estimation SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 39 3.1.2 Risk quantification 39. Dimensions of the assessment of the risk quantification. The validation function should assess the input data, challenge all methodological choices used during the risk quantification and perform statistical tests between estimates and observed data, in order to form an opinion on the three dimensions of the risk quantification performance of the model: a. The accuracy of the best estimates122 in terms of alignment with the long-run averages per grades or pools, in relation with the observed realised DR,123 LGD124 and CF respectively;125 b. The conservatism of the risk estimates, taking into account in particular the quantification of the MoC;126 c. For the LGD and CF parameters, the appropriateness of the estimates for an economic downturn, if those are more conservative than the long-run average.127 40. Assessment of the input data. The input data used for the risk quantification is expected to be reviewed to ensure that any uncertainty related to a deficiency is sufficiently covered through a MoC.128 It is expected to include: a. An opinion on the data quality of the full RDS, as part of the validation function activities described in [section 3.2.1]; b. A review of the completeness of the RDS, in terms of historical experience and empirical evidence in order to check that all the available data was considered for the risk quantification, as well as in terms of scope (obligors, facilities and default identification) and information (values of the risk drivers at the relevant dates, data necessary for calculating the realised DR, realised LGD and realised CF and any other relevant data used in risk differentiation or risk quantification);129 c. A review of all the procedures for data collection and data cleansing applied to the data used by the rating system and the compliance of the data preparation with the regulatory requirements.130 For these years used for the risk quantification, the validation function is expected to check that all observations have been taken into 122 Paragraph 38 of the GL on PD and LGD estimation 123 Article 180(1)(a) and Article 180(2)(a) of the CRR 124 Article 181(1)(a) of the CRR 125 Article 182(1)(a) of the CRR 126 Article 179(f) of the CRR, section 4.4.3 of the GL on PD and LGD estimation 127 Articles 181(1)(b) and 182(1)(b) of the CRR 128 Section 4.4, and in particular paragraph 37(a) of the GL on PD and LGD estimation 129 Article 174(b) of the CRR, Article 42(1)(a) of the CDR on assessment methodology, sections 5.3.1 and 6.1.2 of the GL on PD and LGD estimation 130 Article 11(2)(a) of the CDR on assessment methodology SUPERVISORY HANDBOOK ON THE VALIDATION OF IRB RATING SYSTEMS 40 account, with the exception of exclusions in the specific circumstances mentioned in the regulation (i.e. wrong rating model assignment or default identification),131 and that those exclusions and data cleansing are duly documented.132 In particular, the treatment of the cases with non-standard or outdated ratings (as referred to in paragraph [33.b]) are expected to be carefully reviewed, as further described in the Focus Box [4]. With respect to the LGD parameters, the realised LGDs of the cases with no loss or with a positive outcome should be floored at 0% for the purpose of the calculation of the observed average LGD and the estimation of the long-run average LGD.133 FOCUS BOX 5: ACCURACY OF THE RATING ASSIGNMENT IN THE CONTEXT OF MODEL (RE-) DEVELOPMENT [Q&A 2021/5761 AND Q&A 2019/5029] A key input during the risk quantification is the rating assigned to each obligor or facility based on the model developed during the risk differentiation. As mentioned in the previous section, the rating assignment used for the risk quantification should be as accurate as possible to ensure homogenous grades or pools. In particular, in the context of model development, the rating assignment of obligors or facilities of the past years should not include the additional conservatism added due to insufficient information (as per section 8.1 of the GL on PD and LGD estimation), and may also require some assumptions (with hence associated limitations) by the CRCU to determine the rating assignment of past obligors or facilities when human judgement was involved. In this context, the validation function is expected to assess the materiality of the cases with non- standard or outdated ratings, as well as the severity of the deficiency (in terms of magnitude of the uncertainty on the real rating of the obligor or facility) in the calibration segment, and check that any related uncertainty is sufficiently covered by a MoC.134 d. A review of the representativeness135 of the data used for the risk quantification. The validation function is expected to develop statistical tests or metrics for this task,136 and check that any related uncertainty is sufficiently covered by a MoC.137 As such, the validation function is expected to have an opinion on the representativeness of the samples used for risk quantification (i.e. samples used to calculate long-run- 131 Paragraph 71 of the GL on PD and LGD estimation 132 Article 42