On January 17, 2024, the New York Department of Financial Services (the “Department”) published a letter containing proposed guidance regarding the use of artificial intelligence systems (“AIS”) and external consumer data and information sources (“ECDIS”) in insurance underwriting and pricing. Below is a high-level summary of the requirements outlined in the letter, and our takeaways regarding the letter as it relates to the broader regulatory landscape.  This is the second recent regulatory effort in the insurance AI arena, after the Colorado Department of Insurance.  As described in detail below, the Department focuses intently on the types of Governance, Fairness Principles and Transparency that the National Association of Insurance Commissioners identified in their Principles on Artificial Intelligence. 

Applicability: 

The letter is directed to “all insurers authorized to write insurance in New York state, licensed fraternal benefit societies and the New York State Insurance Fund.” 

Definitions:

• Artificial Intelligence System: any machine-based system designed to perform functions normally associated with human intelligence, such as reasoning, learning, and self-improvement, that is used, in whole or in part: (i) to supplement traditional medical, property, or casualty underwriting or pricing; (ii) as a proxy for traditional medical, property, or casualty underwriting or pricing; or (iii) to establish “lifestyle indicators” that may contribute to an underwriting or pricing assessment of an applicant for insurance coverage.
• External Consumer Data and Information Sources: includes data or information used, in whole or in part: (a) to supplement traditional medical, property, or casualty underwriting or pricing; (b) as a proxy for traditional medical, property, or casualty underwriting or pricing; or (c) to establish “lifestyle indicators” that may contribute to an underwriting or pricing assessment of an applicant for insurance coverage. 

Overview:

The letter aims to identify some potential issues with the use of AIS and ECDIS and outline possible solutions, specifically outlining three areas of focus: Fairness Principles, Governance and Risk Management, and Transparency. 

• Fairness Principles: An insurer should only use ECDIS or AIS for underwriting or pricing purposes if the insurer can establish that the data source or model does not use, and is not based in any way, on any protected class pursuant to Insurance Law Article 26 or if such use would permit any unfair discrimination or otherwise violate New York Insurance Law and its regulations (together “Insurance Law”). In conducting this analysis insurers should consider:
  • Data Actuarial Validity: Insurers should be able to demonstrate that the ECDIS are supported by accepted actuarial standards and based on actual or reasonably anticipated experience. Insurers should further be able to demonstrate that the ECDIS employed do not serve as a proxy for any protected classes that may result in unfair or unlawful discrimination. 
  • Unfair and Unlawful Discrimination: Insurers are responsible for complying with anti-discrimination laws regardless of whether the insurer has collected data itself or relies on external vendors. A comprehensive assessment is required to determine whether the underwriting or pricing guidelines derived from ECDIS or AIS unfairly discriminate. This assessment includes steps such as assessing possible adverse effects and seeking less discriminatory variables or methodology that would meet the insurer’s legitimate business needs. 
  • Analyzing for Unfair or Unlawful Discrimination: Insurers should document their testing methodologies and be prepared to make such documentation available to the Department upon request. Insurers should document the frequency of testing prior to putting the AIS into production, as well as testing thereafter. Insurers must use Quantitative and Qualitative metrics in the analysis. 
    •  Quantitative assessment metrics may include:
      • Adverse impact ratios; 
      • Denials odds ratios;
      • Marginal effects;
      • Standardized mean differences;
      • Z-tests and T- tests; and 
      • Drivers of disparity. 
    • Qualitative assessments are in addition to quantitative tests and should be able to “explain, at all times, how the insurer’s AIS operates and to articulate the intuitive logical relationship between ECDIS and other model variables with an insured or potential insured individual’s risk.” 
• Governance and Risk Management: The Department letter identifies that the requirement under 11 NYCRR Section 90.2 for an insurer to have a corporate governance framework applies to ECDIS and AIS. The letter outlines essential principles for insurers to establish the corporate governance framework that “provides appropriate oversight of the insurer’s use of ECDIS and AIS to ensure compliance with the Insurance Law”:
  • Board and Senior Management Oversight: An insurer’s board of directors, or other governing body, plays a key role in providing oversight for the insurer’s activities, including the development and management of ECDIS and AIS. The board may delegate specific duties but such delegation must be clearly defined, and appropriate lines of reporting should be established. Reporting should include the facts necessary for the board to understand the insurer’s activities and risk as they relate to the use of ECDIS and AIS. Senior management is tasked with the day-to-day implementation of the insurer’s strategies, including by engaging a cross-functional management committee with representatives from key function areas across the organization. 
  • Policies, Procedures, and Documentation: Insurers using ECDIS or AIS should formalize their development and management processes through written policies and procedures which are then reviewed and approved by the insurer’s board of directors or senior management. The policies and procedures must be updated at least annually. Policies and procedures should include defined roles and responsibilities, monitoring and reporting requirements, and training for relevant personnel. Insurers should maintain a detailed list of documentation for all their use of AIS, whether it was developed internally or supplied by a third party. Insurers must also be prepared to respond to consumer complaints or inquiries regarding the use of AIS and ECDIS and maintain records of such complaints. All such documentation is to be provided to the Department upon request. 
  • Risk Management and Internal Controls: 
    • Manage risks at each stage of AI system lifecycle, individually and in aggregate. These risks should be managed with an existing enterprise risk management function or separately as part of an independent program. 
    • Have standards for model development, implementation, use, and validation. Promote independent review to challenge the AI risk analysis, validation, testing, development, and other processes related to ECDIS and AIS development and risk management.
    • Have qualified personnel with clear roles and responsibilities oversee AI risk management.
    • Extend the internal audit oversight to assess the overall effectiveness of the AIS and ECDIS risk management framework. Such auditing may include:
      • Verifying compliance with policies and procedures
      • Verifying records to test for weakness in validation activities 
      • Assessing the accuracy and completeness of documentation
      • Evaluating the processes for establishing and monitoring internal controls
      • Assessing supporting operational systems
      • Assessing potential biases in ECDIS or other data that may result in unfair or unlawful discrimination
      • Assessing the sufficiency of reporting to board or governing body 
  • Third Party Vendors: The use of third party vendors does not eliminate insurer responsibility for ECDIS and AIS. Insurers still retain responsibility for understanding any tools, ECDIS, or AIS and establishing appropriate oversight of third party vendors. Insurers must develop written standards, policies, procedures, and protocols for the appropriate oversight of third-party vendors, as well as to report to, remediate and eliminate incorrect information held by, such third-party vendors. 
• Transparency: Transparency is highlighted, requiring that insurers using ECDIS or AIS include details about all information upon which the insurer based its decision in its notice to the insured. The notice should include whether the insurer uses AIS in its underwriting or pricing process, whether the insurer uses data about the person obtained from external vendors, and that the insured has the right to request information about the specific data that resulted in the underwriting or pricing decision.  The Department also reminds insurers in specific lines of the limits under existing laws concerning declination, refusal to insure, limits, or rate differential that are implicated by ECDIS and AIS, and identifies specific notifications to insureds. Most importantly, the Department states that the “failure to adequately disclose to the insured or potential insured any other specific reason or reasons for refusal, limitation, or rate differential may be deemed to be an unfair or deceptive act and practice in the conduct of the business of insurance and may be deemed to be a trade practice constituting a determined violation, as defined in Insurance Law section 2402(c), and in such case may be a violation of Insurance Law section 2403.”

Takeaway:

The Department’s letter confirms the trend in regulations toward enterprise risk frameworks and requiring senior management or board involvement in risk management.  As we previously wrote about in our analysis of the Colorado Division of Insurer’s Regulations, regulations in the privacy, cybersecurity, and, increasingly, the artificial intelligence spaces, have recently focused on board involvement or liability. Here, the Department goes a full step deeper into the risk management requirements than did the Colorado DOI. The Department’s specific audit and documentation requirements of an insurer,  make clear that risk management at all levels, including the board and senior executives, govern the intricacies of an AI Risk Management Program, from the top, down to individual systems. Along the same line, the use of a third party vendor does not eliminate insurer responsibility. Instead, insurers are required to comply with the Insurance Law and confirm compliance of any vendors, including confirming the vendor is not violating discrimination laws.   

Feedback Request: The Department is requesting feedback on its letter before March 17, 2024. Interested parties can submit comments to innovation@dfs.ny.gov with “Proposed Circular on the use of AI and ECDIS in Insurance Underwriting and Pricing” in the subject line.