New York City Issues Proposed Rules on Law Aimed at Curbing Artificial Intelligence Bias in Employment Decisions

Seyfarth Synopsis: On September 23, 2022, New York City’s Department of Consumer and Workplace Protection (“DCWP”) released the highly anticipated proposed rules implementing Local Law Int. No. 1894-A, which regulates the use of automated employment decision tools (“AEDT”) in hiring and promotion decisions and takes effect on January 1, 2023. The proposed rules attempt to address some of the ambiguities under the law by providing additional details on employer obligations. DCWP will hold a public hearing on the proposed rules on October 24, 2022 and is also accepting public comments on the proposed rules until that date.

As background, the New York City Council passed Local Law Int. No. 1894-A on November 10, 2021 to amend the City’s Administrative Code. The law was designed to prohibit employers and employment agencies from using an AEDT unless such tool has been subject to a bias audit within one year of the use of the tool, information about such audit is publicly available, and certain notices have been provided to employees or job candidates. You can read more about the law in our past coverage here.

The administrative rules being proposed aim to address some of the ambiguities under the current law and provide additional clarification on employer obligations. Several provisions are highlighted here:  (1) new and expanded definitions for certain terms found in the law, (2) clarification on the requirements for bias audits and the results of such audits that must be made publicly available, and (3) notices regarding the use of an AEDT that employers and employment agencies must provide to employees and candidates for employment.

What Is An Automated Employment Decision Tool

As currently defined, an AEDT is "any computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision making for making employment decisions that impact natural persons." While the proposed rules maintain this broad definition, the rules provide additional context by defining certain terminology and phrases found in the AEDT definition including: “machine learning, statistical modeling, data analytics, or artificial intelligence,” “simplified output,” and “to substantially assist or replace discretionary decision making.”

• Machine learning, statistical modelling, data analytics, or artificial intelligence: means a group of mathematical, computer-based techniques: (1) that generate a prediction (an expected outcome for an observation, such as an assessment of a candidate’s fit) or a classification (an assignment of an observation to a group, such as categorizations based on skill sets or aptitude), (2) where a computer at least in part identifies the inputs, the relative importance placed on those inputs, and other parameters for the models in order to improve the accuracy of the prediction or classification, and (3) for which the inputs and parameters are refined through cross-validation or by using training and testing data.
• Simplified output: means a prediction or classification as specified in the definition for “machine learning, statistical modelling, data analytics, or artificial intelligence.” A simplified output may take the form of a score (e.g., rating a candidate’s estimated technical skills), tag or categorization (e.g., categorizing a candidate’s resume based on key words, assigning a skill or trait to a candidate), recommendation (e.g., whether a candidate should be given an interview), or ranking (e.g., arranging a list of candidates based on how well their cover letters match the job description). It does not refer to the output from analytical tools that translate or transcribe existing text, e.g., convert a resume from a PDF or transcribe a video or audio interview.
• To substantially assist or replace discretionary decision making: means to rely solely on a simplified output (score, tag, classification, ranking, etc.), with no other factors considered, or to use a simplified output as one of a set of criteria where the output is weighted more than any other criterion in the set, or to use a simplified output to overrule or modify conclusions derived from other factors including human decision making.

These broad definitions mean that a wide variety of automated tools will likely be covered by these rules. Some examples include resume scanners that prioritize applications using certain keywords, employee monitoring software that rates employees on the basis of their keystrokes or other factors, virtual assistants or chatbots that ask job candidates about their qualifications and reject those who do not meet pre-defined requirements, and testing software that provides job fit scores for applicants or employees regarding their personalities, aptitudes, or cognitive skills.

Independent Bias Audits and Publication of Results

The proposed rules also address the lack of substantive requirements for bias audits as provided under the law.  As currently defined, a bias audit is “an impartial evaluation by an independent auditor” that includes testing the AEDT to assess its “disparate impact” on employees and applicants based on race, ethnicity, and sex.

Although the text of the law omitted a definition of what constitutes an independent auditor, the proposed rules define an independent auditor as “a person or group that is not involved in using or developing an AEDT that is responsible for conducting a bias audit of such AEDT.” While this seems to indicate that a third-party will be required to conduct the AEDT bias audits, the definition fails to specify the level of independence required (i.e., whether individuals within a company that do not use the AEDT can conduct the independent audit).

The rules also address the disparate impact mechanism employers and employment agencies need to use when conducting the bias audit. The proposed rules establish that an auditor must calculate the selection rate for each EEO-1 race, ethnicity, and sex reporting category (“EEO-1 demographic category”), and compare those selection rates to the most favored group to determine an impact ratio. In other words, the analysis must first identify the specific group with the highest selection ratio and then analyze the impact of the tool against every other relevant demographic category.  The examples set forth in the proposed rules suggest the required analyses are “cross-sectional” analyses based on the prescribed EEO-1 race/ethnicity and sex categories.

The proposed rules go on to explain that there are two distinct audits that must be run depending on the type of AEDT being utilized. The first involves situations where an AEDT selects individuals to move forward in the hiring process or classifies individuals into groups. Under this type of AEDT the bias audit must, at a minimum: (1) calculate the selection rate for each EEO-1 demographic category; (2) calculate the impact ratio for each  EEO-1 demographic category, and (3) where an AEDT classifies individuals into groups, the calculations in paragraphs (1) and (2) above must be performed for each such classification. On the other hand, when an AEDT scores applicants or candidates, the bias audit must, at a minimum: (1) calculate the average score for individuals in each category and (2) calculate the impact ratio for each category.

The proposed rules provide additional guidance on the requirements for posting AEDT audit results.  Before using a specific AEDT tool, employers and employment agencies in New York City must make the following information publicly available: (1) the date of the most recent bias audit for the AEDT being used, (2) a summary of the results, including the selection rates and impact ratios for all EEO-1 demographic categories, and (3) the date the employer or employment agency began using a specific AEDT. These requirements can be met by including a clearly identified hyperlink on the website of an employer or employment agency. The proposed rules require that this information remain posted for at least six months after last using such AEDT for an employment decision.

Even with this additional guidance, important questions with regard to the audit and publication provisions still remain and will need to be addressed during the notice and comment period. Among others, such questions include whether company or vendor personnel not involved with the use or development of an AEDT can qualify as an “independent” auditor, how the audits findings are to be utilized, and whether the AEDT must “pass” an audit to be utilized, and if so, what is considered a passing score under the NYC law. 

While certain provisions of the law clearly apply only to NYC residents, it is not clear whether the audits may appropriately include the results of decisions made outside of NYC or based on candidates or employees that do not reside in NYC.  Indeed, it is rarely the case that an AEDT is focused on such a narrow demographic region so the feasibility of limiting the results to “residents” of NYC may present unforeseen challenges to employers and vendors.

The proposed rules also appear to lack clarity on the timing of the audits. The text of the law specifies that AEDT tools cannot be used unless the tool has been the subject of a bias audit “no more than one year prior to the use of the tool.” Given the law’s upcoming January 2023 effective date and the lack of finalized rules, it is not clear whether NYC employers will have the opportunity to use AEDT tools in January 2023 unless the rules are quickly finalized and employer audits that comply with the proposed rules are quickly conducted.   

There are concerns that certain of these timing issues and requirements will slow down innovative technological advances for NYC workers and employers. NYC employers who want to use AEDT may have to deal with a “chicken and the egg” problem. Put differently, if a tool cannot be used unless an audit is conducted, what is the appropriate population on which to conduct such an audit.  And to the extent employers are permitted to rely on audits that use the same AEDT based on other geographic regions or indeed, as applied to other employers, the proposed rules should provide further clarification. 

Notice Requirements

Employers who use an AEDT must notify all candidates and employees residing within New York City that such tools will be used to assess their candidacy and  must provide the job qualifications and characteristics the tool will be assessing. This notice must be given at least ten business days before the AEDT is used to ensure employees and candidates have an adequate opportunity to “request an alternative selection process or accommodation.”

The proposed rules add to this already stringent notice requirement by defining a “candidate for employment” as “a person who has applied for a specific employment position by submitting the necessary information and/or items in the format required by the employer or employment agency.” If adopted, this overly broad definition could require employers and employment agencies to notify any candidate that submits a job application for an open position regardless of whether or not they are considered or qualify for the job. This could be especially problematic for those employers that utilize high volume requisitions to fill high turnover or entry level positions.

The proposed rules attempt to limit this burden by providing several ways to comply with the notification requirement. Employers and employer agencies may provide notice by either: (1) including the notice on their careers website in a clear and conspicuous manner (for candidates) or in a written policy or procedure (for employees), (2) listing the notice on a job posting, or (3) providing the notice via U.S. mail or e-mail. In a seeming contradiction, the proposed rules require that the notice include instructions for how to request an alternative selection process or accommodation, but also clarify that nothing requires that an employer or employment agency provide an alternative selection process.

Further, employers and employment agencies are required to retain information about the data collected for the AEDT being used and its data retention policy. This information can be disclosed by either: (1) including notice on the careers or jobs section of its website in a clear and conspicuous manner, or (2) providing written notice in person, via U.S. mail or e-mail within 30 days of receipt of a written request for such information. If the notice is not included on the career website, employers and employment agencies must post instructions for how to make a written request for such information.

Public Hearing and Opportunity to Comment

DCWP will hold a public hearing on the proposed rules on Monday, October 24, 2022 at 11:00 a.m. ET which will be accessible by phone and videoconference. For more information on how to attend the public hearing please click here. Persons that wish to comment on the proposed rules at the public hearing will be given up to three minutes to speak and must sign up to speak prior to the hearing. As for those interested in submitting public comments on the proposed rules to DCWP, the deadline is also October 24, 2022. Comments can be submitted through the NYC Rules website or by email to Rulecomments@dcwp.nyc.gov.  

In the meantime, Seyfarth will continue to monitor any developments with New York City’s artificial intelligence in hiring restrictions and provide updates if and when they become available.