United States Department of Health & Human Services
Plan for Promoting Responsible Use of Artificial Intelligence in Automated and Algorithmic Systems by State, Local, Tribal, and Territorial Governments in Public Benefit Administration
Introduction
The United States Department of Health and Human Services (HHS) funds billions of dollars in public benefits1 across numerous programs annually. Many of these federally funded programs are administered by state, Tribal, local, and/or territorial government entities (STLTs), serving millions of Americans.
Recent advances in the availability of powerful artificial intelligence (AI)2 in automated or algorithmic systems (referred to as “automated systems”3) open up significant opportunities to enhance public benefits program administration to better meet the needs of recipients and to improve the efficiency and effectiveness of those programs. These capabilities do not come without risk, however, and it is incumbent on government agencies to ensure that these technologies are deployed responsibly to get the best of what they offer without imposing significant risks on those administering or being served by these programs.
To promote the equitable administration of public benefits, Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (EO 14110) Section 7.2(b) requires the Secretary of Health and Human Services, in consultation with other agencies, to publish a plan addressing the use of automated or algorithmic systems in the implementation by States and localities of public benefits and services administered by the Secretary. This plan fulfills this EO 14110 requirement, and is also in alignment with the principles outlined in the National Institute of Standards and Technology Artificial Intelligence Risk Management Framework, and the Blueprint for an AI Bill of Rights.
As called for in EO 14110, this plan addresses the use of AI-enabled automated or algorithmic systems in the implementation by States and localities of public benefits and services funded by HHS, “such as to promote:
• assessment of access to benefits by qualified recipients;
• notice to recipients about the presence of such systems;
• regular evaluation to detect unjust denials;
• processes to retain appropriate levels of discretion of expert agency staff; • processes to appeal denials to human reviewers;
analysis of whether algorithmic systems in use by benefit programs achieve equitable and just outcomes.”
As a part of this plan, HHS will work to ensure alignment with White House Office of Management and Budget Memorandum M-24-10 Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence published March 28, 2024, and in consultation with relevant agencies, to ensure consistency between guidance to federal agencies and to STLTs in operationalizing AI and automated systems in public benefits programs and services.
Objectives
HHS’s vision is that STLTs will be better positioned to procure, develop, and maintain responsible AI- enabled automated systems within the next 2-3 years.
The objectives of this plan are to:
1. Provide recommendations to STLTs and their technology vendors for how they should choose, procure, design, govern, and manage AI in the administration of public benefits and services
2. Outline HHS’s plans over the next twelve months to offer STLTs support in developing their own policies and practices for using AI in automated and algorithmic systems for public benefits programs and services
Scope
This plan provides recommendations for balancing opportunities and risks in using automated and algorithmic systems in the administration of HHS-funded public benefits programs and services. This plan directly applies only to the use of AI for automated and algorithmic systems while also providing best practices for consideration for all automated and algorithmic systems. The recommendations in this plan provide a foundation for STLT programs to create their own guidance tailored to program needs and operations and in line with their authorities. Although these recommendations are not intended for public benefits and services that are administered in a mixed format between government entities and community-based organizations, STLTs may find them instructive for such programs and HHS encourages mixed format benefits and services programs to align with the recommendations described herein as they find them applicable. HHS intends to initiate Tribal consultation in the coming months.
While this framework does not indicate any change in HHS’s approach to overseeing STLT agencies’ use of automated systems that do not use AI, STLT agencies are strongly encouraged to apply these principles and best practices where applicable to non-AI enabled automated systems. STLT agencies are particularly encouraged to identify non-AI enabled automated systems with rights-impacting or safety-impacting uses as described below, evaluate those systems for bias or risks to rights or safety, and mitigate potential risks or harms. All STLT systems used to administer HHS programs should protect rights and safety, advance equity, uphold accountability, and engender public trust, no matter what technology is used. HHS welcomes STLT agencies as well as beneficiaries and their advocates, to proactively engage with us to assess how to best support implementation of these principles and best practices, including with respect to automated systems that do not use AI.
The recommendations laid out in this plan are not mandatory and are general in nature. We support STLTs incorporating these recommendations as appropriate. HHS encourages responsible innovation by the STLT agencies that administer public benefits programs and exploration and use of technologies that improve administration of the programs. While this document references States, Tribes, localities, and territories as a group, HHS does recognize that Tribes are sovereign nations and that as recommendations and policies are developed at a program-specific level, HHS will differentiate between those entities as appropriate.
This plan does not supersede existing relevant federal policies or requirements including those that apply to automated and algorithmic systems such as policies that relate to enterprise risk management, privacy, cybersecurity, and civil rights.
HHS recognizes that some STLTs would have to undertake substantial changes in order to meet the vision laid out in this plan. Over the next twelve months, HHS will engage with STLTs directly to understand how it may support STLTs that want to follow HHS recommendations but face resource or other constraints. Based on what the Department learns, HHS plans to develop guides, playbooks, and other supports to help STLTs implement this plan.AI used in benefits administration
HHS, in alignment with OMB Memorandum M-24-10, is committed to strengthening governance, advancing responsible innovation, and managing risks in the use of AI-enabled automated or algorithmic systems. HHS recommends that STLTs use AI-enabled automated and algorithmic systems to improve program effectiveness and efficiency, educate STLT staff, enhance benefits access, fairness, and customer experience for the recipients they serve. As noted in Executive Order 14110:
“Responsible AI use has the potential to help solve urgent challenges while making our world more prosperous, productive, innovative, and secure. At the same time, irresponsible use could exacerbate societal harms such as fraud, discrimination, bias, and disinformation; displace and disempower workers; stifle competition; and pose risks to national security.”5
AI can be applied in public benefits in both user-facing applications or processes to improve the experience of applicants and benefits recipients; and behind-the-scenes to streamline administrative and operational functions. While AI has the potential to solve some of the key challenges that STLTs face in administering benefits and services, without proper governance and guardrails, such systems can also create significant harms.
Key areas of opportunity
Areas that HHS intends to ensure that STLT partners can capitalize on, and staff and recipients of Federal financial assistance can benefit from, include:
Access: ability to make programs more accessible to underserved communities6 who currently face barriers to enrolling in and using public benefits;
• Efficiency: faster transaction processing and quicker responses to the public including for transactions such as applications, eligibility determinations, benefits distribution, and appeals; a reduction in administrative burdens faced by STLT and service provider staff;
• Effectiveness: more accurate and timely compliance assessments and eligibility determination; advanced analytics to enhance program and benefit design; targeting staff-time to value-added, recipient-focused activities; enhanced program integrity and risk monitoring;
• Customer experience: more comprehensive understanding of customer feedback; needs-based personalization of recommendations and customer support; real-time multilingual support; more convenient user authentication and access options (biometrics for program resource access, etc.); timely decision-making on applications and eligibility; more rapid and accurate answers to questions; proactive information on potential program or eligibility changes such as benefits cliffs, etc.
AI and automation technologies are rapidly maturing and diffusing in information technology systems. This will increase the breadth and depth of use cases to which the capabilities of AI can be applied and their potential impacts on public benefits delivery.Key areas of risk
A key goal of this plan is to catalyze proactive adoption of AI-enabled automated or algorithmic systems- based technologies in use cases where they can improve public benefits delivery for recipients. Balanced against these opportunities, HHS recognizes the importance of deliberate governance and oversight to mitigate the potential and known risks posed by use of automated or algorithmic systems. As stated in EO 14110, “harnessing AI for good and realizing its myriad benefits requires mitigating its substantial risks.”7
OMB Memorandum M-24-10 notes that use of AI comes with a wide range of risks, some of which will already be addressed by existing requirements and recommendations. However, AI-based technologies present specific and unique risks to the rights and safety of the public which must be specifically addressed. These include risks related to efficacy, safety, equity, fairness, privacy, transparency, accountability, reliability, appropriateness, or lawfulness of a decision or action resulting from the use of an automated or algorithmic system to inform, influence, decide, or execute that decision or action.
Uses of AI Presumed to Impact Rights or Safety
This framework centers protection of civil rights in the use of AI stemming from its assignment in Section 7.2 of Executive Order 14110, “Protecting Civil Rights Related to Government Benefits and Programs.” Section 7.2(a) directs Federal agencies to “use their respective civil rights and civil liberties offices and authorities – as appropriate and consistent with applicable law – to prevent and address unlawful discrimination and other harms that result from uses of AI in Federal Government programs and benefits administration.”
AI holds enormous potential to streamline access to public benefits, reduce administrative burden, and improve the customer experience. That potential must be realized on a foundation of preserving the rights and safety for all populations affected by a program. The public should not need to trade or risk their rights or safety to benefit from AI, and AI must not be used in ways that exacerbate bias or inequality. AI uses must comply with nondiscrimination laws, rules, and regulations, as applicable.
HHS will work with STLT agencies to manage risks proportionate to the risk presented by a use of AI. This section describes three tiers of risk, from uses that present the highest risk to uses that present the lowest risk. These tiers are not meant to be comprehensive but provide examples of risk categorization.
Uses of AI that are rights-impacting or safety-impacting present the most risk and may require HHS notification, review, and governance. Uses of AI considered rights-impacting or safety-impacting include both AI systems that directly control outcomes and AI systems that influence outcomes or human decision making. For example, an AI tool that directly makes and enacts program eligibility decisions (if allowed under applicable law) would be considered a rights-impacting use of AI, as would an AI tool that recommends eligibility decisions for a case worker to review and act on.
The following table lists specific uses of AI that are presumed to be rights-impacting and/or safety- impacting in OMB Memorandum M-24-10 and provides examples of potential uses of AI in public 8 benefits administration that would, by extension, be considered rights-impacting or safety-impacting. Please note that not all examples are applicable to all HHS programs, as individual programs may have laws prohibiting such uses.
HHS programs may designate other uses of AI as presumed to be rights-impacting or safety-impacting. These uses may or may not mirror uses of AI enumerated as presumed to be rights-impacting or safety- impacting in OMB Memorandum M-24-10, as its lists are not to be considered exhaustive.
In addition to the specific uses described above, the application of AI-enabled automated or algorithmic technologies to the following functions are also generally presumed to be rights- or safety-impacting:
• Program design and management: triaging cases before assigning to case workers, balancing caseload across workers; determining the terms and conditions of employment, including pre- employment screening, pay or promotion, performance management, hiring or termination, time- on-task tracking, virtual or augmented reality workplace training programs, or electronic workplace surveillance and management systems.
• Application processing: processing application inputs for accuracy and completeness; answering certain questions regarding program requirements and applicability.
Benefits administration: case management including issuing benefits, renewing benefits, issuing notices, managing appeals, providing customer service, health or nutrition screening, decisions regarding medical devices, medical diagnostic tools, clinical diagnosis and determination of treatment, medical or insurance health-risk assessments, drug-addiction risk assessments and associated access systems, suicide or other violence risk assessment, mental-health status detection or prevention, systems that flag patients for interventions, public insurance care- allocation systems, or health-insurance cost and underwriting processes.
• Benefits change or disenrollment: processing changes on a case that affect eligibility or benefit levels, terminating benefits regarding access to, eligibility for, or revocation of government benefits or services.
This is not intended to be an exhaustive list of rights- or safety-impacting functions or use cases. Additional functions or use cases may require consideration according to specific program and local context.
As described in the policy recommendations below, STLT agencies may have specific responsibilities to identify and notify and/or obtain approval for AI uses that are presumptively rights- or safety-impacting if those AI uses fall under an existing requirement to notify and/or obtain HHS approval.Uses of AI that may impact rights or safety
Some uses of AI are not presumptively rights-impacting or safety-impacting but could impact rights or safety depending on how they are applied. These uses of AI do not patently influence outcomes or decision-making but do inform outcomes or decision-making in ways that could impact rights or safety.
Examples include, but are not limited to:
• Staff support tools – Internal AI-powered chatbots or policy summarization tools used by call
center staff, eligibility workers, or other program administration staff to answer questions in real time. These tools may be rights-impacting if they lead to increased rates of incorrect information being provided or adverse outcomes.
• Customer service tools – Public-facing AI-powered chatbots or phone support may become rights-impacting depending on how they are used to support service delivery. HHS programs will work with STLT agencies to consider the impact of risks from the use of AI, such as increasing errors or adverse outcomes, leading eligible populations to believe they are ineligible for public benefits, providing incorrect program information, or creating barriers to accessing public benefits.Uses of AI that are unlikely to impact rights or safety
Uses of AI-enabled technologies that are not directly or presumptively rights-impacting or safety- impacting and do not patently influence outcomes present lower risks. Examples of these uses include:
• Interactive Voice Recognition (IVR) technology for call centers that uses voice recognition to assist callers in navigating menus and routing calls;
• Optical Character Recognition (OCR) to transcribe information from uploaded documents or paper forms;
• Chatbots using natural language processing to better understand user questions, with human- coded, logic-based preset outputs, not generative AI responses;
• Sentiment analysis/natural language processing to categorize major themes and trends in unstructured text for customer experience and customer satisfaction surveys, helpdesk tickets, or social media posts referencing a benefit program;
• Creation of synthetic data for testing information technology systems; and
• AI-enabled search tools that answer questions about program requirements or policies by
directing caseworkers to the relevant section of an official policy manual or other primary source;
• Internal facing chatbots that answer questions about program requirements or policies by
directing caseworkers to the relevant section of an official policy manual or other document, where caseworkers are required to cross-check the manual to confirm the citation before proceeding;
• Converting scans of physical documents into machine-readable formats for further analysis and/or improved accessibility;
• Summarizing case management files, relevant research documents, or other large sources of narrative information to provide a starting point for someone to understand high-level situation before diving into details;
• Creating initial drafts of documents or suggesting ways to rewrite a document in plainer language or with better visual layout (similar to what HHS is doing with the simpler Notice of Funding Opportunities project));
• Transcribing conversations as a back-up set of notes;
• Creation of first drafts of translated materials that a human with language expertise will review
and approve.
This is not intended to be an exhaustive list. Whether the use of these or any other such technologies are actually rights- or safety-impacting will depend on the specific circumstances and context implemented by the STLT agency.
As described in the policy recommendations below, STLT agencies may have specific responsibilities to identify and notify and/or obtain approval for AI uses that may be rights- or safety-impacting if those AI uses are determined to be rights- or safety-impacting and fall under an existing requirement to notify and/or obtain HHS approval.Other risks, applicable across risk tiers
The following is a non-exhaustive group of general risks involved with using AI in public benefits activities in addition to rights- or safety-impacting uses. STLTs should consider these risks and potential mitigation options for each new AI application:
• Inappropriate or invalid use: automated or algorithmic-based technologies applied outside of original model design parameters or use of automated or algorithmic-based technologies with low performance such as to reduce or limit benefits access inappropriately (e.g., high rates of procedural terminations and/or erroneous auto-terminations);
• Unintended consequences of otherwise appropriate use: an ‘appropriate’ use of AI to optimize efficiencies in one area (such as financial metrics) might have a negative impact on service delivery (such as providing positive health care outcomes) elsewhere. In other words, competing priorities or metrics where an AI system is meant to choose one over the other;
Inattention to equity: discriminatory outputs based on data imbued with underlying and/or systemic inequities and biases; embedded discriminatory profiling of individuals; automation and rapid proliferation of discriminatory outputs;
Automation bias: processes or practices that deter agency staff from thinking critically about the outputs of an automated or algorithmic system; for example, policies that require a pace that prohibits meaningful review, or practices that penalize employees for questioning a system’s outputs;
Customer alienation: recipient inability to access accurate information; reduction in opportunities for human-led customer support; lack of transparency regarding uses of automated or algorithm- enabled technologies and causes of benefits-related decisions;
Due process: recipient ability to understand and appeal decisions, including access to information about the inputs and the working of the automated or algorithmic system, made by automated or algorithm-enabled technologies;
Unmanaged scope: automation and rapid diffusion of inaccurate, unjust, or unsafe algorithms; lack of human intervention/review in decision-making/insufficient processes to address failures of automation; undue reliance and inability to overrule model outputs;
Privacy risks: implicit coercion to share biometrics or other personally identifiable information, privacy risks associated with use of other public data (e.g., social media postings) in automated and algorithmic systems used in agency decisions and activities;
Agency control of automated or algorithmic systems: over-reliance on private sector vendors and proprietary technology, which prevents knowledge of details on system outcomes and decision- making drivers;
Quality assurance of automated or algorithmic systems: lack of standards, technologies, expertise, processes to assess quality and risk of automated or algorithmic technologies or automated systems and to monitor performance on an ongoing basis; lack of public scrutiny;
Insufficient governance and oversight capabilities and processes: to make informed procurement decisions, manage vendors, and audit systems.Policy recommendations
HHS highly encourages STLTs to adopt already existing AI best practices as described herein. This plan provides recommendations based on previously issued federal guidance pertinent to the use of AI in public benefits. It also provides a list of potential future guidance and/or recommendations that may be issued at the HHS- or program-level as per below, which should further help clarify practices for STLTs.
Of note, using AI-enabled automated and algorithmic systems does not change programs’ responsibility to comply with existing applicable federal, state, and local laws, and regulations, including those addressing privacy, confidentiality, intellectual property, cybersecurity, human and civil rights, and civil liberties.Recommendations from existing guidance
The federal government has numerous guidance and policy documents pertaining to automated and algorithmic systems. These include the Blueprint for an AI Bill of Rights, the National Institute of Standards and Technology (NIST) AI Risk Management Framework, Executive Order 14110, and OMB Memorandum M-24-10. By issuing recommendations and by creating additional resources, HHS seeks to help STLTs find ways to align with them as well. STLTs to adopt the following priorities in their use of AI in automated or algorithmic systems, consistent with OMB Memorandum M-24-10:
Advance Responsible Innovation for Automated and Algorithmic Systems
1. Manage Risks from the Use of Automated and Algorithmic Systems 3. Strengthen Governance for Automated and Algorithmic Systems
The above areas, policies, and principles are reflected below in HHS’s recommendations to STLTs.
Advance Responsible AI Innovation
HHS encourages STLT administrators of public benefits that HHS oversees to responsibly use AI-enabled automated and algorithm-based technologies to advance the quality of public benefits programs for recipients. For those STLTs that either are already using AI-enabled automated or algorithmic systems or plan to in the near future, HHS will provide recommendations and technical assistance over the next 12 months to establish and execute a plan to increase their capacity to appropriately utilize, and evaluate, to root out and mitigate risks to rights, safety and equity of AI in automated and algorithmic systems to achieve their missions and take steps to advance the responsible use of AI in automated and algorithmic systems across the public benefits administration enterprise, including across the various functions described above.
Key enablers of effective adoption of AI in automated and algorithmic systems include:
• IT infrastructure;
• High quality data for use in training, testing, and operating automated or algorithmic systems; • Appropriate safeguards (e.g., cybersecurity, privacy, confidentiality, and civil rights)
• Technical talent and staff training;
• Sharing and collaboration in areas such as: models and code, data sets, policies, best practices,
and lessons learned, technical resources, procurement, and implementation resources, etc.;
• Quality assurance policies and processes to build and measure trust in adopted automated and
algorithmic systems;Manage Risks from the Use of AI
STLT agencies are responsible for notifying HHS and/or obtaining HHS approval for the AI uses outlined in OMB Memorandum M-24-10 if those AI uses fall under an existing requirement to notify and/or obtain HHS approval. STLT agencies must determine which program system changes and functions incorporating AI trigger notice to and/or approval by HHS per existing HHS program requirements, particularly uses that are presumptively or may be rights-impacting or safety-impacting AI.
HHS strongly encourages STLTs using AI-enabled automated and algorithmic systems in administering the programs HHS listed in this plan to seek to mitigate risks stemming from their use of AI. As per OMB Memorandum M-24-10, such risk considerations may include impacts to safety, security, civil rights, civil liberties, privacy, democratic values, human rights, equal opportunities, worker well-being, access to critical resources and services, agency trust and credibility, and market competition. NIST defines the essential building blocks of AI responsibility and trustworthiness to include accuracy, explainability and interpretability, privacy, reliability, robustness, safety, security, and the mitigation of harmful bias.
HHS strongly encourages STLTs to prioritize risk management for AI-enabled automated and algorithmic systems that have the potential to impact the public’s rights or safety. HHS strongly encourages STLTs to identify their uses of AI-enabled automated or algorithmic systems which may impact rights or safety through the process described in Section ‘Identifying Automated and Algorithmic Systems Which Impacts Rights or Safety’. For rights- and safety-impacting AI, HHS strongly encourages STLT to follow the policy principles described below, as well as the minimum risk management practices in M-24-10 Section 5(c).Ensuring safety and security
The NIST AI Risk Management Framework outlines key elements of safe operations of automated and algorithmic systems:
• responsible design, development, and deployment practices;
• clear information to deployers on responsible use of the system;
• responsible decision-making by deployers and end users;
• explanations and documentation of risks based on empirical evidence of incidents.
HHS strongly encourages STLTs to use a safety-by-design approach across the technology lifecycle for automated and algorithmic systems to maintain operational safety and security, whereby safety considerations are embedded throughout the product lifecycle of AI-enabled technologies. Traditional procurement approaches that evaluate systems during the purchasing process but apply little or no diligence to ongoing monitoring will not suffice for automated or algorithm-enabled technologies which are dynamic in nature. Other key elements include the ability to shut down, modify, or have human intervention into systems that deviate from intended or expected functionality. This is especially important in automated systems to prevent rapid proliferation of safety issues, as is establishing mechanisms and processes for identification and reporting of adverse events.
HHS strongly encourages STLTs to implement the following practices to promote safe development, use, and operations of automated or algorithmic systems:
• Conducting an impact assessment to determine the estimated benefit from the automated or algorithmic system as compared to its potential risks;
• Measuring the quality and appropriateness of the data used in the automated or algorithmic system’s training, testing, and prediction;
• Piloting systems in a real-world context before full deployment to assess whether automated or automatic systems are mature enough to be accurate and helpful;
• Leveraging pilots and limited releases with strong monitoring, evaluation, and safeguards in place, to carry out the final stages of testing before a wider release;
• Ensuring human facilitation and intervention where needed in the event that an automated system fails or causes harm without disruption of service delivery. No matter how rigorously an automated system is tested, there will always be situations for which the system fails;
Conducting ongoing monitoring to track performance of the automated or algorithmic system on a continuous basis, to detect degradations on functionality and to detect changes to impact on rights and safety.Supporting workers
STLTs must abide by merit staffing requirements and appropriately fulfill collective bargaining obligations where applicable. Deployment of automated and algorithmic systems should empower, not debilitate, or demoralize, staff charged with administering public benefits programs.
HHS strongly encourages STLTs to consult workers and provide adequate training for all staff around developing, using, enhancing, and maintaining automated and algorithmic systems. Automated or algorithm-enabled technologies can eliminate rote or tedious work and use staff resources for higher value tasks leveraging the top of their skill sets. The use of automated or algorithmic systems informed by human domain expertise to support agency experts can enhance the ability of public benefits agencies to meet the growing complexity and volume of information within staff and resource constraints.9 However, the utility of automated or algorithmic systems is only as good as the ability of staff to appropriately use and maintain such systems. Training for staff should address the risk of automation bias, which results when system design, organization processes, or culture deter staff from making judgments including overruling or questioning automated or algorithm-based outputs. Program oversight is also paramount; the program owner should have product knowledge and know enough about these systems to understand how AI is being used.
HHS strongly encourages STLTs to ensure system design and policies sustain staff judgment which can be particularly important in complex or nuanced situations. STLTs should consider which decisions or actions could result in a significant impact on rights or safety, and provide additional human oversight, intervention, and accountability in those cases. Additionally, ongoing monitoring of the automated or algorithmic system must include periodic human reviews to determine whether the deployment context, risks, benefits, or agency needs have evolved.
HHS strongly encourages STLTs to seek to exercise control over automated and algorithm-based technologies when needed rather than to hand control over to third party vendors. STLTs should take steps to ensure that their contracts retain sufficient rights to data and any improvements to that data to facilitate STLTs’ continued design, development, testing, and operation of automated and algorithmic systems. STLTs should ensure that their contracts afford sufficient information about and access to systems to assess efficacy and rights safety impacts. Additionally, STLTs should consider contracting provisions that protect data inputs and STLT information used by vendors in the development and operation of automated and algorithmic systems, and that protect such data from unauthorized disclosure and use, especially where that data includes personally identifiable information.Advancing Diversity, Equity, Inclusion, Accessibility, and Civil Rights
HHS has issued a Joint Statement on Enforcement of Civil Rights, Fair Competition, Consumer Protection, and Equal Opportunity Laws in Automated Systems with the Consumer Financial Protection Bureau, Department of Justice, Department of Housing and Urban Development, Department of Education, Department of Labor, Department of Homeland Security, Equal Employment Opportunity Commission, and the Federal Trade Commission. HHS will continue to issue joint guidance with other agencies when appropriate.
HHS strongly encourages STLTs to identify and assess impacts of automated or algorithmic systems on equity and fairness and mitigate algorithmic discrimination when it is present. STLTs should take steps to mitigate discrimination throughout the design, development, implementation, iteration, and ongoing monitoring after deployment. Such an approach could include proactive assessment of equity considerations as part of system design, ensuring model development based on representative and robust data, and ongoing disparity assessment and mitigation.11 Such assessment should include whether their rights-impacting AI materially relies on information about a class protected by federal nondiscrimination laws, or variables that are proxies for it, in a way that could result in algorithmic discrimination or bias against that protected class.
HHS, in alignment with Section 7.2 of EO 14110, strongly encourages STLTs to “use their respective civil rights and civil liberties offices and authorities—as appropriate and consistent with applicable law—to prevent and address unlawful discrimination and other harms that result from uses of automated and algorithmic systems in federal government programs and benefits administration.”12
HHS encourages STLTs to be inclusive of diversity, equity, inclusion, and accessibility (DEIA) throughout the design, development, implementation, iteration, and ongoing monitoring after deployment. Negative impacts will likely occur without developing responsible AI practices with strong DEIA practices.
Bias can occur when employing AI even in the absence of prejudice, partiality, or discriminatory intent.13
HHS strongly encourages STLTs employing AI to proactively assess and mitigate factors that contribute to bias, algorithmic discrimination, or that create inequitable outcomes for protected classes or underserved communities. These factors include, but are not limited to, tools, datasets, system flows, and business processes. These assessments should not be a one-time activity; they should occur before any procurements and also be integrated into processes for design, development, implementation, testing, training, and ongoing monitoring.14 Assessments should also consider the risks of AI beyond its intended use.
HHS strongly encourages STLTs to evaluate rights-impacting and safety-impacting uses of AI for biased or inequitable outcomes under conditions that mirror real-world use before they are used in program administration. Rights-impacting and safety-impacting uses of AI should be reevaluated for bias on at least an annual basis and after any significant modification to the AI or the conditions or context in which the AI is used. Periodic reevaluation is important to detect any emergent biases as participant demographics and program rules change over time. If an evaluation determines that inaccurate, biased, or disparate outcomes are produced, the AI use should be discontinued. Evaluations should assess disparate impacts on classes protected by federal nondiscrimination laws and should also assess impacts on underserved communities that are particularly vulnerable to additional burden, barriers, or disruptions in benefit access.
HHS recommends STLTs to consider and manage the following three major categories of AI bias identified by NIST: systemic, statistical, and computational, and human.15 Each of these biases can occur in the absence of prejudice, partiality, or discriminatory intent.
• Systemic bias can be present in datasets, the organizational norms, practices, and processes across the system lifecycle, and the broader society that uses automated or algorithmic systems. For example, if there is racial bias in historical child welfare placements and separation decisions, using longitudinal data to train a decision support tool has a high risk of perpetuating bias.
• Statistical and computational biases, also known as algorithmic biases, occur when the datasets used to train and update AI models lack adequate representation of different groups or demographics in the data, leading to skewed outcomes. For example, in order for chatbots to answer user questions accurately, they must be trained on input speech from a diverse user base reflecting the full geographic, ethnic, and age range of intended users. AI-enabled program design recommendations must be based on a diverse group of program users, not just those for whom large volumes of existing training material were readily available.
Human biases relate to how an individual or group perceives system information to inform a decision or fill in missing information, or how humans think about purposes and functions of a system. Human-cognitive biases are omnipresent in institutional, group, and individual decision- making processes across the AI lifecycle and system use, including the design, implementation, operation, and maintenance of automated and algorithmic systems. For example, a human may overly trust an AI-generated draft document or decision recommendation as impartial or correct, leading them to insufficiently review and reconsider the AI content.Preserving options to opt out of AI
Even with the above protections in place, some individuals and groups will not want to use AI-enabled systems. HHS encourages STLT agencies to provide options to opt-out of the use of public-facing AI for a human alternative, wherever practicable. Where a human alternative is not feasible, an automated system that does not use AI could be provided. For example, if an applicant contacts a call center during business hours, they could be able to opt out of interacting with an AI-powered virtual agent to talk with a human representative. If the applicant calls after hours, when call center staff are not on duty, they could be able to opt out of the virtual agent to access a multilingual phone tree of common functions, such as checking benefit balances or hearing application status.
STLTs may seek to ensure opt-out options are easy to find and access and be provided in all languages required to be supported. Opt-out options are most useful when broadly accessible to people with disabilities and provide a level of service (e.g., wait time, administrative burden) that is not disproportionate to AI alternatives. When designing opt-out options and services, STLT agencies may consider that people with disabilities; individuals with limited English proficiency; people experiencing disasters, financial shock, or trauma; and people from underserved communities may be more likely to opt out of AI for human alternatives.
Opt-out options may be less critical for back-end processes, employee-facing functions, or for enabling functions, such as scanning forms using AI-enabled OCR. Opt-out alternatives may also be less critical when AI is solely used for the prevention, detection, or investigation of fraud.Human oversight of automated and algorithmic processes
Uses of AI-enabled technologies that include human oversight or review can present lower risks if appropriate practices for human-in-the-loop processes are followed. In these cases, although the risk is lower when there is sufficient human review of the output, there may be substantial risk if the review does not occur or is done by someone without proper expertise to evaluate the AI output. Therefore, HHS encourages STLT agencies to establish clear, effective human oversight protocols, document those processes, train staff, and conduct periodic oversight to ensure adherence. It is also important to ensure that such human oversight does not overcompensate and completely negate the benefits sought from the technology in the first place. HHS programs may seek to review such protocols and oversight mechanisms and require reporting on the use of AI.
Some risks from the use of AI can be mitigated by requiring a “human in the loop”, with human oversight of AI functions or requirements that a human approve AI recommendations before they are enacted. It is important to carefully design these “human in the loop” processes to ensure that human oversight provides the intended validation of AI outputs and does not result in human actions becoming a “rubber stamp” without the expected scrutiny. Refactoring AI business processes to add human oversight is an insufficient alternative to addressing a root cause of bias or errors in an AI system.
STLT agencies may seek to ensure that staff providing human oversight understand how the AI system functions, what an accurate decision looks like, and how to evaluate a system’s decisions. Staff may benefit from understanding the types of errors their role is meant to detect and have a workload appropriate for providing the expected level of oversight. STLT agencies may seek to ensure staff have the authority to override or alter the decision under review and should be able to escalate patterns of errors they have observed for further analysis and remediation.
Staff who provide human oversight for AI-enabled functions may benefit from training on relevant AI topics and should receive sufficient training in business processes to assess outputs of AI functions or models for accuracy. For example, staff who use an AI-enabled tool to advise them in benefit calculations could be trained in how to calculate benefit amounts without the assistance of the AI tool, so they can evaluate the AI’s recommendation.
STLT agencies may seek to regularly evaluate business processes that interact with AI systems by observing their execution under real-world conditions to determine the effect of automation bias. If AI outcomes that are intended to have human review or that are intended to influence, but not direct, decision-making are being accepted without appropriate scrutiny, STLT agencies may stop using the AI until business processes can be refactored and staff can be retrained.Protecting the Public
HHS strongly encourages STLTs to ensure automated and algorithm-based technologies advance the interests of the recipients of public benefits programs and services and maximize their ability to access public benefits programs. Critical aspects of ensuring that recipients are best served by AI- and algorithm- enabled technologies include:
• Transparency to recipients about the use of such systems, including plain language and accurate multilingual communication about the use of such systems;
• Consulting and incorporating feedback from affected communities and the public;
• Providing the ability to opt out of use of automated and algorithmic systems in favor of an easily
accessible human alternative, where applicable, including consideration of the needs of recipients
with disabilities or in need of reasonable accommodations;
• Providing mechanisms for the public to provide feedback on the customer experience and to
report incorrect information or issues;
• Notifying negatively affected individuals;
• Providing the ability to appeal decisions that were made in reliance on automated or algorithm-
based technology outputs.
Safeguarding Privacy, Civil Rights and Civil Liberties
HHS strongly encourages STLTs to work to protect recipients from violations of or undermining trust in privacy protections through design choices that ensure such protections are included by default, including ensuring that only data strictly necessary for the specific context is collected, and with recipient consent.
Considering the expanding scale of private data collection utilized by AI, it is imperative that AI-based technologies supporting public benefits programs and services be based on ethical principles that prioritize the protection of recipient privacy and civil liberties.
Data collection should be limited in scope, minimized as much as possible, and determined to be strictly necessary for meeting specific, narrowly identified purposes. Data collected based on these specific, narrowly defined purposes should not be used in a different context without assessing for new privacy risks and implementing appropriate mitigation measures, which may include express consent.
Entities creating, using, or governing automated systems should follow privacy and security best practices designed to ensure data and metadata are not utilized beyond the specific consented use case. Best practices could include using privacy-enhancing cryptography or other types of privacy-enhancing technologies or fine-grained permissions and access control mechanisms, along with conventional system security protocols.
AI- and algorithm-based systems should be designed and built with privacy protected by default. Privacy risks should be assessed throughout the development lifecycle, including privacy risks from reidentification, and appropriate technical and policy mitigation measures should be implemented. This includes potential harms to those who are not users of the automated system, but who may be harmed by inferred data, purposeful privacy violations, or community surveillance or other community harms.Strengthen Governance for Automated and Algorithmic Systems
Promoting innovation and managing associated risks for automated and algorithmic systems requires effective and deliberate governance. In general, the goal of governance is to ensure management of risks stemming from reliance on automated or algorithmic system outputs in public benefits programs to inform, influence, decide, or execute program decisions or actions. Ineffective governance could undermine the efficacy, safety, equitableness, fairness, transparency, accountability, appropriateness, or lawfulness of such decisions or actions.
HHS encourages STLTs to establish governance for a program-wide risk management framework covering use of AI technologies in public benefits programs supported by HHS, with defined roles and responsibilities for clear understanding and communication of AI uses, assumptions, and limitations, and publish on a publicly available website their governance and risk-management framework. STLTs may benefit from the designation of a central Chief AI Officer role to help lead this effort.
It is important to recognize that broad applicability of automated or algorithm-based technologies requires a governance approach integrated with enterprise risk management strategies and processes alongside other system-wide areas such as cybersecurity, privacy, and regulatory compliance. This will ensure that AI gets the required level of governance attention, is integrated with other system-wide focus areas, and will facilitate operational alignment and organizational efficiencies.
The NIST AI Risk Management Framework identifies 6 key pillars of AI governance:
1. Policies, processes, procedures, and practices across the organization related to the mapping, measuring, and managing of AI risks are in place, transparent, and implemented effectively.
2. Accountability structures are in place so that the appropriate teams and individuals are empowered, responsible, and trained for mapping, measuring, and managing AI risks.
3. Workforce diversity, equity, inclusion, and accessibility processes are prioritized in the mapping, measuring, and managing of AI risks throughout the lifecycle.
4. Organizational teams are committed to a culture that considers and communicates AI risk.
5. Processes are in place for robust engagement with relevant AI actors.
6. Policies and procedures are in place to address AI risks and benefits arising from third party
software and data and other supply chain issues.
Applying these governance principles to reviewing, approving, and managing automatic and algorithmic- enabled technologies will help ensure a comprehensive and responsive approach to responsible AI innovation.
HHS strongly encourages STLTs to establish formalized processes to explicitly evaluate and put in place risk mitigation steps for automated and algorithm-based technologies used in public benefits decisions and activities for programs supported by HHS.
Development, procurement, and use of automated and algorithm-based technologies should undergo analysis of potential risks and adverse impacts; include practices to mitigate identified risks; and incorporate policies and controls for governance, such as how data is acquired, managed, and used. Automated and algorithm-based technologies should be evaluated on a variety of key elements, including validity, reliability, robustness, fairness, intelligibility, safety, security, and privacy.
HHS strongly encourages STLTs to collect and maintain an inventory of automated and algorithm-based technologies developed or procured and used in public benefits programs decisions and activities including descriptive information regarding the developers, purposes, risks, validation, and maintenance of such technologies. This inventory should be made public to the extent possible.
Sound governance requires visibility and awareness of automated and algorithmic systems being used across the public benefits administration enterprise, and transparency regarding the development and maturity of such technologies. HHS strongly encourages the inventory of automated and algorithmic systems capture key elements for each application including:18
1. Developer details and designed output
2. Purpose of the technology
3. Out-of-scope uses
4. Development details and input features
5. Process used to ensure fairness in design
6. Process to ensure integrity and suitability of data for the machine learning task
7. External validation process
8. Quantitative measures of performance in test environments
9. Quantitative and qualitative measures of performance and efficacy in real-world environments
HHS strongly encourages STLTs to ensure that this information be included by vendors included in procurement requirements of AI-based technologies or by developers for agency-built applications. It is recognized that not all this information may be available for or applicable to each automated or algorithm- based technology, and lack of such information does not have to be disqualifying. However, explicit consideration of these elements and thresholds for each is critical to robust governance of automated and algorithmic systems and system auditing.
HHS strongly encourages STLT agencies to use the acquisition process, if applicable, to gather information from vendors that will assist with identifying and mitigating risks from the use of AI, including but not limited to:
• Intended use – Vendors should identify uses that are suitable for their AI and purposes, contexts, and uses that are not suitable for their AI’s use;
• Training, test, and evaluation data – Vendors should provide details about the model’s training data, including the source of the data, the time periods the data describes, and the extent to which the data reflects real-world contexts. For training data that reflects populations, vendors should provide summary statistics of the population whose data it was trained on that are relevant to a public benefits context (e.g., race/ethnicity, age, gender, income brackets);
• Testing results – Vendors should provide information about how they have tested their system, what contexts they tested it in, findings from their testing, and changes they made to address any issues discovered during the testing process. Vendors should provide testing results broken out by the population demographics that are relevant to a public benefits context (e.g., race and ethnicity, age, gender, and income brackets);
• Audit results – Vendors should disclose whether the system and data have been assessed for bias and/or other risks, the results of any assessments, and any actions taken to mitigate findings.
HHS strongly encourages STLT agencies to only acquire and use AI if a vendor provides all necessary information to evaluate suitability and risk. Purchasing from federal or STLT acquisition schedules may simplify the process of gathering information if disclosure is required by the schedule’s terms and conditions.
HHS strongly encourages STLT agencies to consider acquisitions that enable the agency to try AI technologies before committing to expensive or lengthy contracts. Agencies could consider testing the technology in a controlled, isolated environment known as a sandbox or conducting a limited pilot to evaluate suitability, risks, and gains from using AI.
As AI becomes more commonplace, STLT agencies may anticipate that vendors may respond to proposals with solutions containing AI, even if AI features were not requested or anticipated for that acquisition.Topics of potential future guidance
The next section delineates existing federal policy that HHS encourages STLTs to align with as they use AI in administration of public benefits programs. In addition, HHS will seek to further clarify general and program specific principles through new guidance and/or recommendations in the coming months. HHS is continuing to assess and consider for additional clarification in the next 12 months:
• Parameters around automation. The role of human intervention in administering rights and safety impacting automated and algorithmic systems;
• Appeal rights. When applicants and participants can appeal benefit determinations to a human reviewer for reconsideration due to the use of automated and algorithmic systems;
• Customer support. When must applicants and participants have access to a human being for customer support;
• Generative AI used to support public facing use cases and backend administrative use cases;
• IT and data readiness needed to use algorithmic systems in particular use cases;
• Success metrics and service delivery benchmarks for system implementations, including
application turnaround, eligibility determinations, appeals decisions, equity, cybersecurity, and privacy;
• Guidance on how this plan will apply to Tribes;
• Best practices on how to streamline application/eligibility process across chatbots and other
customer-assistance tools, and multilingual translation.
HHS will work with STLTs to refine this list to meet STLT needs and priorities.
Providing resources and other supports to STLTs
HHS recognizes that the teams responsible for delivering benefits and services face significant challenges in their day-to-day work, ranging from budgetary constraints and staffing shortages to out-of-date technology systems. Safe and responsible adoption and use of automated and algorithmic systems by STLTs in the administration of public benefits and services will require significant resources. HHS understands that STLTs may face challenges in adopting new practices outlined here considering resource constraints.
Federal agencies, including HHS and the General Services Administration (GSA), have already developed training and practical technical assistance guides that offer IT implementation best practices. HHS seeks to better connect the field with these existing resources.
Specific areas in which HHS will explore how it can provide additional supports and/or take additional actions include, but are not limited to:
• Providing information on available funding mechanisms for STLTs;
• Convening learning collaboratives or other knowledge sharing structures;
• Providing training data sets and/or training environments, such as those that reflect the diversity
of those eligible for benefits programs or are specific to a particular program;
• Developing frameworks to support AI quality assurance for benefits programs;
Connecting STLTs with educational implementation guides.Public Engagement strategy
To develop this plan in 180 days, HHS engaged with key affected communities for preliminary consultation and feedback including state and local benefits and services employees, human services professional associations, civil rights advocates, disability rights advocates, privacy advocates, technologists, and legal and labor representatives. As HHS develops guidance and resources over the next twelve months, it will conduct further engagement with the public, including underserved communities, to help inform the development and implementation of the guidance, tools, and support discussed in this plan.
Through the coming year, HHS will use a variety of participation and engagement activities to:
• Host ongoing listening sessions with advocates and members of affected and underserved communities to understand potential harms and promote compliance with federal civil rights and privacy laws;
• Engage state, local and community partners, including advisory committees to gather feedback and foster discussion about forthcoming guidance, consistent with applicable law and government-wide guidance;
• Conduct Tribal consultation, as well as Urban Confer through the Indian Health Service;
• Broaden public engagement to ensure the consideration of a range of lived experiences and
perspectives in policymaking;
• Host webinars and workshops with benefits and service delivery staff in partnership with select
non-governmental entities to enable information sharing;
• Share learnings and feedback received with STLTs for incorporation into the design,
development, and use of AI, and to inform agency decision-making regarding automated and
algorithmic systems;
• Improve STLT access to non-governmental organization-developed training, educational
materials (including playbooks, toolkits, frameworks, and other “best practice”-type content).