Today, we’re excited to share a behind-the-scenes look at how New Jersey cracked the benefits bottleneck for countless workers who struggle to fill out forms in English by using generative AI (gen AI) to scale their proven approach to language access. Through the development of a set of Large Language Model (LLM) training materials designed to improve language access to unemployment insurance (UI) for both English and Spanish speakers, New Jersey has effectively optimized its statewide generative AI platform to improve language access and to continue paving the way for other states to develop their own tailored solutions.
These training materials were a true cross-sector collaboration, bringing together government, non-profit, and private sector expertise. U.S. Digital Response (USDR) worked closely with the State of New Jersey and included pro bono technical support from Google.org Fellows, showcasing the power of diverse partnerships in tackling complex public challenges.
Starting today, these materials – a no-code approach to expanding language access – is available as an open-source resource to any state or local government agency nationwide. These training materials can be used to improve the results of off-the-shelf LLMs, such as Chat GPT, Gemini, or Claude. These materials include:
- An LLM training corpus (plain language glossary of specialized terms)
- LLM prompts (instructions like "translate this at an 8th grade reading level")
- Rubrics for evaluating the quality of translations produced by the gen AI model
By making these materials open source, this collaboration ensures that other governments can benefit from and contribute to its development. Also, there’s no vendor lock-in, promoting flexibility and innovation, and no wasted procurement efforts.
Before we dive deeper into these training materials, let’s explore how we got here.
A strong foundation: real human-centered design
I noticed a lot of the Spanish speakers are paying a third-party agency when they should be able to make a claim for free by themselves online. — Call center agent, NJ DOL
This last Spring, New Jersey took a significant step forward in breaking down language barriers by releasing a modernized unemployment insurance (UI) intake form for the first time in both plain English and plain Spanish. This groundbreaking effort was the result of a collaboration between the New Jersey Department of Labor & Workforce Development (NJDOL), the New Jersey Office of Innovation, Nava, Truss, and USDR.
The initial translation process was deeply rooted in human expertise and cultural understanding:
- Co-designing content with bilingual front-line staff and UI subject matter experts
- Validating with Spanish-speaking residents from 10 different Spanish-speaking countries
- Employing native Spanish speakers to lead user research and content strategy
This human-centered approach involved:
- Eight months of intensive content co-creation
- Collaboration with 15 Spanish-speaking New Jersey residents with lived experience interfacing with the UI system
- A dedicated team of four Spanish-bilingual user experience researchers and content designers
The impact of NJ’s UI modernization efforts across the UI system was immediate and significant:
- Reduced average form completion time from over three hours to just 28 minutes
- Achieved parity in completion time between English and Spanish speakers
- Decreased average call center wait time from 40 minutes to two minutes
- Lowered the rate of claims requiring manual review by 14%
Out of that project, we published a set of Spanish language content resources, co-authored by Barbara Niveyro, bilingual content designer, and Marcie Chin, language access product delivery manager at USDR, so that other states could benefit from all the work that went into localizing the intake form to Spanish.
- Plain Spanish Glossary of Unemployment Insurance Terms, containing Spanish translations that have been validated through user testing
- Spanish Translation Guide for Unemployment Insurance, to preserve the guidelines and approach we took for translating Spanish in a human-centered and culturally relevant way.
Human-led, gen AI assisted: where we’re going
New Jersey has been at the forefront of adopting AI technologies to improve government services, just this summer launching the NJ AI Assistant, offering a safe, “sandbox” environment for staff at State departments and agencies to use AI to responsibly improve government services and operations. The state emphasizes responsible and ethical AI use, with a focus on enhancing equity and accessibility.
Building on the human-led success, these training materials represent one of New Jersey's first forays into using gen AI for language access. Thanks to the support from Google.org Fellows, the team was able to conduct user research with Spanish speakers, design and engineer prompts for the LLMs, and produce a workshop to train staff on how to use these training materials in order to achieve language parity in the new content.
Here's how we’ve transitioned:
Creating a foundation: We published resources based on our human-led work, including:
- A Plain Spanish Glossary of Unemployment Insurance Terms
- A Spanish Translation Guide for Unemployment Insurance
Leveraging human expertise: These human-created resources are now being used to fine-tune, or instruct/prompt, LLMs, ensuring they capture the nuances and accuracy of expert human translation.
Testing gen AI capabilities: We conducted a series of research studies to compare the accuracy and effectiveness of the fine-tuned LLM-generated translations with human translations, Google Translate, and unprompted LLM’s. Our fine-tuned LLM translations are nearly indistinguishable from the expert human translations.
Keeping humans in the loop: While our gen AI-assisted translations show promise, we emphasize the continued importance of human oversight, testing, and validation.
Our advice: Keep humans in the loop to maximize value
While our studies demonstrated that using USDR’s resources to fine-tune an LLM can produce translations that are almost equal to an expert human translator, you still need to keep humans in the loop.
For language access, this means fine-tuning the model with human-centered data sets, like our plain Spanish glossary, translation guide, and custom prompts. Workflows should ensure that translations generated by the model are always tested for readability by end users and reviewed for accuracy by a competent human translator paired with a subject matter expert.
The road ahead: scaling access with gen AI while preserving human insight
As we move forward, our goal is to use gen AI to amplify, not replace, human expertise. That’s why we’re excited to share this no-code approach that allows states to leverage gen AI to expand language access while maintaining the accuracy, accessibility, and cultural relevance established by our human translators. The open-source nature of these training materials ensure that, as we refine and improve our approach, all users can benefit from these advancements.
This approach promises to:
- Save time in developing plain language translations up to three times faster
- Maintain consistency across various communication channels
- Ensure compliance with relevant federal, state, or local policies
- Potentially expand services to additional languages and benefits programs
Power to the people
We're grateful to the State of New Jersey for their willingness to innovate and their commitment to equitable access.
As we continue this journey, we invite other state and local government agencies to join us in exploring how gen AI can enhance, but not replace, human-centered language access efforts. By leveraging these open-source training materials, you're not just adopting a technology – you're joining a community dedicated to breaking down language barriers and ensuring equal access to critical services and benefits for all residents, regardless of their primary language.
Together, we can harness the power of gen AI responsibly and ethically, setting new standards for accessible and inclusive government services.
Photo by Centre for Ageing Better on Unsplash
