The Social Security Administration is implementing a new AI (artificial intelligence) system to improve the accuracy and efficiency of the automatic recording and transcription of hearings. The Hearing Recording and Transcriptions, HeaRT, system aims to support all hearing formats — in-person, telephone, and video — without having to rely on outdated recording equipment.
- HeaRT System: A new artificial-intelligence (AI) software program being used by the Social Security Administration to improve the automatic recording and transcription process of hearings.
- Automatic Speech Recognition (ASR) Systems: AI systems used to convert spoken word into written text and transcribed text.
- Large Language Models (LLM): Complex artificial-intelligence programs that use an abundance of inputted text and written training data to understand and create written word that reads as if it were written by a person.
- AI Hallucinations: Outputs and answers presented by an AI model—usually an LLM—that are fabricated because the model works off of patterns and not verified facts, resulting in an answer that can resemble a guess.
What Is SSA’s HeaRT System and How Does It Work?
The new AI Hearing Recording and Transcriptions (HeaRT) system has been rolled out and is helping to streamline processes associated with the automatic recording and transcription of hearings related to Social Security matters. By streamlining and replacing old systems and equipment used to record Social Security–related hearings, SSA can allocate budget and human resources to other areas of its operation.
How the HeaRT System Aims to Assist Social Security Disability Benefit Recipients
- SSA wants to use the HeaRT system to improve the accuracy and quality of hearing recordings for disability cases.
- The HeaRT system seeks to replace bulky and outdated equipment used by manual transcription processes, allowing SSA employees to focus on other key areas of SSA operations.
- According to SSA, 500,000 Social Security customers will benefit from this new system per year.
- SSA states that the HeaRT system will “result in fewer hearing delays or cancellations due to equipment failure or technical issues,” allowing hearings to proceed more reliably and helping ensure claimants experience faster resolution of their cases.
Potential Roadblocks for the HeaRT System from a Technological Perspective
Artificial-intelligence (AI) systems that transcribe spoken language are typically known as Automatic Speech Recognition (ASR) systems. Their primary function is to convert speech, like testimonies from a hearing, into written text. Most people interact with ASR systems regularly and are often unaware they are using these tools in their day-to-day lives. Google Translate, Google Assistant, Apple’s Siri, and Amazon Alexa are examples of ASR programs that are embedded into many of our daily routines and workflows. Like all AI algorithms, Automatic Speech Recognition (ASR) systems can have their own unique shortcomings. ASR systems can be negatively impacted by the following circumstances:
- Especially in a hearings office, rustling papers, noisy HVAC systems, individuals talking over each other, or low-quality microphones can impact ASR system transcription performance.
- ASR systems will underperform with fast speech, testimony that may come from individuals with regional accents or non-native English speakers, or speech that is less audible to an AI system (soft-spoken speech).
- Homophones, legal jargon, filler words, or stutters can affect the quality output of the transcription.
- If the ASR model does not have robust training data, lack of context awareness and diarization errors (i.e., attribution for who is saying what/when someone new begins speaking) will negatively impact the output transcription.
- The system might not accurately attribute punctuation—sometimes inferring that pauses are periods, which is not always correct in spoken English.
What Can Happen if the ASR Model Is Paired with an LLM for Transcribing Social Security Hearings
If the system uses a hybrid Large Language Model (LLM) to help process these transcriptions, the final transcription can be subject to AI hallucinations, which are outputs that are either partly or entirely fabricated due to the model’s reliance on patterns rather than verified inputs.
What Will Be Needed to Ensure That the HeaRT System Remains Accurate
While the new Hearing and Recording Transcriptions (HeaRT) system represents a step toward modernizing the Social Security Administration, there will need to be significant human oversight to ensure the accurate recording and transcribing of hearings. The attorneys at Michael Armstrong Law have always thoroughly reviewed transcribed hearing recordings as a general practice; thus, we already have processes in place to protect the due-process rights of our clients. We hope this new system will improve the quality of these recordings, yet only time will tell. This is another reason why it is important to have a proven and knowledgeable team of advocates on your side when seeking disability benefits.