HeyJay!: A Corpus of Atypical Speech for Spoken Language Understanding and Automatic Speech Recognition, United States, 2023-2024 (ICPSR 39448)

Time Period(s) View help for Time Period(s)

Date of Collection View help for Date of Collection

Data Collection Notes View help for Data Collection Notes

Study Purpose View help for Study Purpose

The purpose of the study was to use part of HeyJay! to create synthetic speech on new intent domains (to replicate SLURP), train SLU models, and then evaluate the resulting models with speech from actual atypical speakers.

Study Design View help for Study Design

Participants were asked to record a minimum of 320 complex commands, although not all participants finished the assigned tasks. Responses were recorded using the open-source web platform, Hermespeech Recorder.

The recordings took place in two different scenarios:

• In the clinic, supervised a research coordinator and used the same computer equipped with a Focusrite Scarlett external soundcard and an AKG headset condenser microphone.
• At home using the participant's own computer and microphone. In these cases, the participants received a video with clear recording instructions followed by a short training over videoconference with a research assistant that ensured that they understood the recording requirements. All participants first recorded three test utterances that were reviewed by the researchers of this project to ensure that the recording conditions were optimal and no significant background noise or channel distortions were present. In the cases in which researchers observed recording issues, participants were guided on how to solve them before starting the actual recordings.

Some participants were recorded in one scenario of their choice or both. All recordings had a single channel and a sampling rate of 22 kHz. Participants were presented with various sentences in English to be read as if they were talking to a speech assistant named Jay. For that reason, all sentences started with the wakeword Hey, Jay which gives the name to the corpus. The sentences overlapped in intent and transcriptions with those from the Fluent Speech Commands dataset (FSC) for 18 participants. The rest of the speakers recorded sentences overlapped with transcription and intent with the SLURP (short for Spoken Language Understanding Resource Package) dataset. These two references were selected as they represent basic (FSC) and challenging (SLURP) benchmarks of SLU systems. This will also allow researchers to combine HeyJay! with these two benchmarks to train larger models in future experiments.

In the portion of HeyJay! compatible with the FSC dataset (HeyJay-FSC), the intent of an utterance was labeled using three slots:

• Action
• Object
• Location

For instance, if the utterance is "Hey Jay, turn off the living room's lights" the action will be "turn off", the object, "lights", and the location "living room". As in the original FSC dataset, HeyJay-FSC contains 248 unique phrases and 31 different intents.

On the other hand, the sentence annotation from SLURP (HeyJay-SLURP) follows a more complex hierarchical structure:

1. Scenario: Represents the general domain or context of the utterance (e.g., "calendar", "weather").
2. Intent: Captures the user's specific objective within that domain (e.g., "calendar/set_event", "weather/temperature").
3. Slots and Actions: Semantic slot annotations that identify key elements in the utterance such as times, places, names, or items that are essential to fulfilling the task.

For instance, in the utterance "Set a reminder for 3 PM tomorrow", the system would tag it under the "reminder" scenario, assign it the intent "reminder/set", and recognize "3 PM May 12 2025" as a time-related slot. The SLURP dataset offers a collection of intent annotations tailored for goal-directed spoken interactions across common real-world domains, including topics like weather, music, travel, and cooking. This comprehensive annotation approach makes SLURP particularly valuable for developing and benchmarking SLU models that need to interpret both user intent and detailed semantic information.

Time Method View help for Time Method

Universe View help for Universe

Adults with confirmed neuromotor disorders and perceived speech impairment. The speech impairments ranged from mild to severe, and the diagnoses include Parkinson's Disease, Ataxia (Episodic, Spinocerebellar, Cerebellar), Multiple System Atrophy, Amyotrophic Lateral Sclerosis, Stroke, and Dystonia.

Unit(s) of Observation View help for Unit(s) of Observation

Data Type(s) View help for Data Type(s)

Mode of Data Collection View help for Mode of Data Collection

Description of Variables View help for Description of Variables

The data includes variables about disease etiology, severity of the speech impairment, and annotation of the quality of voice of each participant. In particular, the overall severity of atypical speech characteristics included overall dysarthria severity, overall articulatory severity, imprecise consonant articulation, prolonged phonemes, repeated phonemes, irregular articulatory breakdowns, presence of distorted vowels, overall voice quality, harsh voice, hoarse/wet voice, breathy voice, strained/ strangled voice, voice stoppages, and voice flutter. Demographic variables include age and sex.

Presence of Common Scales View help for Presence of Common Scales

Each of the three expert speech and language pathologists received a set of eight audio samples per speaker and evaluated the overall severity of atypical speech characteristics. Ratings ranged from 0 to 4, with 0 representing typical speech and 4 reflecting a highly impaired condition, in accordance with the Rating Scale for Deviant Speech Characteristics. Not all pathologists were able to evaluate all participants or traits.

Original Release Date View help for Original Release Date

2025-08-19

Version History View help for Version History