Beyond Words: Signal-Based AI for Neurodivergent and Nonspeaking Users

Executive Summary

Artificial intelligence is rapidly reshaping how we interpret and express human communication. Yet these systems often fail to recognize and respond to the diverse signals used by nonspeaking and neurodivergent individuals, treating gaze, gesture, metaphor, or silence as noise, delay, or absence. This gap excludes millions from full participation in digital life and limits AI's potential as a medium of human connection.

This paper centers on the lived experiences of individuals with intellectual and developmental disabilities (IDD), including Autism Spectrum Disorder (ASD) and nonspeaking individuals with Cerebral Palsy (CP). We approach all communication as intentional and valid, even when nonlinear, nonverbal, or nonconforming to typical language models.

We propose a signal-first, ethically adaptive AI framework rooted in multimodal inputs, large language models (LLMs), and multi-agent systems (MAS). This paper reimagines AI not as a corrective tool but as a collaborative system that learns from users, extending dignity, flexibility, and agency to those long marginalized by rigid, voice-dependent design.

The Communication Gap

Imagine a user typing: "the bees... are loud tonight, and the walls... breathing, again, again. I'm trying not to touch the edges." For a neurotypical interface, this may be interpreted as poetic noise. For a neurodivergent user, it may be a precise sensory report of overwhelm.

Today's AI systems often reduce such expressions, those that are metaphorical, delayed, gestural, or nonlinear, into categories of error or ambiguity. These systems miss the signal behind the signal. When meaning is associative or expressed through pauses, rhythm, or tension, AI frequently substitutes interpretation with normalization or default correction.

Communication from neurodivergent and nonspeaking users requires systems that listen differently. Systems that can hold space for uncertainty without overriding the user's intent. The gap is not in the user's expression. The gap is in the system's listening.

A Broken Bridge

Mainstream AI systems are built on assumptions of linearity, speed, and clarity. But what happens when a user's communication does not conform? For those with CP, ASD, or other developmental conditions, signals may arrive through a blink, a gaze, a stillness.

Too often, these signals go unsupported or untranslated, treated as broken when in fact they are whole but unrecognized. When no path exists between input and interpretation, the system fails. Not the user.

A truly inclusive system builds interpretive scaffolding: layers that translate gaze into words, rhythm into syntax, stillness into intentional pause. This is not remediation. It is translation.

Our Proposed Approach

An inclusive future for AI begins with three commitments:

• Recognize atypical patterns: Design systems to identify and respect communication forms beyond speech and text.
• Design for neurodiversity: Move from adapting to neurodivergent users toward designing with their input from inception.
• Build adaptive responses: Ensure AI responds in ways that mirror, learn from, and evolve with individual expression over time.

Multimodal Input Systems

Communication happens in layers. We propose AI systems that begin with multimodal listening. A glance. A still hand. An irregular pause. Each a potential signal.

These systems must be designed to identify and respect communication forms beyond speech and text, adapting with the user rather than against them, and evolving their responses as individual expression changes over time.

Integrating LLMs for Inclusive Dialogue

Large Language Models are uniquely positioned to interpret nonlinear, associative, or metaphor-rich communication. When trained ethically and aligned with individual pacing, LLMs offer a bridge to meaningful exchange.

Case studies

• ASDChat: Embeds LLMs into clinically validated frameworks like VBMAPP, matching therapist effectiveness for autistic children.
• Gaze-Based CP Assistant: Learns from gaze patterns and adjusts response timing, extending agency to users with delayed motor planning.
• Metaphor-Sensitive Agent: Interprets poetic language ("My arms are fireworks") and responds with nuanced follow-up ("Are you feeling tense or anergized?").

These examples show LLMs functioning not as translators of disorder but as interpreters of difference.

Insights from LLM Case Studies

Key principles for responsible use of LLMs in neurodivergent contexts:

• Clinical alignment: Use validated frameworks (such as VBMAPP) to ground AI interaction in evidence-based approaches.
• Ethical safeguards: Employ emotional tone control, user override options, and human oversight to prevent coercion.
• Personalization over time: LLMs should evolve with the user, reflecting changes in mood, preference, or cognitive rhythm.
• Context-aware limits: These systems are not replacements for human support but extensions of a communication environment that prioritizes dignity.

A signal-first system treats each user as a dynamic communicator, not a subject to be corrected but a partner in mutual meaning-making.

Multi-Agent Systems for Communication, Ethics, and Learning

Multi-Agent Systems (MAS) distribute interpretation across multiple agents, each focused on a unique input: gaze, timing, tone, gesture. This model mirrors how humans interpret complexity.

One agent may track timing. Another may focus on metaphor. A third might arbitrate ethical coherence, ensuring suggestions respect user pacing, consent, and preference.

The power of MAS lies in collective interpretation: not assuming intent, but allowing for emergence, and discovering it patiently, contextually, and with integrity.

Building on Our Foundation

To move this work forward, we propose:

• Publishing expanded case studies across disability categories.
• Deepening MAS and LLM integration models.
• Formalizing a signal-first design protocol for inclusive AI.
• Collaborating with standards bodies to ensure ethical deployment from the start.

Conclusion: Designing With, Not For

Neurodivergent and nonspeaking users are not afterthoughts. They are the architects of what is next. Their communication, rich with timing, pattern, and poetics, demands systems that reflect their complexity without judgment.

What is designed for the neurodivergent community applies across entire populations of users.

We do not build for them. We build with them. And in doing so, we design technology that listens, waits, learns, and adapts, not only for one group, but for all.

Author Biographies

Rebeca Aguirre

Rebeca Aguirre is the author of Amplifying Voices and a thought leader in signal-based AI design for self-expression. A graduate of the USC Annenberg School for Communication and Journalism, she brings lived and professional experience to her work on adaptive systems that support meaningful communication in caregiving relationships.

Dr. Rodney Sappington

Dr. Rodney Sappington is a researcher, ethnographer, and leader in the field of artificial intelligence. A graduate of Johns Hopkins University, he has over 20 years of experience in developing neural technologies, medical diagnostics, computer vision, and AI safety applications. His current research focuses on agent-based systems in areas of decision making, machine consciousness, and moral dilemma.