Aditya R. Vaidya

Email: avaidya [at] utexas.edu

recent & upcoming presentations

Jul 19, 2025

ICML 2025 Workshop on Machine Learning for Audio

BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Language

Nov 15, 2023

Society for Neuroscience (SfN)

Replicating fast auditory intracranial responses using fMRI and large neural network models

selected publications

images/publication_preview/brainwavlm_arxiv2025bad.png

BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Language

Nishitha Vattikonda^*, Aditya Vaidya^*, Richard Antonello, Alexander Huth

2025

Bib Url

:::bibtex
@misc{brainwavlm_arxiv2025bad,
    author = "Vattikonda, Nishitha and Vaidya, Aditya R. and Antonello, Richard J. and Huth, Alexander G.",
    title = "BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Language",
    author+an = "1=jointfirst;2=jointfirst",
    year = "2025",
    eprint = "2502.08866",
    archivePrefix = "arXiv",
    primaryClass = "cs.CL"
}

images/publication_preview/lm-repeating-text_conll2023.png

Humans and language models diverge when predicting repeating text

Aditya Vaidya, Javier Turek, Alexander Huth

In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), December 2023

Abs Bib Url

Language models that are trained on the next-word prediction task have been shown to accurately model human behavior in word prediction and reading speed. In contrast with these findings, we present a scenario in which the performance of humans and LMs diverges. We collected a dataset of human next-word predictions for five stimuli that are formed by repeating spans of text. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory (or in-context learning) begins to play a role. We traced the cause of this divergence to specific attention heads in a middle layer. Adding a power-law recency bias to these attention heads yielded a model that performs much more similarly to humans. We hope that this scenario will spur future work in bringing LMs closer to human behavior.

:::bibtex
@inproceedings{lm-repeating-text_conll2023,
    author = "Vaidya, Aditya and Turek, Javier and Huth, Alexander",
    editor = "Jiang, Jing and Reitter, David and Deng, Shumin",
    title = "Humans and language models diverge when predicting repeating text",
    booktitle = "Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)",
    month = "December",
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    doi = "10.18653/v1/2023.conll-1.5",
    pages = "58--69",
    abstract = "Language models that are trained on the next-word prediction task have been shown to accurately model human behavior in word prediction and reading speed. In contrast with these findings, we present a scenario in which the performance of humans and LMs diverges. We collected a dataset of human next-word predictions for five stimuli that are formed by repeating spans of text. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory (or in-context learning) begins to play a role. We traced the cause of this divergence to specific attention heads in a middle layer. Adding a power-law recency bias to these attention heads yielded a model that performs much more similarly to humans. We hope that this scenario will spur future work in bringing LMs closer to human behavior."
}

images/publication_preview/fmri-encoding-scaling_neurips2023.png

Scaling Laws for Language Encoding Models in {{fMRI}}

Richard Antonello, Aditya Vaidya, Alexander Huth

Advances in Neural Information Processing Systems, December 2023

Bib Pdf Url

:::bibtex
@article{fmri-encoding-scaling_neurips2023,
    author = "Antonello, Richard and Vaidya, Aditya and Huth, Alexander",
    title = "Scaling Laws for Language Encoding Models in {{fMRI}}",
    year = "2023",
    month = "December",
    journal = "Advances in Neural Information Processing Systems",
    volume = "36",
    pages = "21895--21907",
    langid = "english"
}

images/publication_preview/ssl-speech-brain_icml2022.png

Self-Supervised Models of Audio Effectively Explain Human Cortical Responses to Speech

Aditya Vaidya, Shailee Jain, Alexander Huth

In Proceedings of the 39th International Conference on Machine Learning (ICML), 17--23 jul 2022

Abs Bib Pdf Url

Self-supervised language models are very effective at predicting high-level cortical responses during language comprehension. However, the best current models of lower-level auditory processing in the human brain rely on either hand-constructed acoustic filters or representations from supervised audio neural networks. In this work, we capitalize on the progress of self-supervised speech representation learning (SSL) to create new state-of-the-art models of the human auditory system. Compared against acoustic baselines, phonemic features, and supervised models, representations from the middle layers of self-supervised models (APC, wav2vec, wav2vec 2.0, and HuBERT) consistently yield the best prediction performance for fMRI recordings within the auditory cortex (AC). Brain areas involved in low-level auditory processing exhibit a preference for earlier SSL model layers, whereas higher-level semantic areas prefer later layers. We show that these trends are due to the models’ ability to encode information at multiple linguistic levels (acoustic, phonetic, and lexical) along their representation depth. Overall, these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.

:::bibtex
@InProceedings{ssl-speech-brain_icml2022,
    author = "Vaidya, Aditya R. and Jain, Shailee and Huth, Alexander",
    editor = "Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan",
    title = "Self-Supervised Models of Audio Effectively Explain Human Cortical Responses to Speech",
    booktitle = "Proceedings of the 39th International Conference on Machine Learning (ICML)",
    pages = "21927--21944",
    year = "2022",
    volume = "162",
    series = "Proceedings of Machine Learning Research",
    month = "17--23 Jul",
    publisher = "PMLR",
    abstract = "Self-supervised language models are very effective at predicting high-level cortical responses during language comprehension. However, the best current models of lower-level auditory processing in the human brain rely on either hand-constructed acoustic filters or representations from supervised audio neural networks. In this work, we capitalize on the progress of self-supervised speech representation learning (SSL) to create new state-of-the-art models of the human auditory system. Compared against acoustic baselines, phonemic features, and supervised models, representations from the middle layers of self-supervised models (APC, wav2vec, wav2vec 2.0, and HuBERT) consistently yield the best prediction performance for fMRI recordings within the auditory cortex (AC). Brain areas involved in low-level auditory processing exhibit a preference for earlier SSL model layers, whereas higher-level semantic areas prefer later layers. We show that these trends are due to the models’ ability to encode information at multiple linguistic levels (acoustic, phonetic, and lexical) along their representation depth. Overall, these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex."
}