HUEBER

Thomas

CNRS researcher

PublicationsMedia/Press/TalkSoftwaresSupervisionTeachingResearchAbout me

The objective here is to design a system capable of capturing and interpreting a normally "articulated" but "non-vocalized" speech. The speaker moves his various articulators (jaw, tongue, lips, soft palate) but he does not send air into his oral and nasal cavities; he therefore emits (practically) no sound. The principle of a "silent speech interface" is (1) to capture the "inaudible" traces of this "silent speech", such as the movements of the articulators or the nervous and muscular activity, and (2) to transform them either into a sequence of words (recognition), or directly, and in real time, into an "audible" speech signal (direct synthesis). The targeted applications are the assistance to laryngectomized persons and to persons suffering from a neurodegenerative disease that can lead to a loss of oral communication. The approach I am mainly studying is based on the capture of articulatory activity by ultrasound and video imaging, and its processing using machine learning and speech synthesis techniques. 

Text-to-Speech (TTS) systems are now of sufficient quality to be deployed in general purpose applications and to be used by people who have lost the use of their voice. However, a classical TTS synthesizer works at the sentence level: text analysis and sound synthesis are triggered each time the user has finished typing a complete sentence. The knowledge of the beginning and end boundaries of a sentence is important for its linguistic analysis, especially for determining its syntactic structure, which is important for determining the prosody, the "melody" of the synthesized voice. However, this paradigm introduces an important latency (proportional to the length of the sentence) which can be at the origin of a certain frustration for the communication partner then constrained to wait for the end of each sentence. Incremental TTS aims at improving the interactivity of an oral communication carried out by means of a TTS, by delivering, as the text is entered, a speech synthesis of a quality close to the one obtained with the help of a traditional TTS (working at the sentence level). The speech synthesis "shadows" the text input (see figure below, taken from Maël Pouget's thesis). One of our approach consists in exploiting neural language models (GPT-like) to predict the future of the textual input and to integrate this prediction into the speech synthesis system (Brooke Stephenson's PhD).

Hueber, T., Girin, L., Alameda-Pineda, X., Bailly, G. (2015), "Speaker-Adaptive Acoustic-Articulatory Inversion using Cascaded Gaussian Mixture Regression", in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 12, pp. 2246-2259 (preprint pdf, source code) 

Girin, L, Hueber, T., Alameda-Pineda, X ,(2017) Extending the Cascaded Gaussian Mixture Regression Framework for Cross-Speaker Acoustic-Articulatory Mapping, in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 3, pp. 662-673 (preprint pdf, source code)

Axis 2: Self-supervised learning of sensori-motor speech representation

Constraining the latent representations of a variational auto-encoder 

In the context of a collaboration with the company Arturia (CIFRE thesis of Fanny Roche), we are interested in the control of the timbre of a synthetic sound from high level symbolic descriptors. The targeted synthesizer must for example allow to make a sound more "aggressive", "warmer", etc.. We have proposed a technique based on VAE variational autoencoders, whose latent dimensions we have constrained to approach a set  "psychoacoustic" dimensions, extracted from human evaluation of musical sounds.

Roche, F., Hueber, T., Garnier, M., Limier, S., & Girin, L. (2021). Make That Sound More Metallic: Towards a Perceptually Relevant Control of the Timbre of Synthesizer Sounds Using a Variational Autoencoder. Transactions of the International Society for Music Information Retrieval, 4(1), pp. 52–66.

Roche, F.,. Hueber, T., Limier, S, Girin. L., “Autoencoders for music sound modeling : a comparison of linear, shallow, deep, recurrent and variational models”. In Proc. of SMC. Malaga, Spain, 2019.

We then applied this regularization technique to speech. We were interested in simulating a motor simulation process during the processing of an auditory stimulus by the brain. Here, the VAE encoder is used to "project" an auditory stimulus to a latent space whose dimensions are constrained to encode articulatory information. We were able to show that this articulatory constraint improves learning speed and model accuracy, and gives better performance in a speech denoising task than an unconstrained VAE.

Georges M-A, Girin L., Schwartz J-L, Hueber, T., "Learning robust speech representation with an articulatory-regularized variational autoencoder", Proc. of Interspeech, 2021, pp, 3335-3349 (preprint)

Self-supervised learning of acoustic-articulatory relationships

Most of the artificial speech perception and production systems (i.e. ASR, articulatory synthesizer, TTS) exploit statistical models whose parameters are generally estimated by using supervised machine learning. This supervised learning of the links between the different spaces of speech representation (acoustic, articulatory, linguistic) seems to be far from the way a human perceives, processes and produces speech. In particular, we are interested in how a child learns the relationship between the sound of speech and the associated articulatory gesture. We are developing a communicating agent, based on deep learning techniques, capable of learning these sensory-motor relationships in a self-supervised manner, by attempting to repeat the auditory stimuli it perceives (Marc-Antoine George's PhD).    

Georges M-A, Diard, J., Girin, L., Schwartz J-L, Hueber, T.,  "Repeat after me: self-supervised learning of acoustic-to-articulatory mapping by vocal imitation", Proc. of ICASSP, to appear

Predictive coding of auditory and audiovisual speech

One of the common pre-text tasks for self-supervised representation learning  is the prediction of the future of an input signal, based on its past. This paradigm, called predictive coding, is also very present in cognitive science and neuroscience. It assumes that our brain implements an online inference algorithm of future sensory inputs, from past sensory inputs. For speech, this prediction would be based on the search for regularities in the acoustic signal, but would also recruit the motor and linguistic levels. Using convolutional networks trained on large multi-speaker databases, we quantified the usefulness of such a strategy, for different time spans (from 25 to 150ms), for auditory and audiovisual speech.

Hueber, T., Tatulli, E., Girin, L., Schwartz, J-L., "Evaluating the potential gain of auditory and audiovisual speech predictive coding using deep learning", Neural Computation, vol. 32 (3), to appear (preprint, source code, dataset/pretrained models)