FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks

Project Webpage
[ code]

Abstract

Large language models have revolutionized natural language processing through self-supervised pretraining on massive datasets. Inspired by this success, researchers have explored adapting these methods to speech by discretizing continuous audio into tokens using neural audio codecs. However, existing approaches face limitations, including high bitrates, the loss of either semantic or acoustic information, and the reliance on multi-codebook designs when trying to capture both, which increases architectural complexity for downstream tasks. To address these challenges, we introduce FocalCodec, an efficient low-bitrate codec based on focal modulation that utilizes a single binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec delivers competitive performance in speech resynthesis and voice conversion at lower bitrates than the current state-of-the-art, while effectively handling multilingual speech and noisy environments. Evaluation on downstream tasks shows that FocalCodec successfully preserves sufficient semantic and acoustic information, while also being well-suited for generative modeling.

This page presents the samples utilized in the user study. Hereafter, we include also EnCodec and DAC originally excluded from the user study due to the poor quality compared to the other baselines. We also include voice conversion samples.

Please note:

Chrome is the preferred browser. The samples may not always run in Firefox.

Use of headphones/earphones recommended.

Please tune your volume appropriately before playing any sample.

The audio files may take some time to load. If they do not load even after waiting, it can be a temporary hosting server issue. In that case, please visit again after sometime.

Audio Samples: User Study

Sample A

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample B

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample C

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample D

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample E

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample F

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample G

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample H

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample I

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Sample J

Reference
FocalCodec@50 (0.65 kbps) (Ours)
FocalCodec@25 (0.33 kbps) (Ours)
FocalCodec@12.5 (0.16 kbps) (Ours)
BigCodec (1.04 kbps)
DAC (1.00 kbps)
EnCodec (1.50 kbps)
Mimi (0.69 kbps)
SemantiCodec (0.65 kbps)
SpeechTokenizer (1.00 kbps)
Stable Codec (0.70 kbps)
WavLM6-KM (0.45 kbps)
WavTokenizer (0.48 kbps)

Audio Samples: Voice Conversion

Input
Reference
FocalCodec@50 (0.65 kbps)
SpeechTokenizer (1.00 kbps)