Haru-4B: Low-Latency Multilingual Full-Duplex Spoken Dialogue AI

Haru-4B is a 5.3B parameter AI model for real-time, continuous multilingual conversation that listens and speaks simultaneously, released on Hugging Face.

By Sci Herald Editorial Team ·

Released on Hugging Face by higurashi-ai — view the original study

Haru-4B: Low-Latency Multilingual Full-Duplex Spoken Dialogue AI

TL;DR: Haru-4B is a new 5.3 billion-parameter AI model that conducts real-time, continuous spoken conversations in nine languages. It is significant because it operates in full-duplex, meaning it listens and speaks at the same time, just like a human on a phone call.

Key Facts

Haru-4B at a Glance

SpecificationDetail
TaskAudio-to-audio
Parameters5,270,000,897
LicenseApache 2.0
Languagesen, es, pt, de, fr, zh, ja, ru, ko
Frame Rate7.5 Hz
Audio CodecVibeVoice, 24 kHz
Checkpoint Size~11 GB (BF16)

What's New

Haru-4B is a pretrained base model for research that introduces a novel architecture for real-time, continuous spoken dialogue. Its core innovation is native full-duplex operation. Every 133 milliseconds, the model processes one frame of user audio and simultaneously generates one frame of its own audio and text output. This is achieved by summing user audio, model audio, and model text into a single position of its language model, eliminating the wait time typically associated with turn-taking in AI conversations.

Unlike many audio models that use a separate transformer and discrete tokens, Haru-4B generates audio as a 64-dimensional latent per frame. A small, efficient 4-layer MLP head with MeanFlow sampling produces this audio in a single step directly from the language model's hidden state. The model also features character control via a text prompt and voice control through a short 3-second to 3-minute reference audio clip of a speaker.

Recommended Settings

The model card provides specific guidance for generation. The audio head uses a guidance scale that can be set from 1 to 2, with a default value of 1.5. For the lowest possible latency during interaction, the publisher recommends using wired headphones and a microphone, as Bluetooth devices can add over 100 milliseconds of delay.

Hardware Requirements

QuantizationWeights-Only Memory Estimate
16-bit (BF16)~11 GB
8-bit~6 GB
4-bit~3 GB

Table: Our memory estimates for loading the Haru-4B model weights. Runtime requires additional memory for context and key-value cache.

Can You Run It Locally? Hardware Requirements

Haru-4B is small enough for most recent gaming GPUs and Apple Silicon laptops once quantized to 4-bit. The table below is Sci Herald's own estimate from the parameter count, not a figure from the publisher.

PrecisionWeightsPractical minimumNVIDIA GPUApple Silicon Mac
16-bit (BF16/FP16, full quality)~11 GB~13 GB16 GB GPU (e.g. RTX 4060 Ti 16 GB, RTX 5080)24 GB unified memory or more
8-bit (Q8 / FP8)~5.3 GB~6.3 GB8 GB GPU (e.g. RTX 4060)16 GB unified memory or more
4-bit (Q4 / NF4, small quality loss)~2.6 GB~3.2 GB8 GB GPU (e.g. RTX 4060)16 GB unified memory or more

How we estimate: weights = parameters × bytes per parameter; "practical minimum" adds about 20% for the runtime and a short context. Long contexts, batching and image or audio inputs need more memory.

How to Run It

The official repository provides multiple implementations. For running on an NVIDIA GPU with approximately 12 GB of memory, the following commands are provided to clone the repository and launch a server.

``bash git clone https://github.com/higurashi-ai/Haru && cd Haru pip install -e./haru python -m haru.server --hf-repo higurashi-ai/Haru-4B ``

Developers can also integrate the model directly into a Python application using the provided API to load the model, prepare a voice reference, and manage a continuous dialogue session, as shown in the code snippet from the model card.

License and Commercial Use

Haru-4B is released under the permissive Apache License 2.0. This license allows for commercial use, modification, and distribution, making it suitable for both academic research and commercial product development.

Limitations

As stated on its model card, Haru-4B is a pretrained base model released for research. It is designed to hold a natural conversation but is not a reliable source of facts and does not follow instructions in the manner of an AI assistant. A critical limitation and ethical consideration is its voice imitation capability; users must only provide reference audio clips for voices they have the right to use.

Frequently Asked Questions

Is Haru-4B free for commercial use?

Yes, the model is released under the Apache 2.0 license, which permits commercial use, modification, and distribution.

What hardware is needed to run Haru-4B?

As shown in the hardware table above, the model can be run on a single high-end consumer GPU, with memory requirements varying based on the quantization level used.

Can Haru-4B speak multiple languages?

Yes, the Haru-4B model supports conversation in nine languages: English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian, and Korean.

Does the model come with pre-set voices?

No, Haru-4B ships without voice clips. Users must provide their own reference audio clip of a speaker to set the model's voice, ensuring they have the right to use that voice.

Source: higurashi-ai/Haru-4B on Hugging Face