C++17 GGML/GGUF inference for SAM Audio — Meta's text- and span-prompted audio source separation model. SAM can isolate a sound ("man speaking", "footsteps", "a car horn") from a mix using natural language.

Audio-only port: no vision encoder, no visual prompting, no rerankers. Text prompting supported.

Given an audio or video file and a text description, audiosam separates it into two stems:

• target — audio matching the description
• residual — everything else

This is source separation / stem extraction, not audio generation: the model conditions on and reconstructs the input audio, it does not synthesize new sound.

git clone --recursive https://github.com/audiohacking/audiosam.cpp
cd audiosam.cpp

# build (see Build section for CUDA/Metal)
make cpu

# download + convert official checkpoints (requires HF access + SAM License acceptance,
# see Models section) — produces models/{small,base,large}/
./models.sh

# separate a speaking voice from a video — works directly on mp4/mkv/etc,
# the audio track is decoded via ffmpeg. This is upstream's own office.mp4
# example (github.com/facebookresearch/sam-audio/tree/main/examples).
curl -LO https://github.com/facebookresearch/sam-audio/raw/main/examples/assets/office.mp4
./build-cpu/audiosam -m models/small -i office.mp4 -p "man speaking" -o speech.wav --residual background.wav

Input can be a WAV file or any container ffmpeg can decode (mp4, mkv, mp3, ...) — video files work directly, only the audio track is used.

audiosam -m <model_dir> -i in.wav -p "text prompt" [-o target.wav]
         [--spans "+0.5:2.0,-3.0:4.0"] [--residual residual.wav]
         [--steps 16] [--seed 42]

• -m — directory containing codec.gguf, dit.gguf, t5_base.gguf, spiece.model (one per model size, see below)
• -p — text description of the sound to isolate
• --spans — optional manual time-span prompting: +start:end marks a span that should contain the target, -start:end marks one that should be ignored (seconds, comma-separated)
• --steps — ODE solver steps (default 16, matching upstream)
• --seed — noise seed for reproducibility

make cpu          # CPU-only
make cuda         # NVIDIA CUDA (incl. Grace-Blackwell / GB10 unified-memory iGPU)

Metal (AUDIOSAM_METAL=ON via CMake) is wired but not yet validated on real hardware — see DEVROADMAP.md.

Requires CMake ≥ 3.16, a C++17 compiler, libsentencepiece, and (for non-WAV input) ffmpeg on PATH.

make test         # CPU parity tests (ctest)
make test-cuda    # CUDA parity tests (looser tolerances — GPU matmuls aren't bit-identical to CPU)

Checkpoints are gated on Hugging Face under Meta's SAM License — request access at facebook/sam-audio-small, -base, -large, then hf auth login and run ./models.sh (downloads + converts all three sizes; pass e.g. ./models.sh small to fetch one).

models.sh converts straight from the official checkpoint.pt (see convert/convert_codec_from_checkpoint.py, convert/convert_dit_from_checkpoint.py). Converted GGUFs are not redistributed here due to the SAM License.

wav → resample 48kHz mono → DAC-VAE encode → [T5 text encode ‖ 16-step midpoint-ODE flow-matching DiT] → DAC-VAE decode → target + residual

The DiT jointly denoises target and residual as stacked latent channels in a single pass. See DEVROADMAP.md for the full architecture writeup, module-by-module parity numbers, and the non-obvious porting pitfalls (attention head layout, ONNX weight de-anonymization, the AudioSeal watermarker tap, GPU ConvTranspose performance) discovered along the way.

Text prompting is implemented and validated end-to-end (parity-tested per module + a listening check against upstream's office.mp4 example). Span prompting (--spans) is implemented but not yet validated against upstream's span-prompting example. See DEVROADMAP.md for the phase checklist and open items.