Can I Run Nemotron 3.5 Transcribe Locally? Yes, on 16GB
NVIDIA released Nemotron 3.5 Transcribe on 28 October 2026. It transcribes 40 languages, labels speakers, and takes recordings up to one hour. The model has 34B parameters but reads only about 4.3B per token, so a 16GB GPU runs it. llama.cpp added support on 3 November.
Bottom Line
- Yes, a 16GB GPU runs it. The smallest decoder (AD-Q2_0_M) is 12.64 GiB, plus a 1.56 GiB encoder. AtomicChat reports a 15.6 GiB peak on a 16GB card.
- A 24GB card is the comfortable tier. The 4-bit AD-IQ4_NL decoder is 18.33 GiB, and it transcribes as well as the BF16 model on AtomicChat’s test.
- It is an offline transcriber, not a live one. Give it a recording of up to one hour. It labels speakers and supports 40 languages.
- Pick the runtime first. NVIDIA’s NeMo-Speech.cpp gives word timestamps. Upstream llama.cpp (b12087, 3 November) does not.
Choosing a card for this? See the tested hardware list by VRAM tier.
What Nemotron 3.5 Transcribe Is
NVIDIA calls it a Speech-Augmented Language Model. An audio encoder turns the recording into tokens. A language model then writes the transcript.
| Spec | Value |
|---|---|
| Hugging Face repo | nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B |
| Parameters | 34B total, about 4.3B active, encoder included |
| Language model | Nemotron 3.5 Lightning 30B-A3B (Mamba-2 and Transformer MoE) |
| Audio input | 16kHz mono, recordings up to 1 hour |
| Context | 65,536 tokens |
| Languages | 40 |
| Tasks | Transcription, speaker labels, translation to and from English, language ID, questions about the audio |
| License | OpenMDW-1.1 |
The active size is the number that matters for hardware. About 3.5B of the 4.3B active parameters are in the language model, and about 800M are in the audio encoder. The rest of the 34B sits in experts that each token mostly skips.
Every GGUF Size
We summed the Hugging Face file trees on 3 November 2026. GiB = bytes / 1,073,741,824. Every setup needs the encoder file as well.
| File | Repo | Bytes | GiB | With encoder |
|---|---|---|---|---|
| Encoder-F16 (required) | AtomicChat | 1,670,410,080 | 1.56 | — |
| AD-Q2_0_M | AtomicChat | 13,572,208,800 | 12.64 | 14.20 |
| Q2_K | Jmica | 13,946,026,784 | 12.99 | 14.55 |
| AD-Q2_0_L | AtomicChat | 15,547,824,288 | 14.48 | 16.04 |
| Q3_K_S | Jmica | 17,294,261,024 | 16.11 | 17.67 |
| IQ4_NL | sakasegawa | 18,997,225,856 | 17.69 | — (no encoder in repo) |
| AD-IQ4_NL | AtomicChat | 19,678,656,672 | 18.33 | 19.89 |
| AD-NVFP4 | AtomicChat | 19,678,656,416 | 18.33 | 19.89 |
| Q4_K_S | Jmica | 21,672,721,184 | 20.18 | 21.74 |
| AD-IQ4_NL_L | AtomicChat | 22,552,279,200 | 21.00 | 22.56 |
| AD-Q5_0_XL | AtomicChat | 29,257,398,432 | 27.25 | 28.81 |
| Q8_0 | AtomicChat | 35,004,643,232 | 32.60 | 34.16 |
The F16 file in the sakasegawa repo is 62.79 GiB. NVIDIA also publishes FP8 and NVFP4 safetensors for vLLM.
What Fits Your Machine
These rows are our arithmetic, except where a row says community-reported. Leave about 1-2 GiB for buffers and the audio cache. We did not measure that number.
| Machine | Decoder | Total GiB | Verdict |
|---|---|---|---|
| 16GB GPU (RTX 5060 Ti 16GB, RTX 4060 Ti 16GB) | AD-Q2_0_M | 14.20 | Fits. 15.6 GiB peak, community-reported (AtomicChat) |
| 24GB GPU (RTX 3090, RTX 4090) | AD-IQ4_NL | 19.89 | Fits with room. The best pick for most people |
| 32GB GPU (RTX 5090, R9700) | AD-Q5_0_XL | 28.81 | Fits. Near Q8_0 quality |
| 32GB GPU | Q8_0 | 34.16 | Does not fit on the GPU alone |
| 48GB+ GPU or 64GB Mac | Q8_0 | 34.16 | Fits. AtomicChat’s own guidance |
| 24GB Mac | AD-Q2_0_L | 16.04 | AtomicChat sizes this file for 24GB Macs. Not measured by us |
| 12GB GPU | AD-Q2_0_M | 14.20 | Too big for the GPU alone. Move some experts to system RAM |
Blackwell cards (RTX 50 series, RTX PRO 6000, DGX Spark) can use the AD-NVFP4 file. It is the same size as AD-IQ4_NL and stores the experts in native NVFP4.
For a 24GB card, a used RTX 3090 is the usual buy. Used units averaged about $1,563 on resaleprices.com when we read it on 30 October 2026. For the 16GB tier, a new RTX 5060 Ti 16GB runs the 2-bit file; check current listings for its price.
Does a Small Quant Hurt Accuracy?
Less than you would expect. Two independent tests agree on English.
llama.cpp pull request #31471 (one RTX PRO 6000, temperature 0):
| Test | Q8_0 | IQ4_NL | BF16 (model card) |
|---|---|---|---|
| LibriSpeech test-clean WER (2,620 utterances) | 1.03 | 1.02 | 1.09 |
| AMI IHM cpWER (16 meetings, 8.8 hours) | 9.91 | 10.38 | 10.29 |
AtomicChat’s test (2,200 utterances, 6.7 hours, change in error against BF16, community-reported):
- AD-IQ4_NL and AD-NVFP4 stay within noise on every set they tested.
- AD-Q2_0_M stays within noise on English, German, French and Spanish.
- AD-Q2_0_M adds +1.53 points of character error on FLEURS Chinese. That interval does not include zero.
So the 16GB file is fine for English and the major European languages. For Chinese, use a 24GB card and the 4-bit file.
Two Runtimes, Two Feature Sets
This is the choice most guides skip. The most-downloaded GGUFs were built for NVIDIA’s own runtime, not for upstream llama.cpp.
| Feature | NeMo-Speech.cpp (NVIDIA) | llama.cpp b12087+ |
|---|---|---|
| Transcription, 40 languages | Yes | Yes |
| Speaker labels | Yes | Yes |
| Word timestamps | Yes (--word-times) | No (“beyond the scope” of PR #31471) |
| Streaming | No (offline model) | No |
| OpenAI-style server | /v1/audio/transcriptions | llama-server, needs --reasoning off |
| Backends | CUDA, CPU, Metal, Vulkan | Validated on CPU and CUDA |
| Files | AtomicChat GGUFs | The PR converted its own with convert_hf_to_gguf.py |
AtomicChat’s model card names NeMo-Speech.cpp as the target runtime. We did not confirm that its files load in upstream llama.cpp. If one fails there, convert the NVIDIA checkpoint yourself, as the pull request did.
The --reasoning off trap. The chat template turns thinking on by default. Without the flag, llama-server returns only reasoning and no transcript. llama-mtmd-cli needs no flag.
Run It With NeMo-Speech.cpp
These are the build steps from AtomicChat’s model card. You need CMake 3.26+, SentencePiece 0.2.0+, and CUDA 12.8+ for the batched CUDA path.
git clone https://github.com/NVIDIA/NeMo-Speech.cpp && cd NeMo-Speech.cpp git submodule update --init --depth 1 llama.cpp third_party/cpp-httplib scripts/configure.sh cuda-server && cmake --build --preset cuda-server hf download AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF --local-dir models \ --include "*Encoder-F16.gguf" --include "*AD-IQ4_NL.gguf" build/cuda-server/bin/nemo-speech transcribe audio.wav \ --model models/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-AD-IQ4_NL.gguf --format json --word-times
Keep the encoder’s file name as it is. The decoder file names its encoder.
How Fast Is It?
We found no measured speed on a consumer GPU. Two numbers are published, and neither is a desktop figure:
- NVIDIA’s NIM on one RTX PRO 6000 Blackwell Server Edition transcribes up to 2,238 seconds of audio per second. That uses vLLM, NVFP4 and DFlash2 speculative decoding with many requests at once.
- The llama.cpp pull request reports that MTP drafting is 1.38x faster on a 47-minute meeting, at 80% draft acceptance.
Our estimate: with about 4.3B active parameters, a single 24GB card should transcribe much faster than real time. Test your own card with a known recording before you plan a workload around it.
Should You Use It Instead of Whisper?
Use it when you need speaker labels, long recordings, or many languages in one model. Its card reports a 15.80 mean cpWER on multi-speaker meetings and 4.71 mean WER on the Open ASR Leaderboard English sets.
Do not use it for live captions. It is an offline model. Also check the size: a small Whisper model fits a laptop GPU, and this model needs a 16GB card.
Sources
- nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B — model card: parameters, languages, limits, benchmarks, NIM throughput, OpenMDW-1.1. Read 2026-11-03.
- llama.cpp PR #31471 — merged 2026-11-03; released in b12087. WER table,
--reasoning off, no word timestamps. - AtomicChat GGUF repo — files, 15.6 GiB peak on 16GB, error table (community-reported).
- Jmica and sakasegawa GGUF repos — byte sizes from the HF tree API, 2026-11-03.
- NVIDIA/NeMo-Speech.cpp — Apache-2.0, release v0.3.0 (2026-10-30).
- Hacker News discussion, 2026-10-28, and the Hugging Face launch post.
See Also
- The Cheapest Rig That Runs Nemotron 3.5 Lightning — the language model this transcriber is built on
- Best Local LLM for 16GB VRAM — what else the 16GB tier runs
- Best Local LLM for 24GB VRAM — the tier where the 4-bit file fits with room
- Is a Used RTX 3090 Worth It? — the usual 24GB buy, priced
- NVFP4 vs MXFP4 — what the AD-NVFP4 file format means on Blackwell
Before you order parts, check the tested hardware list for current prices.