Running whisper.cpp on NixOS

Whisper is OpenAI's multilingual speech recognition model. It's open source and can be used to transcribe audio files into text. whisper.cpp is a high-performance C/C++ implementation of that model.

Setup

Assuming you are running NixOS already, creating a whisper.cpp container is simple. Here is the recipe:

Download one of the Whisper models in ggml format. (This format is needed to be able to load them in C/C++).

In the NixOS configuration file, define the whisper-server container which will run as a systemd service:

  # enable common container config in /etc/containers
  virtualisation.containers.enable = true;

  # enable podman or docker
  virtualisation.podman.enable = true;

  # define the container to run:
  virtualisation.oci-containers.containers = {
    whispercpp-server = {
      # official image of whisper.cpp
      image = "ghcr.io/ggml-org/whisper.cpp:main";
      autoStart = true;
      # expose the whisper-server port 
      ports = [ "8080:8080" ];
      # mount a models/ directory
      volumes = [ "/path/to/whisper/models:/models" ];
      # run the server passing the model downloaded from step 1
      # -t is the thread count
      cmd = [ "whisper-server --host 0.0.0.0 -m /models/ggml-tiny.en-q8_0.bin -t 4" ];
    };
  };

Switch to the new configuration with nixos-rebuild switch

Check the logs with journalctl -u podman-whispercpp-server.service. If everything worked correctly, we'd see something like:

podman-whispercpp-server-start[757611]: Trying to pull ghcr.io/ggml-org/whisper.cpp:main...
podman-whispercpp-server-start[757611]: Pulling image //ghcr.io/ggml-org/whisper.cpp:main inside systemd: setting pull timeout to 5m0s
podman-whispercpp-server-start[757611]: Getting image source signatures
podman-whispercpp-server-start[757611]: Copying blob sha256:12b1c5ee63096390f18d258f966580d4f4012cbe0f28b800e82cfc4e1fef115d
...
systemd[1]: Started podman-whispercpp-server.service.
podman-whispercpp-server-start[757611]: e4d48ec9817ae3742fb43443516cccad247e2986eaacdfd07795a15cdc8ae943
whispercpp-server[759423]: whisper_init_from_file_with_params_no_state: loading model from '/models/ggml-small-q8_0.bin'
whispercpp-server[759423]: whisper_init_with_params_no_state: use gpu    = 1
whispercpp-server[759423]: whisper_init_with_params_no_state: flash attn = 1
...

Transcribe audio files

The current version of whisper.cpp expects audio files to be 16-bit WAV files, so we need to transform them first:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

If you don't have ffmpeg locally, you can use whisper.cpp's image since it already has ffmpeg installed:

sudo podman run -it ghcr.io/ggml-org/whisper.cpp:main '[ffmpeg command above]'

P.S. We need sudo because our container's user is root.

Now we can visit the server running on http://127.0.0.1:8080/. Upload files from the UI or using curl:

curl 127.0.0.1:8080/inference \
    -H "Content-Type: multipart/form-data" \
     # the '@' reads the content of the file
    -F file="@<file-path>" \
    -F temperature="0.0" \
    -F temperature_inc="0.2" \
    -F no_speech_thold="0.6" \
    -F response_format="json"

=> {"text":" Test test 123.\n"}

Benchmark testing

We can test the performance of the server to see how fast transcription can be. Given the following test environment:

Model: ggml-tiny.en-q8_0.bin
CPU: Intel Core i7-8550U @ 1.80 GHz (no GPU)
Logical CPUs: 8
Inference thread count: 4 (the -t 4 option passed to whisper-server)
Duration of the audio sample: 11 seconds

I used autocannon to run a 10-second test with 10 concurrent connections.

npx autocannon -m 'POST'  -F '{ "file": { "type": "file", "path": "/path/to/samples.wav" } }' http://127.0.0.1:8080/inference

The results were the following:

Average latency: 3.8 sec
Average throughput: 1.6 req/sec
Total requests: 28
Timeouts: 2

So transcribing an 11-second audio file takes 3.8 seconds on average.

Running whisper.cpp on NixOS

Setup

Transcribe audio files

Benchmark testing

See also