VERIFICATIONChecked against current official documentation on 2026.08.04; hardware-specific performance is not generalized.

30-SECOND SUMMARY

What to take away

  • Choose releases, Docker, or source build—not all three.
  • Check the model card and license.
  • Make the CLI work before starting a server.
RUNTIME 01

The shortest GGUF path

Make CLI work before the server.

  1. 01
    INSTALL

    Official package

  2. 02
    GGUF

    Model and license

  3. 03
    CLI

    Single prompt

  4. 04
    SERVER

    Local API

Use it this way Change one backend or quantization at a time.
SECTION 01

Choose an install path

The official repository provides prebuilt releases, Docker instructions, and source builds. Match CUDA, Metal, Vulkan, or CPU support to the machine.

SECTION 02

Select a GGUF

GGUF stores tensors and metadata. Start near Q4 and confirm the chat template, publisher, and license.

SECTION 03

Run the CLI

The current quick start can download a Hugging Face model directly.

llama-cli -hf ggml-org/Qwen3.5-0.8B-GGUF
SECTION 04

Start locally

Use the compatible server on localhost first. External access needs a separate authentication layer.

llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF
FAQ

Frequently asked questions

Is GGUF a model name?

No. It is a file format for tensors and metadata.

Must I compile from source?

No. Official binaries and Docker are alternatives.

Can it run without a GPU?

Yes, but speed depends heavily on model and CPU.

Primary sources

Check the original documentation for version-specific details.

llama.cpp repository Hugging Face GGUF

READ NEXT

Install Ollama and Run Your First Local ModelLM Studio Setup: Run Your First Local LLM