AI

NUMALLAMA

NUMALLAMA is a lightweight C inference engine designed to run modern Large Language Models directly on CPUs — without GPUs, CUDA, or heavyweight runtime dependencies.

Its primary goal is to make large multi-socket ARM servers effective LLM inference platforms by taking full advantage of their CPU cores and memory bandwidth.

Source code: github.com/miloserdev/numallama

Why I created NUMALLAMA

Modern LLM inference is often associated with GPUs. However, many servers already have hundreds of CPU cores, substantial RAM capacity, and considerable memory bandwidth.

The problem is that multi-socket systems are not uniform memory architectures. Each CPU socket has its own local memory, and accessing memory attached to another socket introduces additional latency and consumes inter-socket bandwidth.

During testing on a dual-socket Ampere Altra Max server, I found that simply increasing the number of CPU cores did not guarantee better inference performance. In fact, llama.cpp performed better on one socket than on both sockets for several tested configurations.

I created NUMALLAMA to address this problem.

Instead of treating the entire server as one large CPU, NUMALLAMA explicitly organizes inference around NUMA topology, keeping computation and memory access close to the socket that performs the work.

Key advantages

NUMA-aware execution

  • Distributes model weights and KV cache across NUMA nodes.
  • Uses per-socket work queues and buffers.
  • Keeps CPU workers close to the memory they access.
  • Reduces expensive cross-socket memory traffic.

ARM-optimized computation

  • NEON vectorized kernels.
  • ARM dot-product instructions (SDOT).
  • Repacked weights for efficient matrix operations.
  • Batched prompt processing and multi-token decoding.

Efficient memory usage

  • Q8 and asymmetric Q4 quantization.
  • Importance-matrix-based quantization.
  • Support for large models and Mixture-of-Experts architectures.

Inference features

  • Persistent thread pool and tree-based synchronization barriers.
  • Exact speculative decoding using prompt lookup or a draft model.
  • OpenAI-compatible HTTP API.
  • Streaming responses and tool calling.
  • Prompt-prefix cache.
  • CLI chat with sandboxed tool execution.

Supported models

NUMALLAMA supports several modern model families:

  • Llama 3.x
  • Qwen 2.5
  • Qwen 3
  • Qwen 3 MoE
  • Qwen 3.5 / Qwen 3.8 hybrid models

Supported architectures and quantization formats depend on the model implementation.

Benchmark results

The primary test platform:

ComponentConfiguration
CPU2 × Ampere Altra Max M128-30
ArchitectureArm Neoverse N1
CPU cores256
Clock3.0 GHz
RAM1 TB
NUMA nodes2
Memory bandwidth~180 GB/s per socket

Decode performance

ModelNUMALLAMA (2 sockets)llama.cpp (best, 1 socket)
Llama 3.2 3B Q876 tok/s36 tok/s
Llama 3.2 3B Q499 tok/s64 tok/s
Llama 3.1 8B Q839 tok/s19 tok/s
Llama 3.1 8B Q459 tok/s29 tok/s

Larger models

ModelQuantizationDecodePrefill
Qwen3-30B-A3B (MoE)Q491 tok/s~500 tok/s
Qwen3.8-27BQ417 tok/s172 tok/s

These results are measurements from one specific server configuration, not universal performance guarantees. Performance depends on hardware, model, quantization, and runtime configuration.

Full benchmark methodology and results:

NUMALLAMA Benchmarks

Building from source

NUMALLAMA is primarily written in C and uses a compact inference runtime.

On Linux, clone the repository and build it:

text
git clone https://github.com/miloserdev/numallama.git
cd numallama

make

The optimized kernels target ARM64 systems with NEON and dot-product instruction support, including Ampere Altra and other Neoverse-based processors.

x86-64 builds use portable C fallbacks and are not currently optimized.

Preparing a model

The following example prepares Qwen3-30B-A3B-Instruct-2507 using the included export scripts.

Export the model weights:

text
python3 scripts/export_hf_q8.py \
    hf/Qwen3-30B-A3B-Instruct-2507 \
    qwen3_30b_q80.bin

Export the tokenizer:

text
python3 scripts/export_tokenizer.py \
    hf/Qwen3-30B-A3B-Instruct-2507 \
    qwen3_tokenizer.bin

The model directory should contain the Hugging Face model files required by the export scripts.

Running the inference server

Start the server:

text
./runq qwen3_30b_q80.bin \
    -z qwen3_tokenizer.bin \
    -m server \
    -P 8080

The server exposes an OpenAI-compatible API:

text
http://127.0.0.1:8080/v1

You can connect existing applications and tools that support the OpenAI API format.

For example, using Python:

text
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="qwen3",
    messages=[
        {"role": "user", "content": "Explain NUMA architecture."}
    ]
)

print(response.choices[0].message.content)

The model value should match the model identifier exposed by the running server.

Limitations

  • Linux only.
  • Optimized kernels currently target ARM64 with NEON and dot-product instructions.
  • No GPU acceleration or GPU offloading.
  • One active request at a time; additional requests wait in a queue.
  • No multi-user continuous batching.
  • Text-only inference.
  • Model support is limited to implemented architectures.

Project

NUMALLAMA is an open-source project focused on CPU inference, ARM optimization, and NUMA-aware execution.

The project explores an alternative approach to LLM infrastructure: using the CPU resources already available in large servers instead of requiring dedicated GPU hardware for every inference workload.