Edge of Context
Search

· Updated

LoRAX Serving Guide: Operating Many LoRA Adapters on Kubernetes

One base model and many LoRA adapters create an unusual serving problem. The base weights are shared, but every request may need a different set of adapter weights. A conventional one-deployment-per-variant design wastes GPU memory when most variants are idle.

LoRAX addresses that long tail. It loads adapters on demand, batches requests for different adapters, and moves adapter weights between GPU and CPU memory. The attractive headline is “thousands of fine-tuned models on one GPU.” The engineering question is narrower: does your active adapter set, arrival pattern, and latency target benefit from exchange scheduling enough to justify another serving runtime?

This guide answers that question, verifies the API locally, and then turns the repository’s starter Helm chart into an explicit production plan.

TL;DR. Choose LoRAX when many compatible LoRA adapters share one base model and traffic is sparse or long-tailed. Capacity depends on the active working set, not the catalog size. Pin the runtime, authenticate and allowlist adapter IDs, add durable artifact caching and probes, route for cache locality, and benchmark cold and warm paths separately.

The serving problem is the working set

LoRA freezes a base model and learns low-rank updates for selected weight matrices. The resulting adapter is usually much smaller than a full checkpoint, but its size still depends on rank, target modules, layer count, and dtype. Fixed claims such as “every adapter is 100 MB” are poor capacity inputs.

For serving, distinguish three quantities:

A catalog can contain thousands of adapters without fitting thousands in VRAM. What matters is how frequently the active set changes, how large those adapters are, and whether requests for different adapters can share useful batches.

LoRAX combines four mechanisms:

  1. The base model remains resident for all compatible adapters.
  2. A request names an adapter, which can be resolved from Hugging Face, Predibase, or a filesystem.
  3. Adapter exchange scheduling prefetches and offloads weights between GPU and CPU memory.
  4. Heterogeneous continuous batching groups requests targeting different adapters.

LoRAX request path from adapter resolution through generation

The project reports that heterogeneous batching keeps throughput and latency nearly constant as concurrent adapter count grows in its benchmarks. Treat that as a hypothesis for your workload. Prompt length, output length, rank, target modules, batch occupancy, cache churn, and GPU generation can change the result.

When LoRAX is a plausible fit

LoRAX deserves a benchmark when all of these are true:

Typical cases include tenant-specific assistants, many domain variants, and online experiments that share a base checkpoint.

It is a weaker fit when a few adapters dominate traffic, models do not share a base, hard latency targets cannot tolerate cold loads, or the platform cannot safely control which artifacts the server loads. A standard vLLM or TGI deployment with a fixed adapter set may be simpler in those cases.

Do not use Kubernetes merely because the catalog is large. First prove the runtime and adapter compatibility on one GPU.

Verify one base and two adapters locally

The LoRAX README recommends its prebuilt container. Use an immutable image digest in a real environment; main is shown here only because it is the repository’s documented quick-start tag.

mkdir -p data

docker run --rm --gpus all --shm-size 1g \
    -p 8080:80 \
    -v "$PWD/data:/data" \
    ghcr.io/predibase/lorax:main \
    --model-id mistralai/Mistral-7B-Instruct-v0.1

The documented minimum is Linux, Docker, an Ampere-or-newer NVIDIA GPU, and CUDA 11.8-compatible drivers. Model licenses and gated repositories may also require a Hugging Face token.

Start with the base model:

curl http://127.0.0.1:8080/generate \
    -H 'Content-Type: application/json' \
    -d '{
      "inputs": "[INST] Give one reason to measure cold-adapter latency. [/INST]",
      "parameters": {"max_new_tokens": 64}
    }'

Then send a compatible adapter:

curl http://127.0.0.1:8080/generate \
    -H 'Content-Type: application/json' \
    -d '{
      "inputs": "[INST] Solve: Natalia sold 48 clips in April and half as many in May. What is the total? [/INST]",
      "parameters": {
        "max_new_tokens": 64,
        "adapter_id": "vineetsharma/qlora-adapter-Mistral-7B-Instruct-v0.1-gsm8k"
      }
    }'

The first request may download and load the adapter; later requests can use cached artifacts and resident weights. Record both paths. A single warm request says little about long-tail behavior.

Use the current OpenAI client

LoRAX exposes an OpenAI-compatible chat endpoint. The model field identifies the adapter:

from openai import OpenAI


client = OpenAI(
    api_key="EMPTY",
    base_url="http://127.0.0.1:8080/v1",
)

response = client.chat.completions.create(
    model="alignment-handbook/zephyr-7b-dpo-lora",
    messages=[
        {"role": "user", "content": "Explain cache locality in two sentences."},
    ],
    max_tokens=100,
)

print(response.choices[0].message.content)

The server does not require an API key by default. That is convenient for localhost and unsafe as an Internet-facing configuration. Put authentication, tenant authorization, quotas, and adapter allowlisting in front of it.

Define a compatibility gate

Before an adapter enters the catalog, verify at least:

Reject incompatible artifacts during registration rather than on a user’s first request.

Understand residency before deploying

Adapter artifact storage and runtime residency in LoRAX

The base model consumes the dominant fixed share of GPU memory. Adapter weights, KV cache, batch workspace, and runtime kernels compete for the remainder. CPU RAM can hold offloaded adapters, while /data stores downloaded artifacts.

These are not interchangeable tiers. A disk or Hub artifact must be read and materialized before it becomes a CPU- or GPU-resident adapter. Measure the transitions separately:

PathWhat it includesMetric to record
GPU hitAdapter already residentqueue time and time to first token
CPU hitTransfer or rematerialization into GPUadapter-load delay and end-to-end latency
Artifact hitRead from local /data cacheread/load delay and cache bytes
Remote missDownload plus validation and loaddownload time, failures, and full cold latency

Capacity planning should replay the real adapter popularity distribution. Uniform random adapter IDs create a different cache problem from a Zipf-like tenant workload.

Deploy the repository chart carefully

The repository contains charts/lorax, so the repeatable starting point is a pinned repository revision and a local chart:

git clone https://github.com/predibase/lorax.git
cd lorax
git checkout <reviewed-commit-or-release>

helm dependency update charts/lorax
helm template lorax charts/lorax -f values.production.yaml
helm upgrade --install lorax charts/lorax \
    --namespace inference \
    --create-namespace \
    -f values.production.yaml

As reviewed on July 15, 2026, the chart defaults deserve attention:

That makes the chart useful scaffolding, not a production policy.

Start from the chart’s actual value structure

The chart nests runtime configuration under deployment, and launcher arguments are a list of name/value pairs. A minimal overlay looks like this:

deployment:
    replicas: 1

    image:
        repository: ghcr.io/predibase/lorax
        tag: "<tested-tag>" # Prefer an immutable digest in rendered manifests.

    args:
        - name: "--model-id"
          value: "mistralai/Mistral-7B-Instruct-v0.1"
        - name: "--max-input-length"
          value: "2048"
        - name: "--max-total-tokens"
          value: "3072"
        - name: "--max-batch-total-tokens"
          value: "8192"
        - name: "--max-batch-prefill-tokens"
          value: "4096"

    resources:
        requests:
            nvidia.com/gpu: "1"
        limits:
            nvidia.com/gpu: "1"

    readinessProbe:
        httpGet:
            path: /health
            port: http
        periodSeconds: 5
        failureThreshold: 600

service:
    serviceType: ClusterIP
    port: 80

The token limits above are example starting values, not sizing recommendations. Derive them from representative prompts, expected concurrency, GPU memory, and load tests.

Patch the gaps explicitly

The current chart template hard-codes emptyDir volumes and plain environment values. A hardened deployment therefore needs a reviewed chart fork, post-render patch, or higher-level manifest layer to add:

Do not place a Hub token directly in a committed values file. Do not assume two replicas create useful high availability until both can load the base model and the router can avoid sending all cold adapters to both pods.

Route for cache locality

Round-robin balancing can turn every replica into a cold cache. A useful router hashes or otherwise affinitizes an allowed adapter ID to a replica, while retaining a failover path when that replica is unavailable.

The routing key must come from authenticated application state, not an arbitrary public URL parameter. Otherwise a caller can force remote downloads, churn caches, or probe private adapter names.

LoRAX or vLLM?

Current vLLM can serve LoRA modules declared at startup and can also load them dynamically through endpoints or resolver plugins. Its own documentation warns that runtime adapter updating has security risks and should not be used in production outside an isolated, trusted environment.

The useful comparison is operational rather than numerical:

QuestionLoRAXvLLM
How are long-tail adapters discovered?Adapter ID can resolve Hugging Face, Predibase, or filesystem artifacts on requestStatic modules, dynamic management endpoints, or resolver plugins
How is residency managed?Explicit adapter exchange scheduling between GPU and CPUConfigured active and CPU LoRA limits plus resolver behavior
What is the request interface?TGI-style /generate, Python client, and OpenAI-compatible chatOpenAI-compatible serving and native Python APIs
What should decide?Cold/warm latency, cache churn, heterogeneous batch throughput, and operational fitThe same workload replay and operational criteria

Avoid rules such as “LoRAX for 1,000 adapters, vLLM for ten.” Catalog size alone does not determine performance. Benchmark both with the same base, adapters, prompts, ranks, arrival trace, and hardware.

A production acceptance test

Before expanding the catalog, run a replay that includes:

  1. A fixed hot set to establish warm throughput and latency.
  2. A long-tail distribution to measure CPU and artifact-cache hits.
  3. A burst of previously unseen but allowlisted adapters.
  4. Pod replacement to measure base and adapter recovery.
  5. One unavailable or corrupt adapter to verify isolation and fallback.
  6. Concurrent tenants to verify authentication, quotas, and metric labels.

Track request rate, queue time, time to first token, inter-token latency, total latency, adapter load time, cache-hit class, GPU and CPU memory, download bytes, and failures by reason. Keep adapter IDs out of unbounded metric labels; map them to controlled dimensions or sampled traces.

Define acceptance thresholds before the test. Examples include a maximum cold-path error rate, a warm P99 target, a cache-hit target for the observed popularity distribution, and a recovery-time objective after pod loss.

Conclusion

LoRAX turns many compatible fine-tunes from a fleet of base-model copies into an adapter placement problem. That can be a strong design for long-tail workloads, but it does not make cost or latency flat by definition. The active working set, exchange path, batch mix, and storage layer still set the result.

Prove those mechanics on one GPU first. Then take Kubernetes seriously: pin artifacts, preserve the cache, protect credentials, authorize adapters, route for locality, and measure every residency path. If LoRAX beats a current vLLM configuration under the same replay, the deployment decision has evidence behind it.

References