Two-Node Multi-GPU Inference Cluster
A self-built two-node multi-GPU cluster serving local models with a privacy-first architecture, thermal-aware autonomy, and GPU-wedge diagnosis and recovery. The RTX 3090 node is down with a hardware fault until I repair it, so the RTX 5090 serves alone for now.
- 2 nodes
- multi-GPU cluster measured
Private infrastructure, not published.
Context
Serving local models reliably means living inside hard VRAM and thermal limits, and recovering gracefully when a card wedges. I built the cluster and the tooling to do exactly that.
What I built
The cluster is two nodes: a Windows workstation (RTX 5090) paired with a Linux agent machine (RTX 3090), serving local models over llama.cpp and Ollama. The 3090 node is down with a hardware fault until I repair it, so for now the 5090 serves alone. VRAM-bound tuning of KV-cache and context windows fits context lengths inside a hard memory ceiling without wedging the fragile 3090. Thermal-aware autonomy backs off before the abort line, and a GPU-wedge watchdog (structured nvidia-smi and dmesg analysis with Xid-fault detection and machine-consumable output) handles fast fault diagnosis and recovery. The network is privacy-first: SSH tunnels and scoped port routing, with the agent machine holding only read-only access to shared services.
What the VRAM ceiling forces
It is the substrate the rest of the stack runs on; the hard VRAM and thermal limits are what forced the tuning and recovery work above.
This is private lab infrastructure, not published as a repository; it is the hardware the public RAG and retrieval projects run on.