Projects

I build AI that runs on the hardware people already own, and the networks that keep it safe.

A smartphone standing on a dark surface, red light leaking from its edges and spreading across the floor in circuit-like lines.
What it is
An all-in-one AI app that runs entirely on your phone, on Android and iPhone.
Hardest problem
Making dozens of models run fast and correctly across phone chips whose GPU drivers disagree on basic math.
Measured result
10,000+ installs in 169 countries and 3,000+ active users. One app, every modality, fully on-device: chat, photo, audio and video understanding, document search, image generation, speech, voice cloning and tool-calling agents. A 35-billion-parameter model runs on a phone at 9 to 12 tokens per second, and the phone’s NPU reads long prompts about 4x faster than its CPU.

Built with

  • Kotlin,
  • C++,
  • Swift,
  • Jetpack Compose,
  • SwiftUI,
  • llama.cpp,
  • MNN,
  • ggml,
  • Vulkan,
  • OpenCL,
  • SPIR-V,
  • Qualcomm QNN,
  • Core ML,
  • MLX,
  • TFLite/XNNPACK,
  • sd.cpp,
  • Kokoro,
  • Sherpa-ONNX,
  • WhisperKit,
  • Room,
  • Hilt
Full technical detail15 items

TokForge is an on-device AI app for Android (Google Play, Android 8 and up) and iOS/iPadOS (public TestFlight beta), with 10,000+ installs in 169 countries and 3,000+ active users. It runs LLM chat, vision, audio and video understanding, speech (Kokoro TTS, ZipVoice voice cloning, speech input), image generation and document RAG entirely on the phone: no account, no telemetry by default (the speed leaderboard is opt-in), and no cloud processing unless you connect your own OpenAI-compatible server. Three engines sit behind one Kotlin/C++ JNI runtime: llama.cpp (GGUF on CPU, Vulkan and OpenCL), a maintained fork of Alibaba MNN (CPU, OpenCL, Vulkan, QNN), and an OpenAI-compatible remote client. A per-SoC routing table (Universality over Stability over Performance) sends each model to the fastest proven backend and gives unknown silicon the slowest proven path. 9 UI languages and 19 public releases since 1.0 in August 2026, including four production releases in 16 days with crash and ANR rates inside Google Play’s thresholds.

What I built

  • LLMs on the phone’s NPU: on Snapdragon phones the Hexagon NPU reads the prompt while the CPU writes the reply, so long prompts are processed about 4x faster than on the CPU alone. Qwen3.5 4B and 9B ship as opt-in NPU packs, each used only after a check on the phone itself.
  • Desktop-class models on phones: Qwen3.6 35B-A3B (35B) at 9 to 12 tokens per second on a 24 GB RedMagic 11 Pro, and Qwen3-30B-A3B (30B) at 14.4 tokens per second on an iPhone 17 Pro Max in the public TestFlight beta. Bonsai 27B (ternary) reached 8.5 tokens per second on Vulkan on a Snapdragon 8 Elite Gen 5 phone in September 2026.
  • One model for photos, audio and video: LFM3 Omni understands text, photos, voice and audio, and video on the phone, with no helper model and nothing sent to a server.
  • Vulkan GPU inference on Adreno, Mali and Xclipse: root-caused vendor driver bugs with ggml test-backend-ops one operation at a time (Adreno 750 q6_K mat-vec, Adreno 840 quantized KV below head size 256, a shader-compiler crash on wide batches) and encoded per-chip rules, a pipeline-cache integrity check and a phase-tagged crash guard. On a Snapdragon 8 Elite Gen 5 phone, GPU decode ran 1.3x to 2.4x faster than the leading competitor’s GPU path on the same models.
  • CPU and GPU on one model: the GPU reads the prompt and the CPU writes the reply, with a GPU self-test and routes measured per phone and model. Multi-token prediction for Qwen3.5 (0.8B to 27B) and draft-model packs switch on only where the phone measures a gain, and multi-token prediction works with the NPU.
  • MoE expert streaming for llama.cpp: a native streamer reads only routed experts from storage with next-layer prefetch, 4 I/O lanes and an LRU cache sized from MemAvailable, so Qwen3-30B-A3B (30B) runs on a 16 GB OnePlus Ace 5.
  • Stable Diffusion 1.5 on the Hexagon NPU through Qualcomm QNN context binaries, non-root: 512x512 in 6 to 10 s. GPU image generation on Mali and Xclipse phones after an on-phone image check. Also sd.cpp (SD-Turbo, SD1.5, SDXL, Flux) and a Sana image edit at 25 s per 512 px frame on a Galaxy S24+.
  • On-device RAG: PDF, DOCX, EPUB, Markdown and CSV ingestion, INT8 BGE-small embeddings on TFLite/XNNPACK (1.5x to 2.4x faster than the MNN encoder on every fleet SoC), hybrid FTS4 plus vector retrieval with MMR, memory facts, reflection and a knowledge graph, and an embedder swap with staged re-embedding.
  • KV cache work: KV reuse across chat turns in MNN so follow-up replies start almost instantly, later refined into token-exact replay that prefills only the new part of the conversation, and a TurboQuant 4-bit KV cache in the MNN OpenCL path (3.5x less KV memory at 8k context, 4x at 32k).
  • Agents and tools: tool-calling agents with live run screens and presets, document search over chat attachments, and an opt-in sandboxed terminal where the model proposes a command and the user approves each one.
  • AutoForge: an in-app micro-benchmark sweep over backend, thread and decode settings per SoC and model, auto-applied on load and feeding an opt-in public leaderboard on a Cloudflare Worker with D1; FleetPerfOptimizer learns per-tuple defaults from clean telemetry.
  • iOS port in Swift and SwiftUI with llama.cpp, MNN, MLX and a Core ML/ANE LLM engine, sd.cpp and Core ML Stable Diffusion, Sherpa-ONNX and WhisperKit, with an mmap-aware memory gate for large models on a 12 GB iPhone.
  • Robustness: a stall watchdog (stop, native abort, recover), persistent per-model OpenCL and Vulkan crash guards with automatic downgrade to CPU, a single RAM-admission rule for every load surface, and model files that are damaged or hostile are refused with a reason.
  • Hardware coverage: every release is tested on 12 Android devices (SM8850, SM8650, SM8250, SM7635, MT6991, MT6989, MT6765, Exynos 2400 and 1380, Tensor G4 and G5) and 4 Apple devices, with crash and ANR monitoring through the Google Play and App Store Connect APIs.
  • 106 public model repos on Hugging Face (MNN conversions, QNN NPU context binaries, image-generation GGUFs, draft packs), with 180,000+ downloads.
Two open black rack servers side by side, one with two graphics cards and one with a row of eight silver fanless cards, joined by a single cable glowing red.

Two servers, ten GPUs, one model

Server 1, stage 1

2 x RTX PRO 6000 Blackwell

96 GB each

Server 2, stages 2 to 5

8 x NVIDIA CMP 170HX

Supermicro AS-4124GS-TNR, 2 x AMD EPYC 7302, 64 GB of GPU memory

The full GLM-5.3 (753B) runs across both servers as one pipeline, at up to 104 tokens per second.
What it is
A ten-GPU AI server I built: eight NVIDIA CMP 170HX cards in a Supermicro 4U chassis, joined to two RTX PRO 6000 Blackwell cards in a second server over 100 GbE RDMA.
Hardest problem
Making ten GPUs in two servers behave as one machine, so a model far larger than any single card answers at interactive speed.
Measured result
Mistral Large 4, a 1-trillion-parameter model, on the eight cards, and the full GLM-5.3 (753B) as one ten-GPU pipeline at up to 104 tokens per second. The recipe is open source, with my fixes offered upstream.

Built with

  • Supermicro AS-4124GS-TNR,
  • AMD EPYC 7302,
  • NVIDIA CMP 170HX,
  • RTX PRO 6000 Blackwell,
  • 100 GbE RDMA,
  • vLLM,
  • llama.cpp,
  • MTP,
  • PipeSpec,
  • NCCL,
  • VMware ESXi,
  • Linux
Full technical detail9 items

An AI server I designed, built and run myself. A Supermicro 4U chassis (AS-4124GS-TNR, two AMD EPYC 7302 CPUs) holds eight NVIDIA CMP 170HX cards with 64 GB of GPU memory unlocked across them. A 100 GbE RDMA link joins it to a second server with two RTX PRO 6000 Blackwell cards (96 GB each), so ten GPUs in two servers work as one machine. The eight cards alone have run Mistral Large 4, a 1-trillion-parameter model. Together the ten GPUs serve the full GLM-5.3 (753B) as a single pipeline at up to 104 tokens per second. The recipe is open source, and my fixes go upstream as pull requests.

What I built

  • The chassis: a Supermicro AS-4124GS-TNR 4U server with two AMD EPYC 7302 CPUs and eight NVIDIA CMP 170HX cards attached directly, four per CPU, with 64 GB of GPU memory unlocked across the eight.
  • 1 trillion parameters on eight cards: Mistral Large 4 (1T total, 52B active) at 3-bit on llama.cpp across all eight CMP cards, at up to 18.7 tokens per second.
  • Ten-GPU pipeline: the full GLM-5.3 (753B total, mixture of experts) runs as one vLLM pipeline across two servers (TP2 x PP5 over 100 GbE RDMA). The Blackwell pair is the first stage and four CMP pairs are the other four.
  • Speed: 95 to 104 tokens per second on one stream and 240 aggregate at 8 streams with MTP speculative decoding and PipeSpec, 3,400 to 4,100 tokens per second prefill, a 524,288-token context and an 817,024-token KV cache.
  • What the second server adds: against the eight CMP cards alone, the Blackwell stage brings about 10% more single-stream speed, 13 to 28% more 8-stream throughput, about 50% faster prefill and 43% more KV cache. The model reloads in 6 to 7 minutes from local NVMe copies.
  • Engine fixes, offered upstream as pull requests: open-source overlays that repair the pipeline hand-off between tensor-parallel stages (restoring output quality), make structured output (JSON schema and tool calls) work with speculative decoding, add probabilistic drafts for sampled requests (+8.8% draft acceptance), port vLLM’s per-request draft retention (each structured-output stream 1.75 to 1.85x faster at 8 streams) and stop empty JSON replies in reasoning mode.
  • Open source: the two-server recipe is public on GitHub, alongside my fork of the community CMP 170HX recipe that carries those pull requests, and my notes on running the cards in VMware ESXi 8 virtual machines with PCI passthrough.
  • The lab behind cmp.wiki: this server supplies measured results to the community wiki I run for these cards, including the two-server GLM-5.3 recipe.
  • In daily use: the full GLM-5.3 on this server is the production model behind Folio, and DeepSeek-V4.1-Flash also runs on the eight CMP cards.
A fanless silver graphics card on a dark workbench beside the same card with its cover removed, a blank notebook and tools, joined by thin red lines like a diagram.
What it is
The community reference for running LLMs on NVIDIA CMP 170HX cards: unlocks, inference recipes, platforms, cooling and troubleshooting, with every fact credited to the person who found it.
Hardest problem
Turning thousands of chat messages and scattered repositories into a reference people can trust, where every claim carries its source and a status.
Measured result
180+ articles, with data tables for 46 recipes, 24 platform reports and 63 confirmations, open as daily JSON snapshots and through a public MCP server for AI agents.

Built with

  • WordPress,
  • MediaWiki,
  • Cargo,
  • Page Forms,
  • Kanboard,
  • Cloudflare Workers,
  • MCP,
  • Discord OAuth,
  • Cloudflare Turnstile
Full technical detail7 items

cmp.wiki is the community reference for the NVIDIA CMP 170HX, the 8 GB and the 10 GB card side by side: unlocks, inference recipes, platforms, cooling and power, troubleshooting and test results. I built it and I run it. A WordPress front door carries the news and the tools, a MediaWiki holds 180+ articles with structured data tables, and a task board lists tests anyone can take. Every fact is credited to the person who found it, and the text is licensed CC BY-SA 4.0. My CMP 170HX AI server is the lab behind several of its measured results.

What I built

  • Three parts: a WordPress front door, a MediaWiki for the articles and a Kanboard task board, with Discord sign-in for contributors and open reading for everyone.
  • Structured data: recipes, platform reports and confirmations are rows in data tables (46 recipes, 24 platform reports and 63 confirmations on November 10, 2026), with a benchmark table that gives each measured speed its source.
  • A status on every claim: Proven, Reported, Theory, Contested or Warning. A recipe moves to Proven once a second person reproduces it.
  • Tools on the front door: a setup matrix (pick your cards and links, see which recipes run and how fast), a memory calculator that shows every step of the arithmetic, a card check that reads pasted lspci, nvidia-smi and dmesg output in the browser, and a question box that answers from the wiki with numbered sources.
  • Open data: the tables are published as daily JSON snapshots on the site and on GitHub.
  • Built for AI agents too: a public, read-only MCP server running as a Cloudflare Worker, an llms.txt file and plain-text views of the articles.
  • Task board: testing and research tasks that anyone can sign in and take.
An old ledger lying open on a dark desk, its pages covered in red strokes, with a faint web of red light points rising above it.
What it is
An AI game master for long-form interactive fiction, phone-first and served from my own GPUs.
Hardest problem
Remembering everything across hundreds of thousands of words, long after it has left the model’s context window.
Measured result
A story ledger that tracks quests, inventory, NPCs and their secrets, events and backstory, recalled at 86% on a 152-question continuity benchmark: more than double the 40% from loading the whole story into context.

Built with

  • Python 3.12,
  • FastAPI,
  • PostgreSQL 16,
  • pgvector,
  • Preact,
  • TypeScript,
  • vLLM,
  • SGLang,
  • MTP,
  • EAGLE,
  • Hugging Face Transformers,
  • PEFT,
  • FSDP2,
  • Qwen3-Embedding,
  • Whisper,
  • XTTS-v2
Full technical detail7 items

Folio is an AI game master for long-form interactive fiction, built phone-first. Its strength is memory: a chronicler keeps a living story ledger (quests with objectives and status, inventory with where each item came from, every NPC with their attitude, debts and secrets, world decisions and consequences, a dated timeline, and the full character sheet), and retrieval pulls the right past scenes back in, so the game master stays consistent across hundreds of thousands of words. It runs on my own GPUs: FastAPI on Python 3.12 with streaming, Postgres 16 with pgvector and per-account row-level security, and a Preact PWA. Pre-alpha, with four invited testers.

What I built

  • The story ledger: a chronicler that reads the story in 40-message chunks and merges changes into structured state (quests, inventory, NPC relationships with debts and secrets, world decisions, a timeline, the character sheet) instead of rewriting it, so facts survive long after they leave the context window, with fast retraction when a scene is deleted.
  • Retrieval: scene-level chunks, Qwen3-Embedding-4B vectors unioned with BM25 and fused by reciprocal rank fusion, and an mxbai cross-encoder reranker. Continuity recall on a 152-question depth-stratified benchmark: 85.5%, against 40.2% for the whole 934k-token story loaded into context and 63.8% for the same system driven by Claude Sonnet.
  • Serving: the full GLM-5.3 (753B) from my own ten-GPU pipeline, with MTP speculative decoding, at up to 104 tokens per second and a 524,288-token context.
  • Consistency checkers that run on every draft (name drift, retired names, contradicted facts) with zero false positives on an 889-turn backtest.
  • Fine-tuning: a custom trainer on Hugging Face Transformers, PEFT and FSDP2 that trains GLM-5.3-Flash (320B) on two 96 GB GPUs by streaming frozen expert weights from host memory one layer at a time; a measured style-drift metric fell 30% on the hardest eval set.
  • Scene and character illustrations with automated moderation, narrated playback with an XTTS-v2 voice, and Whisper speech input.
  • Nightly backups with a rehearsed restore, hash-verified deploys, and Prometheus and Grafana monitoring.
A rugged phone in a rubber case on a creased topographic map over rock at dusk, a brass compass beside it and mountains behind.
What it is
An offline survival and medical guide for Android with a 1,400+ book library on the phone, offline Wikipedia, a medical encyclopedia and topo maps with GPS.
Hardest problem
Giving trustworthy answers with no network, and refusing to guess when the library does not cover a question.
Measured result
Ask in plain words or snap a photo of a plant, an injury or a map and ask about it: a 4-billion-parameter vision model answers on the phone in seconds and cites the page it used.

Built with

  • Kotlin,
  • C++,
  • Python,
  • MNN,
  • SQLite vec0 and FTS,
  • BGE-small,
  • Kiwix,
  • pdf.js,
  • Ed25519,
  • llama.cpp (desktop)
Full technical detail7 items

SurvivalOps puts a survival and medical reference library and an AI that can read it on an Android phone, with no network at all. The library holds 1,400+ books, field manuals and medical references (176,000 searchable passages), plus offline Wikipedia, a 463,000-entry medical encyclopedia and USGS topographic maps with GPS. Qwen3.5 4B with its vision tower runs on the phone through MNN, answers questions about text and photos, and cites its sources; a desktop twin runs Qwen3.8-27B. Retrieval is BGE-small in SQLite vec0 plus keyword search, with a safety floor that makes the app say so instead of guessing when the library does not cover a topic. In private testing on real phones; the public release waits on an independent safety review. Core and desktop are prepared for an Apache-2.0 release.

What I built

  • Three answer paths: prepared cards and a verified facts table that bypass retrieval and the model and answer common questions in 0.3 to 1.5 s, full RAG with the 4B, or an honest refusal.
  • KV cache reuse for follow-ups, so follow-up answers start in under a second.
  • Safety-first quality bar: every answer path is checked against sealed medical and survival test sets before release.
  • Safety layer: a citation guard, a medication-dose guard, a high-recall urgent-signal sensor kept separate from card routing, severity-ranked card priority, and a shared text normalizer for panicked typing.
  • Content packs: an Ed25519-signed catalog and manifests with a per-file SHA-256 chain (a 2.63 GB medical pack with 66 documents and 1,151 verified hashes installed; an unsigned manifest was rejected). Tiers: CORE 10.4, FIELD 19, DEEP 32.5 GiB.
  • Encyclopedias and maps: a 463,451-entry medical ZIM opens an article in 90 ms on the phone; USGS topo sheets render through pdf.js with the UTM zone parsed per sheet, first paint in 1.4 to 1.8 s.
  • Engine port: the app runs on the TokForge inference engine through a compact 36-method JNI bridge.
Three black server racks in a dark data center aisle, rows of red status lights behind perforated doors and cables in a tray above.
What it is
The hosting platform behind 20+ websites, our clients’ and our own, on hardware we own in Michigan, Texas and Ohio.
Hardest problem
Giving small-business WordPress sites a multi-site database cluster of the kind usually reserved for enterprise systems, where any one site can fail without stopping writes.
Measured result
A database cluster with five voting members across four sites. In a real 55-minute data center outage on October 29, 2026 it elected a new primary in about a minute, with no visitor-facing outage on load-balanced sites. A planned drill moved 20 domains, with writes tested, in 101 seconds.

Built with

  • Linux,
  • Apache,
  • PHP-FPM,
  • MySQL 8 Group Replication,
  • Redis,
  • VMware ESXi,
  • VyOS,
  • WireGuard,
  • Tailscale,
  • Cloudflare,
  • Cloudflare Workers,
  • ModSecurity,
  • OWASP CRS,
  • fail2ban,
  • Syncthing,
  • Ansible,
  • Veeam,
  • MinIO,
  • TrueNAS SCALE,
  • VictoriaMetrics,
  • Prometheus,
  • Loki,
  • Grafana,
  • FastAPI,
  • vLLM
Full technical detail11 items

The hosting platform behind 20+ websites, our clients’ and our own, on hardware we own in Michigan, Texas and Ohio. The databases run as one MySQL Group Replication cluster with five voting members across four sites: one primary takes writes, a write commits only when a majority agrees, and if the primary’s site goes dark the cluster elects a new one on its own. Cloudflare load balancing follows the primary, file sync keeps every site’s copy current, and backups land on immutable, self-hosted object storage. GuardAI, our AI monitoring system, watches every site and server around the clock from a Grafana NOC dashboard.

What I built

  • Database cluster: MySQL Group Replication in single-primary mode with five voting members across four sites, so any one site can fail without stopping writes. Cloudflare health checks pass only on the current primary, so traffic follows it.
  • A real outage, handled: on October 29, 2026 a data center went dark for 55 minutes. The cluster elected a new primary in about a minute, and load-balanced sites had no visitor-facing outage.
  • A planned drill: 20 domains moved to another site, with writes, uploads and forms tested, in 101 seconds.
  • A manifest-driven web stack deployed with Ansible, with Syncthing file sync between sites and a DNS-steering Cloudflare Worker that moves plain-DNS sites with the primary.
  • GuardAI: our AI monitoring system reads every server error (pattern match first, then all-MiniLM-L6-v2 semantic matching, then a local vLLM model for anything new), posts an explained incident card on the Grafana NOC dashboard, and fixes common issues on its own within rate limits, with a backup before each change and a rollback path.
  • 24/7 NOC monitoring: VictoriaMetrics, Prometheus, Loki and Grafana, a clustered Alertmanager, a twice-daily fleet canary, a lead-flow watchdog that tests every contact form, and paging over WhatsApp.
  • A zero-downtime move of the whole fleet (about 13 GB, 355,000 files and 20 databases) to a new data center with a two-step Cloudflare edge flip, with every site online through the cutover.
  • A private network across five sites with a full WireGuard mesh on VyOS routers; a direct Ohio-to-Michigan tunnel cut latency from about 60 to 21 milliseconds.
  • Immutable backups on five storage servers (about 255 TB) in three locations.
  • Security: per-site PHP isolation, Cloudflare-only origin access, a web application firewall with the OWASP Core Rule Set and a fail2ban rule keyed to its anomaly score that bans attackers automatically (1,600+ attacks logged), a live threat map, and a security-posture review across all client domains.
  • Growth for clients: SEO and paid search across the client fleet, with audits across 13 Search Console properties, recovered search traffic on rebuilt sites, review alerts by text message, and Google Ads with server-side conversion tracking (one contractor client’s site leads grew 79%, from 42 to 75 a month).
A card payment terminal and a receipt printer with a curl of blank paper on a dark stone counter in a dim restaurant.
What it is
A restaurant point-of-sale and ordering platform with smart card terminals, a kitchen display, receipt printers and staff badges, built by Elevated for its restaurant clients.
Hardest problem
Never losing an order or a payment when a device, the network or the internet goes down.
Measured result
Every sale is an event on an append-only log, so the system can always rebuild its exact state, and the floor keeps taking orders through an internet outage.

Built with

  • TypeScript,
  • Node,
  • Fastify,
  • WebSockets,
  • PostgreSQL 16,
  • event sourcing,
  • Helcim,
  • Stripe Terminal,
  • ESC/POS,
  • Cloudflare Tunnel
Full technical detail5 items

Elevated POS is a point-of-sale and ordering platform for restaurants, built by Elevated and offered to its restaurant clients. It is edge-first: it runs on an appliance inside the restaurant, so the floor keeps taking orders when the internet drops. Every action is stored as an event on an append-only log in Postgres 16 with a gap-free sequence, and the screens are live projections of that log, pushed over WebSockets to the front of house and the kitchen display. About 50 event types and 240 API routes.

What I built

  • Event-sourced core: an append-only event log with gap-free sequencing and a projection cache that folds events into the live state, so any screen can be rebuilt exactly and nothing is lost if a device drops.
  • Payments on Helcim smart terminals (asynchronous charge, poll and reconcile, including charges captured but not yet settled) and Stripe Terminal, with a posture check that refuses to start on an unsafe live configuration.
  • Real-time front of house and kitchen display over WebSockets, with order routing to the line.
  • Receipt printers that find themselves again when the network moves them, cash-drawer control, NFC badge sign-in for staff, and a read-only bridge to a restaurant’s reservation system.
  • Payment webhooks through a Cloudflare tunnel locked to a single path.
A black workstation case with a tinted glass side showing two large graphics cards outlined in red light.
What it is
The private GPU platform around the AI server: one router in front of every model, plus fine-tuning, evaluation, monitoring and home automation.
Hardest problem
Keeping many large models available on a fixed set of GPUs, and swapping between them in seconds.
Measured result
One OpenAI-compatible endpoint for models from 27B to 753B, with hot swaps in seconds: a 320B model at up to 187 tokens per second and a 125B model at 185 tokens per second, each on two GPUs.

Built with

  • vLLM,
  • SGLang,
  • llama.cpp,
  • ExLlamaV3,
  • TabbyAPI,
  • Ollama,
  • PyTorch,
  • PEFT,
  • FSDP2,
  • Hugging Face TEI,
  • Prometheus,
  • VictoriaMetrics,
  • Grafana,
  • DCGM,
  • RabbitMQ,
  • Home Assistant,
  • Wyoming,
  • Frigate,
  • systemd
Full technical detail9 items

The platform around the AI server, run like a small private datacenter: a 32-vCPU AMD EPYC server with 377 GB of RAM, two RTX PRO 6000 Blackwell GPUs (96 GB each) and an NVIDIA L4 utility GPU, a second GPU node in Dallas, five storage servers with about 255 TB, and a build container for test gates. One OpenAI-compatible router sits in front of every model, and the same hardware handles fine-tuning, evaluation, monitoring and home automation.

What I built

  • Model router: a custom OpenAI-compatible router in front of every model from 27B to 753B that hot-swaps single-server models in seconds with per-model server binaries and context lengths, vLLM sleep mode (wake in 2 to 5 s), training-hold sentinels that keep inference off the GPUs during fine-tuning, and a self-heal watchdog that recycles a backend unhealthy for 5 minutes.
  • Engines: vLLM (pinned digests, TP2 with custom all-reduce for +8.4% decode on a 123B dense model, and a two-server TP2 x PP5 pipeline for GLM-5.3), SGLang (NVFP4 for GLM-5.3-Flash), llama.cpp (GGUF with a verified reasoning-budget cap), TabbyAPI/ExLlamaV3 (exl3 with MTP), Ollama; Hugging Face TEI serving Qwen3-Embedding-4B (2560-dim) and mxbai-rerank-base-v2 on the L4.
  • Models: a 123B dense model in AWQ, Mistral-Medium-3.5-128B FP8 with EAGLE, Hy3 W4A4 NVFP4, Qwen3.6-27B (NVFP4 and Q8), Qwen3.6-35B-A3B Q8, Qwen3.5-122B-A10B Q8, Llama 3.3 70B FP8, Qwen3.8 Flash-Next 125B NVFP4 and GLM-5.3-Flash 320B MoE, with the two largest, Mistral Large 4 and the full GLM-5.3, on the CMP 170HX AI server.
  • Speculative decoding: Qwen3.8 Flash-Next 125B on vLLM TP2 with MTP at 185 tok/s single-stream (73% draft acceptance, 3.4x llama.cpp’s 55 tok/s, a 3.18M-token KV pool at 262k context); Mistral-Medium 128B FP8 with EAGLE k=5 at 42 tok/s (2x decode, 12 s TTFT at 32k); GLM-5.3-Flash at 52.6 tok/s on llama.cpp, 72.8 with MTP plus n-gram drafting, 79.8 on SGLang NVFP4 without speculation, 84.5 on ExLlamaV3 over 1,680 live generations at 58.3% MTP acceptance, and 152 to 187 tok/s on SGLang with EAGLE. Hy3 W4A4 serves four concurrent streams at 165.5 tok/s.
  • Training: LoRA fine-tuning with DDP and FSDP2 across both 96 GB GPUs, including GLM-5.3-Flash, a 320B mixture-of-experts model, trained by streaming frozen expert weights from host RAM one layer at a time.
  • Monitoring: an LLM monitoring API with Postgres, RabbitMQ, Prometheus and GPU telemetry, Grafana alerts that trigger an LLM-written root-cause card, and Grafana high availability across two sites.
  • A compute pool that fans builds and test runs out to a storage-server container and a Windows laptop over per-run tunnels, with no keys left on the remote hosts.
  • Agent Cockpit: a Fastify and React 19 control plane for multi-agent work, where a planner assigns a model to each subtask and an opposite-model reviewer checks the result.
  • Home Assistant AI: the CABAL voice assistant (openWakeWord, faster-whisper large-v3-turbo on the L4 over Wyoming, a local LLM with house control through the Assist API, and a Wyoming TTS bridge to a cloned voice with a phrase cache), Frigate NVR with jina-clip semantic search (1,677 events indexed in 38 s), face recognition, audio detection and Qwen3-VL-8B event descriptions, dashboards defined as code over the HA WebSocket API, and a faster-whisper transcription tool with SpeechBrain ECAPA speaker labels.