
- What it is
- An all-in-one AI app that runs entirely on your phone, on Android and iPhone.
- Hardest problem
- Making dozens of models run fast and correctly across phone chips whose GPU drivers disagree on basic math.
- Measured result
- 10,000+ installs in 169 countries and 3,000+ active users. One app, every modality, fully on-device: chat, photo, audio and video understanding, document search, image generation, speech, voice cloning and tool-calling agents. A 35-billion-parameter model runs on a phone at 9 to 12 tokens per second, and the phone’s NPU reads long prompts about 4x faster than its CPU.
Built with
- Kotlin,
- C++,
- Swift,
- Jetpack Compose,
- SwiftUI,
- llama.cpp,
- MNN,
- ggml,
- Vulkan,
- OpenCL,
- SPIR-V,
- Qualcomm QNN,
- Core ML,
- MLX,
- TFLite/XNNPACK,
- sd.cpp,
- Kokoro,
- Sherpa-ONNX,
- WhisperKit,
- Room,
- Hilt
Full technical detail
TokForge is an on-device AI app for Android (Google Play, Android 8 and up) and iOS/iPadOS (public TestFlight beta), with 10,000+ installs in 169 countries and 3,000+ active users. It runs LLM chat, vision, audio and video understanding, speech (Kokoro TTS, ZipVoice voice cloning, speech input), image generation and document RAG entirely on the phone: no account, no telemetry by default (the speed leaderboard is opt-in), and no cloud processing unless you connect your own OpenAI-compatible server. Three engines sit behind one Kotlin/C++ JNI runtime: llama.cpp (GGUF on CPU, Vulkan and OpenCL), a maintained fork of Alibaba MNN (CPU, OpenCL, Vulkan, QNN), and an OpenAI-compatible remote client. A per-SoC routing table (Universality over Stability over Performance) sends each model to the fastest proven backend and gives unknown silicon the slowest proven path. 9 UI languages and 19 public releases since 1.0 in August 2026, including four production releases in 16 days with crash and ANR rates inside Google Play’s thresholds.
What I built
- LLMs on the phone’s NPU: on Snapdragon phones the Hexagon NPU reads the prompt while the CPU writes the reply, so long prompts are processed about 4x faster than on the CPU alone. Qwen3.5 4B and 9B ship as opt-in NPU packs, each used only after a check on the phone itself.
- Desktop-class models on phones: Qwen3.6 35B-A3B (35B) at 9 to 12 tokens per second on a 24 GB RedMagic 11 Pro, and Qwen3-30B-A3B (30B) at 14.4 tokens per second on an iPhone 17 Pro Max in the public TestFlight beta. Bonsai 27B (ternary) reached 8.5 tokens per second on Vulkan on a Snapdragon 8 Elite Gen 5 phone in September 2026.
- One model for photos, audio and video: LFM3 Omni understands text, photos, voice and audio, and video on the phone, with no helper model and nothing sent to a server.
- Vulkan GPU inference on Adreno, Mali and Xclipse: root-caused vendor driver bugs with ggml test-backend-ops one operation at a time (Adreno 750 q6_K mat-vec, Adreno 840 quantized KV below head size 256, a shader-compiler crash on wide batches) and encoded per-chip rules, a pipeline-cache integrity check and a phase-tagged crash guard. On a Snapdragon 8 Elite Gen 5 phone, GPU decode ran 1.3x to 2.4x faster than the leading competitor’s GPU path on the same models.
- CPU and GPU on one model: the GPU reads the prompt and the CPU writes the reply, with a GPU self-test and routes measured per phone and model. Multi-token prediction for Qwen3.5 (0.8B to 27B) and draft-model packs switch on only where the phone measures a gain, and multi-token prediction works with the NPU.
- MoE expert streaming for llama.cpp: a native streamer reads only routed experts from storage with next-layer prefetch, 4 I/O lanes and an LRU cache sized from MemAvailable, so Qwen3-30B-A3B (30B) runs on a 16 GB OnePlus Ace 5.
- Stable Diffusion 1.5 on the Hexagon NPU through Qualcomm QNN context binaries, non-root: 512x512 in 6 to 10 s. GPU image generation on Mali and Xclipse phones after an on-phone image check. Also sd.cpp (SD-Turbo, SD1.5, SDXL, Flux) and a Sana image edit at 25 s per 512 px frame on a Galaxy S24+.
- On-device RAG: PDF, DOCX, EPUB, Markdown and CSV ingestion, INT8 BGE-small embeddings on TFLite/XNNPACK (1.5x to 2.4x faster than the MNN encoder on every fleet SoC), hybrid FTS4 plus vector retrieval with MMR, memory facts, reflection and a knowledge graph, and an embedder swap with staged re-embedding.
- KV cache work: KV reuse across chat turns in MNN so follow-up replies start almost instantly, later refined into token-exact replay that prefills only the new part of the conversation, and a TurboQuant 4-bit KV cache in the MNN OpenCL path (3.5x less KV memory at 8k context, 4x at 32k).
- Agents and tools: tool-calling agents with live run screens and presets, document search over chat attachments, and an opt-in sandboxed terminal where the model proposes a command and the user approves each one.
- AutoForge: an in-app micro-benchmark sweep over backend, thread and decode settings per SoC and model, auto-applied on load and feeding an opt-in public leaderboard on a Cloudflare Worker with D1; FleetPerfOptimizer learns per-tuple defaults from clean telemetry.
- iOS port in Swift and SwiftUI with llama.cpp, MNN, MLX and a Core ML/ANE LLM engine, sd.cpp and Core ML Stable Diffusion, Sherpa-ONNX and WhisperKit, with an mmap-aware memory gate for large models on a 12 GB iPhone.
- Robustness: a stall watchdog (stop, native abort, recover), persistent per-model OpenCL and Vulkan crash guards with automatic downgrade to CPU, a single RAM-admission rule for every load surface, and model files that are damaged or hostile are refused with a reason.
- Hardware coverage: every release is tested on 12 Android devices (SM8850, SM8650, SM8250, SM7635, MT6991, MT6989, MT6765, Exynos 2400 and 1380, Tensor G4 and G5) and 4 Apple devices, with crash and ANR monitoring through the Google Play and App Store Connect APIs.
- 106 public model repos on Hugging Face (MNN conversions, QNN NPU context binaries, image-generation GGUFs, draft packs), with 180,000+ downloads.






