Overview
- Iroh published Mesh LLM's architecture and tooling, describing a localhost OpenAI‑compatible endpoint at http://localhost:9337/v1 that presents pooled GPUs and memory as a single inference API.
- Requests can be served three ways: run on the local GPU, route to a peer that already hosts the model, or split a very large model across multiple nodes using the 'Skippy' pipeline partitioning mode.
- Nodes run iroh endpoints that provide authenticated, NAT‑traversing QUIC links and regional relays so peers can connect directly by public key without relying on a central server.
- The project ships a pluggable runtime and a published catalog of 40+ models from ~0.5B to 235B parameters, with a forthcoming mobile client announced but not yet released.
- The coverage positions Mesh LLM plus Iroh as a privacy‑focused, edge‑capable alternative to cloud hosting while noting that federated training, participant incentives, and wide production deployment are promising but still undeveloped.