Particle.news

Thinking Machines Releases Open Inkling 2 Mini, a 26.4B On‑Device Mixture‑of‑Experts

The 20 GB Apache 2.0 checkpoint makes local suggestion and short translation practical while live bilingual overlays still face a hardware and audio‑engine latency floor.

Overview

  • Thinking Machines Lab published Inkling 2 Mini as an Apache 2.0 checkpoint that is a 26.4 billion‑parameter mixture‑of‑experts model packaged in about 20 GB and runnable on single‑node DGX Spark, Apple Silicon with 32 GB or more, or a capable discrete GPU.
  • The model accepts text and images and supports a very long context window, which lets developers run on‑device suggestion generation, short translation polish, and agentic coding tasks without sending every token to a remote endpoint.
  • Engineers reporting on the release show that, even with fast local inference, the Windows audio capture and early speech‑to‑text stages set the practical latency floor for live overlays and typically produce end‑to‑end times from end of speech to visible text of about 150–400 ms.
  • Producing usable bilingual overlays requires dual‑track capture so both speaker and remote audio remain separate, and on Windows the recommended approach is two IAudioClient instances (mic plus loopback) with non‑blocking callbacks, lock‑free rings, and off‑audio‑thread processing.
  • Practical optimizations that cut latency include staying in WASAPI shared mode with the smallest honored buffer, moving framing and resampling off the audio thread, preferring local or nearby STT, and using Mini only for short polish so the audio path never waits on model inference.