Adaptive, Telemetry‑Guided Compression Cuts GPU Energy on Edge RAG Systems

The authors show runtime policies that use on‑device telemetry can balance the compressor’s own energy cost against savings from smaller model prompts.

Overview

An arXiv preprint describes experiments on a NVIDIA Jetson AGX Thor that measure how compressing retrieved text affects latency, GPU energy, and output quality in retrieval‑augmented generation (RAG).
The study finds generation dominates costs for 7B–8B models on the tested board, accounting for about 90% of per‑query latency and roughly 91% of GPU energy, so cutting prompt size can yield big savings.
The authors identify an intermediate compression operating region where GPU energy fell by up to 53.2% and total SoC energy fell by up to 48.2% while quality losses were negligible.
They warn compression is not free on edge SoCs because the compressor runs on the same chip and uses latency and energy, so a static compression budget can waste resources or harm quality.
The paper recommends runtime, workload‑aware policies that use edge telemetry to pick compression rates and calls for broader replication and validation across other hardware, models, and real workloads.