Adaptive Compression Cuts Edge RAG Energy in Jetson Experiments

Overview

• Researchers propose telemetry‑informed adaptive compression as a way to prune retrieved context at runtime before a model generates text on an edge system.
• Measured on an NVIDIA Jetson AGX Thor with Llama and Qwen generators, the paper finds generation is the dominant cost, accounting for about 90% of per‑query latency and 91% of GPU energy for 7B–8B models.
• The study emphasizes that compression itself uses CPU/GPU time and power on the same system, so careless or fixed compression settings can erase any savings.
• Tuned to an intermediate compression regime, the experiments report up to about 53.2% GPU energy savings and about 48.2% SoC energy savings with negligible drops in answer quality.
• The work is an arXiv preprint based on controlled tests with LLMLingua‑2 compression, specific datasets, and one SoC, and it requires independent replication and production validation before broader adoption.