Overview
- Recent guidance reframes retrieval-augmented generation as one component of a larger system and urges a routing layer that decides when to retrieve, call an API, clarify, refuse, or escalate to a human to avoid exposing incorrect or unauthorized data.
- Provenance practices used in production include content-addressable storage that versions every document with a new UUID and vector, plus signed retrieval manifests saved to an encrypted audit log before any LLM generation occurs.
- To meet HIPAA requirements engineers must log who accessed which version of a record and why, keep short-term full prompts and long-term retrieval manifests with pointers to versioned FHIR resources, and record UserUUID and Purpose of Use for audits.
- Teams will face clear trade-offs because auditability adds latency and storage cost: manifest creation adds about 40–100 ms per request, versioned embeddings make indexes grow faster and require tombstone tiering, and reranking raises query latency by roughly 100–500 ms.
- The practical fix is operational and architectural: improve retrieval quality with semantic chunking, hybrid dense+keyword search and rerankers, enforce ACLs at search time, run strict deterministic verifiers that block unsupported claims, and build eval and observability before tuning prompts.