There is a version of every founder’s story where compliance is the thing you handle after product-market fit. You build fast, you win customers, and then you bolt on SOC 2 because a buyer asked for it. In commercial SaaS that story mostly works. In federal work it is a way to build something you have to throw away.
I learned this building a GovCon intelligence engine and looking hard at what it would take to move AI systems into defense-adjacent work. CMMC, ITAR, and data residency are not a checklist you run before launch. They are constraints that reach all the way down into your stack and decide things you cannot reverse later: which cloud region you run in, which model providers you can call, where a single log line is allowed to live. Get those wrong on day one and no amount of later effort fixes it, because the wrong decisions are already baked into your infrastructure and your data has already been somewhere it should not have been.
Compliance is a set of physical facts about your bytes
The mistake is treating compliance as policy. It is not policy. It is a set of physical claims about where your data lives, who can touch it, and what crosses a border.
ITAR is the sharpest example. If your system handles technical data covered by the export regs, that data cannot be accessed by a non-US person, and “accessed” includes a support engineer at your cloud provider and an inference call to a model hosted who-knows-where. That is not a setting. It is a hard boundary that dictates your provider, your region, and your entire model strategy.
CMMC Level 2 is a set of physical and procedural controls over controlled unclassified information: access control, audit logging, encryption, incident response, all assessed. Data residency for federal work usually means US-only, often a government cloud region, sometimes FedRAMP-authorized services only.
Compliance is not a document you write about your system. It is a description of where your bytes are physically allowed to be. If the architecture puts them somewhere else, the document is fiction.
The day-one decisions you cannot reverse
Here is the concrete list of choices that compliance makes for you before you write a line of business logic. These are the ones that are catastrophic to change later.
| Decision | Commercial default | Federal constraint |
|---|---|---|
| Cloud region | cheapest / nearest | US gov region (GovCloud / IL-rated) |
| Model provider | best API available | US-hosted, contractually scoped, no training on your data |
| Logging / observability | SaaS vendor, anywhere | in-boundary, no CUI leaving the enclave |
| Identity | any IdP | FedRAMP-authorized, MFA, PIV-capable |
| Data store | managed, multi-region | single US region, encryption at rest with managed keys |
| Third-party enrichment | whatever is best | vetted for where it sends and stores data |
Every row of that table propagates. Pick a commercial model API because it is the best one, and the day you handle CUI you discover your inference calls have been crossing a boundary they were never allowed to cross. You do not patch that. You rebuild the inference layer and you have an incident to disclose.
Where this bit me with AI specifically
The LLM layer is where compliance and modern AI architecture collide hardest, because the entire industry’s default is “call the best hosted model over the internet.” That default is incompatible with most federal data handling.
When I think about taking an AI system into CUI-adjacent territory, the model question splits into three:
- Where does inference physically run? A commercial endpoint in some default region is a non-starter for controlled data. The options collapse to a model running inside the compliant boundary, or a provider with a contractually and physically scoped federal offering.
- What does the provider do with the prompt? “We do not train on your data” is a contract term, not an architecture. For federal work the prompt itself may be controlled data, and the question is not just training, it is whether that data ever left the enclave at all.
- What about everything around the model? This is the one people miss. The prompt is controlled, so the logs that capture the prompt are controlled. Your trace store, your eval pipeline, your prompt-debugging tooling, all of it is now handling CUI. I have argued before that you should instrument agent trajectories aggressively. In a federal context, those trajectory logs are CUI, and they cannot live in a commercial observability SaaS.
That third point is the quiet killer. You can get the model provider right and still blow your boundary because your debugging stack ships prompts to a vendor for convenience.
RAG makes residency harder, not easier
The retrieval-augmented approach I lean on for solicitations has a compliance wrinkle that pure generation does not. The moment you build a vector store over controlled documents, your embeddings are derived from controlled data, and embeddings of CUI are arguably CUI. So the embedding model has to run in-boundary, the vector store has to be in-region, and you cannot casually use a hosted embeddings API, because that ships the controlled text out for vectorization.
This is the kind of thing that is invisible in a commercial build and load-bearing in a federal one. The architecture that makes a retrieval engine smart is exactly the architecture that drags compliance into every storage decision.
The pattern: a boundary, drawn first
The way I think about it now is that you draw the compliance boundary before you draw the architecture diagram, and then nothing controlled is allowed to cross it.
┌─────────── compliant enclave (US region, audited) ───────────┐
│ controlled data store in-boundary inference trace/logs │
│ embeddings + vector db identity (FedRAMP IdP) │
└──────────────────────────────────────────────────────────────┘
│ only de-identified / public data crosses
▼
commercial conveniences (public award feeds, dashboards)Public federal data can live outside the enclave because it is public by law. The instant you join it to anything controlled, the join result is inside the boundary. The boundary is the architecture. Everything else is implementation detail.
Why “later” is a trap
The seductive argument is always: get traction first, harden later, because most startups die from lack of customers, not from compliance gaps. For commercial markets that is correct. For federal it inverts, because the compliance posture is a precondition for having a customer at all. There is no “traction” to get first. The buyer cannot legally hand you the data until the boundary exists.
And the rebuild cost is not linear. Retrofitting compliance means re-regioning your infrastructure, replacing your model layer, ripping out your observability stack, and re-implementing identity, all while explaining to a security officer why controlled data passed through systems it should not have. That last part is not an engineering cost. It is a disclosure, and it can end the relationship.
What this changes about how I build
It makes me slower on purpose, and only in federal contexts. For commercial products I still build the simplest thing that works and harden as I learn. For federal-bound systems, the boundary comes first, even before I know exactly what the product is, because the boundary is the one decision that is cheap on day one and ruinous on day three hundred. It is one reason I’m designing Vontra so the enclave seam is a deployment topology, not a rewrite.
It also means I am ruthless about what touches the enclave. Every dependency, every model call, every log sink is a question: does this respect the boundary, and can I prove it. The dependencies that cannot answer do not get in, no matter how much better they are.
Compliance as a checklist is something you pass. Compliance as architecture is something you are. In federal work only the second one survives an assessment, and the difference is decided entirely by choices you make before you have written the interesting code. Built for missions means the boundary is the first thing on the whiteboard, not the last thing before launch.