The United States government publishes the single most valuable sales dataset in federal contracting, for free, and most companies in the market never touch it in any structured way. SAM.gov carries every solicitation, notice, and entity registration. USAspending carries every federal award and the transactions behind them. The data is public. The signal is not. The gap between those two sentences is the entire premise of Atlas, and this finding is about the first half of closing it: the pipeline.
Two sources, two different problems
The two sources look similar and are not. SAM.gov is a live feed of opportunity data: solicitations, amendments, award notices, each with its own lifecycle and its own rate of correction. USAspending is award spending data derived from agency financial systems and FPDS contract records: who won, how much, under which vehicle, with transaction level detail that arrives on agency reporting cycles rather than in real time.
That difference shapes the architecture. SAM.gov data needs freshness, because a solicitation posted this morning is actionable and one corrected last night may invalidate a week of analysis. USAspending data needs depth, because a single award is a header over many transactions, obligations versus outlays, modifications, funding agencies and awarding agencies that are not the same thing. So the pipeline treats them differently: SAM.gov on a frequent incremental sync keyed on posted date and last modified, USAspending on a slower reconciliation sweep that assumes late and amended records as the normal case, not the exception.
The quirks are the work
Every production pipeline over government data is 20 percent transport and 80 percent coping. Some of what Atlas copes with, in no particular order. Recipient names that are not unique and not stable, so the same company appears under several name variants and UEI is the only honest join key, with UEI itself needing history because registrations change. Award amounts that are ceilings, not obligations, and must never be presented as revenue without saying which one they are. NAICS codes and PSC codes that are inconsistently populated across agencies. Place of performance that is a different concept from the contractor’s location. Records amended after award, sometimes materially.
None of these can be fixed by cleaning harder at ingest. They get handled by modeling them: keeping raw records immutable in a landing layer, normalizing into canonical entities with provenance attached, and timestamping every fact so a question about “what we knew in March” is answerable. When a pipeline can replay what it knew and when, data corrections become events instead of mysteries.
Normalized entities, not mirrored tables
The landing layer mirrors the sources. The canonical layer does not. Atlas normalizes into entities a customer’s questions actually reference: company, award, transaction, vehicle, solicitation, agency, place. Entity resolution, mostly UEI anchored with name and address corroboration, is where a pipeline like this earns its keep, because the question customers ask is never “show me row 4,511 of the award table.” It is “what has this company won lately,” and answering it requires having decided, provably, which records belong to that company.
Why this matters beyond Atlas
The reason I consider this finding worth publishing is that it generalizes. The same pattern, public sources, honest landing layer, canonical normalization with provenance, incremental sync that assumes corrections, is how I approach any customer software problem where the raw material is public data. The dataset is free. The intelligence layer built on top is the product, and the pipeline is what makes that layer possible at all.