How We Cut Federal Data Onboarding From Weeks to Hours Without Crossing the Boundary

The situation

The agency runs a growing network of external data repositories, each contributed by a different partner organization, each with its own access method, schema, and quirks. Information specialists were responsible for pulling data from every repository and standardizing it against the agency's common data model so downstream analysts could query consistently.

When we walked in, the work was almost entirely manual. A specialist would investigate a new repository, figure out whether it spoke REST, SQL, file export, or something custom, write the connector, hand-map fields into the common data model, then build and test the ETL. Every new source was a small project unto itself.

The backlog was growing faster than the team could clear it. Hiring more specialists was not on the table.

The challenge

The team was carrying four problems at once.

1. Throughput. Every new repository was a multi-week investment. The source network was expanding faster than specialist capacity.
2. Consistency. Hand-built connectors and ad-hoc mappings drifted over time, which weakened downstream analytics.
3. Sovereignty. Federal data handling rules meant nothing could leave the accreditation boundary. Sending agency data to a commercial AI API was off the table from day one.
4. Governance. Specialist judgment had to stay in the loop. Any AI-assisted system had to be auditable, reviewable, and explicitly approved before any data hit the common data model.

The approach

The operating model: small team, human-in-the-loop by design

Three ExeQut technologists worked alongside the agency's COR, program manager, and analyst team. We treated the analysts as part of the system, not as users of it. The platform's job was to do the boring 80 percent of every integration so the specialists could spend their attention on the 20 percent that actually required domain judgment.

Human-in-the-loop was not a guardrail bolted on at the end. It was the operating model from day one.

Every AI-suggested mapping is reviewed and approved by a credentialed specialist before any data is written to the common data model. Approvals and corrections feed back into the platform, which means the system gets sharper as the analysts work.

The technical architecture: sovereign models, in-boundary inference

The agency operates in AWS GovCloud under a full Authority to Operate. Inference had to stay inside that boundary, which ruled out commercial frontier model APIs. We deliberately avoided top-tier commercial models in favor of sovereign, in-boundary inference. No agency data leaves the accreditation boundary at any point in the workflow.

Sovereignty was not a constraint we worked around. It was a design input that shaped every model and inference decision.

The agent itself is composed of Python Lambdas orchestrated around three jobs: investigate a target repository, propose a field mapping, then execute the validated load. Amazon OpenSearch handles full-text and semantic search across ingested content. Amazon Neptune holds the relationship graph of sources, schemas, and entities, queried with SPARQL. Amazon Cognito handles identity and analyst access.

The common data model: enhance, do not replace

The agency already had a common data model. We did not propose replacing it. We worked inside it, then suggested enhancements where the existing schema was struggling to absorb new source patterns. That kept the analytics layer stable and the modernization politically clean.

Deployment and release: piece by piece, never big bang

We started in 2024 with a fully manual baseline and modernized one workflow at a time. Repository discovery automation came first. Mapping suggestion came next. Auto-execution of approved mappings followed. Each piece was deployed, validated by the analyst team, and ATO-confirmed before the next piece was built.

The platform compounds. Each approved mapping makes the next one faster.

The outcome

What used to take weeks of specialist effort now takes days for novel repository types and hours for patterns the platform has seen before. Twelve repositories are live in the common data model today. The roadmap targets forty in the coming year, and the pattern library is mature enough that the marginal cost of each new source keeps dropping.

The non-financial outcomes matter just as much for a federal program. The platform sits inside the existing ATO. It honors data sovereignty without exception. It gives the COR and program manager an auditable record of every AI suggestion, every analyst correction, and every load. And it scales mission output against the source network rather than against headcount, which is the only growth curve federal budgets actually permit.

What we took from it

A few things worth carrying into the next federal AI engagement.

• Sovereign inference is a feature, not a limitation. Avoiding commercial frontier APIs forced cleaner architecture, sharper prompt design, and full auditability. The result was more defensible at ATO time, not less.
• Make the analysts part of the system. AI agents that try to replace specialist judgment fail federal review. Agents that amplify specialist judgment ship.
• Modernize against the existing common data model. Layering AI onto established standards is faster, safer, and far more politically viable than proposing replacement.
• Compounding beats throughput. Onboarding speed matters less than whether the platform learns from each integration. A flat throughput curve eventually loses to a compounding one.
• Three good engineers and three engaged feds can beat a big team. Small, paired teams with shared accountability move faster than large ones split across vendor and government lines.