For the past several years, much of the conversation around artificial intelligence has focused on models: which models organizations should use, how quickly they are improving, and what new capabilities generative and agentic AI will unlock.
But as AI moves from experimentation to production, another issue is becoming increasingly difficult to ignore: the infrastructure required to support it.
A recent ITPro article highlights just how significant that shift may become. Citing Gartner research, ITPro reports that spending on AI-optimized infrastructure is projected to grow 96% in 2026, reaching $42 billion, with continued growth expected as organizations deploy increasingly sophisticated AI workloads.
The reason is relatively straightforward. AI does not behave like a traditional enterprise application, and agentic AI may amplify that difference considerably.
From Training AI to Running AI
Much of the early infrastructure discussion around AI centered on training large models and the enormous compute resources required to do it. But as AI adoption matures, inference, the infrastructure required to actually run AI models and applications, is becoming an increasingly important part of the equation.
According to the ITPro article, Gartner expects spending supporting AI inference to surpass spending on training for the first time. Gartner also projects that 55% of AI infrastructure spending will support inference, increasing to 59% by 2027. Agentic AI could accelerate that trend.
Unlike a relatively simple chatbot interaction in which a user submits a prompt and receives a response, an AI agent may independently execute multiple tasks, interact with multiple models, retrieve information from multiple data sources, and make numerous model calls before completing a request.
One human interaction can therefore generate a much larger amount of underlying infrastructure activity.
For organizations planning their AI strategies, the implication is important: the challenge is no longer simply gaining access to AI. It is building an infrastructure capable of sustaining AI at scale.
AI Infrastructure Is More Than GPUs
That also means organizations need to think beyond compute.
GPUs are essential to many AI environments, but GPUs alone do not create an effective AI platform. Models need continuous access to enormous volumes of data. That data needs to be stored, moved, governed, and delivered to compute resources quickly enough to avoid creating expensive bottlenecks.
AI infrastructure therefore needs to be viewed as an interconnected system encompassing:
- Accelerated compute
- High-performance storage
- High-bandwidth networking
- AI-ready data pipelines
- File and object services capable of supporting massive unstructured datasets
- Data orchestration and mobility
- Security and governance
- Infrastructure management and observability
The performance of the AI environment ultimately depends on how effectively all those components work together.
As AI workloads grow, organizations may increasingly discover that the limiting factor isn’t the model itself. It is the infrastructure and data architecture feeding it.
There May Not Be One Right Place to Run AI
The infrastructure discussion becomes even more important when organizations consider where different AI workloads should run.
The ITPro article points to growing interest in hybrid and multi-cloud strategies as AI adoption expands. Some organizations may use cloud capacity for compute-intensive model training while deploying inference closer to the data, users or applications that depend on it. Other workloads may remain on-premises because of security, sovereignty, latency, performance or cost requirements.
For federal agencies, these considerations can be particularly significant.
Mission data may reside across data centers, private clouds, public clouds, edge environments, and specialized research or operational systems. Security and data sovereignty requirements may limit where certain information can move. Real-time applications may require inference close to where data is generated. And large-scale training or experimentation may benefit from elastic cloud resources.
The result is unlikely to be a single architecture for every AI workload.
Instead, agencies will increasingly need to answer questions such as:
Where does our data reside? Where should the model run? Where should inference occur? How quickly can data reach the compute environment? What information can move between environments? And what will it cost to operate this architecture at scale?
Those are infrastructure questions as much as they are AI questions.
Hybrid AI Requires a Data Strategy
A hybrid or multi-cloud AI architecture only works if the data supporting it can be accessed efficiently.
Moving massive datasets every time an AI workload changes location can introduce cost, latency, and operational complexity. Creating multiple copies of the same data can introduce additional governance and security concerns.
This is why data infrastructure and data mobility are becoming central to AI architecture.
Organizations need to think about how data can be stored, accessed, governed and orchestrated across environments without creating another generation of infrastructure and data silos.
The objective should not necessarily be to move everything to the cloud, or to keep everything on-premises.
It should be to create an architecture that gives organizations the flexibility to place AI workloads where they make the most sense while maintaining access to trusted, governed data.
Building the Foundation for AI
At Hitachi Federal, this infrastructure-first view is central to how we approach AI readiness.
Our Hitachi iQ strategy brings together accelerated compute, high-performance storage, networking, and AI software to create an integrated foundation for AI workloads. Hitachi’s broader data portfolio extends that foundation with high-performance file and object storage, data orchestration, data integration and infrastructure management capabilities designed to help organizations manage increasingly distributed data environments.
For many federal use cases, maintaining AI infrastructure on-premises provides important advantages around data control, security, predictable performance, and sovereignty. But that does not mean AI architectures need to be isolated from cloud environments.
The goal is to establish an AI-ready data and infrastructure foundation capable of supporting workloads across on-premises, cloud and edge environments, giving agencies greater flexibility as their AI requirements evolve.
Infrastructure Is Becoming an AI Strategy
The next phase of AI adoption will not be defined solely by who has access to the most advanced models.
It will also be defined by which organizations can operationalize those models reliably, securely, and economically.
As agentic AI increases inference demands and AI applications become more deeply integrated into mission workflows, infrastructure decisions made today will have an increasingly significant impact on what agencies can accomplish tomorrow.
The conversation around AI is therefore expanding.
It is no longer simply “What can AI do?”
Federal technology leaders also need to ask: “Is our infrastructure ready to support what comes next?”