How CDNs Are Battling Latency With Edge AI Caching

20 September 2026

Content delivery networks are going beyond caching: they're now hosting AI at an unprecedented scale. A flurry of announcements shows major CDNs building out intelligent orchestration for real-time inference at the edge.

This shifts the balance in AI infrastructure. CDNs, originally built to host static sites, are now rivaling the compute capacity of centralized data centers. And by distributing this capacity near the user, they promise to give developers an alternative to the back-and-forth delays that are an industry-wide bottleneck.

What changed?

Edge AI caching refers to storing AI model outputs or intermediary results close to the end user, and serving these when possible to cut latency and cost. This is distinct from edge compute, which doesn't use the network as a cache: dialogues with AI agents still need to go back to a cluster each time.

The reason is net latency: even central AI inference at speed is still slower than a local cache lookup. Take physical domains like robotics and autonomous vehicles, which often coordinate multiple agents, perform frequent updates, or rely on instant pattern-recognition at a user location. Round trips to a distant cluster to request a single answer or next action can quickly pile up the delay.

Akamai AI Grid: 4,400 Locations and a DIY Orchestrator

In September, Akamai kicked off this transition by announcing the release of AI Grid, a global-scale orchestration system for AI inference across 4,400 edge locations worldwide using NVIDIA AI.

According to Akamai, AI Grid consists of three parts: a plane of compute resources at the edge, one at regional hubs, and one at core cluster locations. AI workloads are then dynamically allocated to one of three by an intelligent orchestrator which balances latency, cost, and performance in real time.

Akamai didn't share exact latency and cost numbers, but described three key metrics: improved cost per token, faster time-to-first-token, and higher throughput. These likely come from improvements to both model performance and reduced backhaul where possible.

The key to the latter is caching. Akamai says AI Grid uses serverless functions like Akamai Functions and EdgeWorkers at the edge, making it easier to deploy semantic caching closer to the user. This can reduce repeated calls, improve model performance, and keep tokens per call down without using extra load.

Semantic caching is a nuanced form of caching which tries to cache based on the meaning, rather than just the syntax of the request. So you would store models of outputs, rather than exact commands or responses.

AI Grid also supports NLP applications, using pre-trained models to predict the rest of a query based on its prefix. This is powerful, but only applies to about 50% of requests. Semantic caching has greater potential, though, given the right architecture and traffic patterns.

AI responses and calls are also cached at a regional level, but only with a higher threshold for whether the costs of storage and reuse justify the latency savings. The core orchestration layer can manage AI tasks which are too heavy or complex for a regional centre, either performing a model inference there, or as a fallback when an edge copy fails.

The result is an all-encompassing AI inference stack, with decision making handled at all three levels. Developers can deploy their models across all three layers, though of course weight depends on the right heuristics for what, if anything, can be cached.

Developers don't need to manually configure AI delineation between layers: this is done by an intelligent orchestration layer which customizes a mapping of tools, models, and compute to infrastructure based on monitored traffic, costs, and heuristics for each deployment.

Ask your agent to make the coffee

Semantic caching isn't the only approach to edge AI. Take Fastly, for example, who announced last year that they've extended this to also handle multi-call AI agents.

Agents are a common architecture for guiding users to the desired outcome, or completing a task. But one challenge is that they often perform multiple calls in a loop, for example iterating over a set of tools to gather the latest information. Each call can result in a significant latency when an MCP server is hosted far from the user.

Early tests showed that hosting these at the edge could cut this time down. According to Fastly, this is especially effective for the 'discovery and listing' portion of an agent: calls like `server/discover`, `tools/list`, `prompts/list`, and `resources/list`. These are often identical for all users and don't require repeated network calls.

Fastly demonstrated this with its offering Fastly AI Accelerator, announced last year. It handles caching responses for repeated queries, and just requires changing an API endpoint. So in effect, this can be entirely handled by updating a single line of code.

What does the industry say?

So far what all agree on is that edge AI caching can be a game-changer for user experiences. But the roadmap from early tests to shipping a complete solution is still being mapped out.

Take one study last year from MDPI, which showed how combining edge caching, mobile edge computing offloading, and reinforcement learning could reduce latency by 88% for requests on a hybrid CDN. The most dramatic results showed latency ranging from 20 - 65 ms versus traditional CDN.

Common customer use-cases came in at 70% content, 30% compute. And that nearly 95% of compute tasks ran at the edge with locality, while over 88% of content requests were cached locally.

All of these cases build a compelling case. So what happens next, and what could go wrong?

If so, edge-cache will face restrictions. Cache efficiency depends on the right workloads: those which produce retrievable results, and serve a user community which shares a large enough pool. Real-time tasks may not process requests predictably. And predictive reasoning may be too situational to cache. Model training may be stuck at the core.

Cost efficiency is another factor: the plans must be affordable to run edge nodes, and have logical heuristics to route the AI traffic. What sizes of inference fit the compute profile of each node? How far can requests be pushed? And how strongly do developers need to isolate their models from competitors?

These are all early day questions, but they'll be a major driver of the competitive landscape in the years to come. The biggest incentive of all is that in rebuilding their infrastructure, the edge providers can position themselves as leaders in AI delivery. The accuracy of Akamai and Fastly's models is still to be proven, but their solutions offer promise for those who need responsive AI applications.