Gcore introduces Global Inference Routing accelerated by NVIDIA Dynamo

Earlier this year we brought NVIDIA Dynamo to Gcore — one-click disaggregated inference that delivered up to 6× higher GPU throughput and 2× lower latency inside a deployment, by separating prefill and decode and routing each request to the GPU that already held its KV cache.

That solved efficiency inside a cluster. But our customers don’t run in one cluster. Everywhere AI runs across on-premises sites, multiple clouds, and edge regions — for latency, for data sovereignty, and for capacity. So, we asked the next question: if the same model is served from a dozen datacenters, which datacenter should serve each request?

Our answer: a KV-cache-aware global router — internally, the AI Grid — that turns a fleet of independent Dynamo deployments into one cache-coherent inference fabric. We’re contributing it back to the NVIDIA Dynamo open-source project.

The problem: routing is now a cache problem

Modern LLM inference is dominated by the KV cache — the attention state a model accumulates while reading a prompt. Reuse it and the first token is fast and cheap; miss it and the GPU recomputes work it already did. Dynamo exploits this within a cluster.

Across datacenters the trade-off gets sharper, and there’s no single right answer:

•     Route to the nearest datacenter and you might recompute a cache another region already holds.

•     Route purely by cache and you might cross an ocean to save a few milliseconds of latency.

•     Route blindly and, under load, a single region saturates while others sit idle.

At scale, these small per-request mistakes compound into real latency and cost. We needed a router that weighs cache locality, network distance, and live load together — globally.

The idea: lift Dynamo’s own algorithm one level up

The elegant part is that we didn’t invent a new routing algorithm. Dynamo already answers “which worker has the best cached prefix?” with a radix-tree match over the KV cache. Our global router reuses that exact match — but treats each datacenter as one synthetic “worker.”

Around that core, every routing decision blends three signals through a composable filter → score → tiebreak engine:

The router keeps a request local when local is best, and spreads it across the grid when a region starts to saturate — without throwing away a warm cache it could have reused.

Making it work over a real WAN

A global cache index sounds expensive. Three design choices keep it cheap and fast:

1. A cuckoo filter, not a full mirror. Mirroring every datacenter’s exact cache across the planet would mean moving tens to hundreds of gigabytes per region. Instead, each datacenter publishes a cuckoo-filter of its cache at ~3–4 bytes per block. 10 million cache blocks → ~32 MiB. Small enough to stream continuously and snapshot for fast recovery. (The router also supports an exact radix-tree backend where memory isn’t a concern.)

2. One mTLS gRPC connection per datacenter. Each datacenter runs a lightweight KV event relay that streams cache events and metrics to the router over a single mutually-authenticated gRPC connection, HTTP/2-multiplexed. We hash model IDs, imply the datacenter from the connection itself, ship incremental deltas, and stamp per-stream identity only once — so the control plane stays lean at hundreds of millions of blocks. Built-in replay + snapshot RPCs close the gap when a router restarts.

Here’s the shape of it:

Figure 1 — Global router architecture: one mTLS gRPC relay per Dynamo site, a cache index, and cache × latency × load routing across 13+ datacenters.

What we measured: 13 datacenters, 4–239 ms, 130 M cache blocks

We validated the grid across 13 real, geographically distributed Gcore datacenters — from Frankfurt (4 ms) to Incheon (239 ms RTT) — against a single global router, over genuine wide-area links (no simulated latency), at roughly 130 million KV-cache blocks across the fabric. We drove it with GPU-free Dynamo serving emulators so we could stress the routing layer at scale without burning GPUs.

Cache sync is distance-independent. A datacenter 239 ms away ingested cache updates at the same ~2,800 blocks/s as one 4 ms away. The relay→router stream is pipelined and throughput-bound, not latency-bound — so a remote region’s view of the cache stays as fresh as a local one.

The whole grid recovers in seconds. After a cold router-replica restart, all 13 datacenters — ~130 M blocks — fully re-synchronized in about 11 seconds (≈ 4.3 s + 0.030 × RTT). That’s the cuckoo filter paying off: a single datacenter’s index is ~32 MiB to move, versus the ~100–200 GB an exact tree would need — three to four orders of magnitude less over the same WAN.

Routing across the grid wins under load. We found a clean crossover:

At light load, local wins — there’s no queue to escape, so paying for a WAN hop makes no sense, and the router keeps traffic local. Once a single datacenter saturates (~77 req/s here), spreading across the grid delivers +48% throughput with lower tail latency.

This is an AI Grid — with a global router

At Gcore we call it the AI Grid internally, and it turns out that’s exactly the right name: AI Grid for distributed inference. NVIDIA defines it as infrastructure that “logically unifies geographically distributed sites into a single, coherent AI fabric,” run by a Control Plane that “intelligently places workloads across the grid” with prompt awareness — and it names NVIDIA Dynamo as the serving layer.

Map that onto what we built:

So this isn’t a router that resembles an AI Grid — it’s the cache-aware, prompt-aware placement engine an AI Grid needs, built on the serving layer the reference design already names, and proven on a real multisite fabric.

How this fits Everywhere AI

Everywhere AI is about running AI anywhere — on-prem, cloud, hybrid, edge — with control over performance, cost, and compliance. The global router is the routing engine that makes the “Everywhere” literal: a request finds the GPU that’s already warm for it, in the region that makes sense, across a sovereign multi-datacenter footprint. Combined with Dynamo’s in-cluster gains and our distribution reach, it lets enterprises treat a distributed GPU estate as one inference system instead of many islands.

What’s next

We’re contributing the global router upstream to NVIDIA Dynamo so the community can build globally distributed inference on a shared foundation, and we’re hardening it from internal validation toward production availability inside Gcore Everywhere AI. If you’re running inference across more than one region, this is the layer that ties it together — talk to us.