Google Open-Sources XSE, the Serving Engine Behind Six Million Queries a Second

Ask an ML engineer where the time and money actually go, and you will rarely hear “the model.” Training gets the glory. Serving, the unglamorous work of answering millions of small requests every second, is where the bills pile up. Google has now open-sourced one of its sharpest answers to that problem: XSE, the XLA Serving Engine, a runtime that takes the Python interpreter out of the serving path completely.

The announcement landed on the Google Open Source Blog on November 5, written by Oleg Shaldybin and Monica Song from Google’s TPU software team. XSE is a general-purpose model export and inference runtime for the OpenXLA ecosystem, released under the Apache 2.0 license. And this is no research toy. Inside Google, XSE already serves over six million queries per second, powering inference pipelines for Search, Google Ads, YouTube Shorts and parts of Gemini Omni.

Train in Python, Serve in C++

Google sums up the workflow in a single line: “Train in Python, Serve in C++.” Engineers keep writing model code the way they already do, in Python, usually with JAX handling the maths. When a model is ready, XSE’s export API packages it as an “XLA SavedModel” holding the model’s function in StableHLO, an intermediate representation the XLA compiler can work with. The XLA Serving Runtime then loads that package and executes it as compiled native code, with no interpreter sitting between the request and the answer.

Why does cutting Python out matter so much? Because at Google’s request volumes, the interpreter itself becomes overhead: extra memory, extra CPU, latency that wobbles when you least want it to. A compiled C++ runtime carries none of that baggage. Google says it began deploying core components of XSE inside its production serving infrastructure last year, which makes this release less of a debut and more of a handover. Google is giving away a tool it already trusts with some of the highest-volume inference pipelines on the internet.

The code and quick-start instructions live in the OpenXLA project’s xla-serving repository on GitHub, and the runtime targets CPUs, GPUs and TPUs. PyTorch users are not left out for long, either. Google plans integration with TorchTPU, so models written in PyTorch can be exported for C++ serving on TPUs as well.

The Benchmarks, With the Usual Caveat

So how fast is it? In Google’s own tests on short requests, XSE delivered two to five times the throughput of the best optimized Python server the company could build, while using a quarter of the memory. Read those numbers with the standard caution. They are vendor benchmarks, run on Google’s workloads, under Google’s conditions, and nobody outside Google has replicated them yet. Still, the direction of travel is hard to argue with. Removing an interpreter from a hot path almost always buys speed, and it usually buys memory headroom too.

Nathan Beach, a Google director of product management, described the target workloads as short requests with very high queries per second and tight latency budgets. Think recommendation rankings refreshed on every swipe. Think ad scoring decided in milliseconds. Think vision models checking images as fast as users can upload them. These are not glamorous jobs, but they never stop, and they never get to be slow.

Not a Chatbot Engine, and That Is Fine

Here is the part some headlines will skip: XSE is not built for large-scale LLM inference. If you are serving a giant language model, Google itself points you towards vLLM or SGLang instead. XSE is a lightweight, one-shot inference engine aimed at the other end of the spectrum: smaller models, sharply defined tasks, one prediction per request, relentless volume.

How much of the world’s AI traffic actually looks like that? Most of it, honestly. Every Shorts swipe, every ad auction, every search ranking call is a small model answering a small question under a brutal deadline. The chatbots get the keynotes, but this quiet layer is where AI turns into infrastructure, and I find a production-hardened engine for it more interesting than another demo. If your team serves compact models at painful scale, XSE was designed with your exact headache in mind.

TPU Sync Arrived in the Same Week

XSE was not the only piece of Google’s serving stack to go open this month. Google Cloud has also open-sourced TPU Sync, a native C++ library with a very different job: moving model state around a TPU cluster. It shuttles KV cache blocks and model weights between TPU high-bandwidth memory, host DRAM, shared storage and the data centre network, treating all four as a single memory hierarchy so accelerators spend less time sitting idle, waiting for data to arrive.

According to Google, TPU Sync cuts KV transfer latency by over 90 percent and reduces time to first token by an average of 90 percent for heavyweight open-weight MoE models, in the 235B to 480B parameter range, in prefix-caching and disaggregated-serving workloads. It also streams updated weights from trainers to samplers in reinforcement-learning pipelines, and it plugs into vLLM and SGLang as a TPU KV cache connector. Those figures are Google’s own claims from its own testing, not independently verified results, so treat them as a claim to test rather than a fact to quote. The timing is not subtle, either. The library landed alongside the new hardware we covered in our look at Google’s TPU 8tx and TPU 8i Max, and the pattern across both releases is easy to read: Google is opening the software layer around its TPUs piece by piece, betting that developers who build on OpenXLA tooling today become TPU customers tomorrow.

Will the bet pay off? Too early to say. But the code is public, the license is permissive, and the production pedigree is unusually concrete for an open-source launch. Six million queries a second is not a slide in a keynote. It is a Tuesday at Google, and now the engine behind it belongs to anyone who wants to read it, run it, or argue with it.