AI economics will be decided in transit

Santosh ErramVice President, Global AI Sales and NVIDIA 360 Partnership | F5

There has been a lot of talk lately—in boardrooms, on earnings calls, and across a steady stream of op-eds—about a single idea: the next major improvement in AI economics will not come from faster chips alone.

The argument is sound, but the lesson is broader. The first collapse in cost per token came from compounding advances across silicon and software: faster accelerators, better memory systems, lower-precision formats, PagedAttention, continuous batching, key-value cache optimization, and inference frameworks such as vLLM. The industry became dramatically better at extracting useful work from every accelerator.

Then the industry went looking for the next place to apply the same discipline, and it found the data path, especially storage. That work is essential, but it is also only half the story.

The data path is only half the story

Storage in an AI factory is no longer passive. Models load from it. Checkpoints restore from it. Retrieval-augmented generation depends on it under latency pressure. At sufficient scale, context and cache tiers may extend beyond GPU and system memory because recomputing work that was already completed is an expensive use of an accelerator.

But AI economics is shaped by two connected flows. One is the movement of the data that feeds the model: where bytes live, how quickly they can be fetched, and how often they must be moved. The other is the path an inference request travels—from a user, application, or agent, through traffic management and policy enforcement, into a model server, and back out.

That second flow carries data in transit. Compared with compute and storage, it remains under-measured and under-optimized in many production environments. This is where the next round of gains —and the next round of waste —will be decided.

The token tax

On a GPU node, the host central processing unit is not idle. It runs networking, terminates transport layer security, enforces policy, handles east-west traffic, generates telemetry, and moves data between interfaces and accelerators. None of that work directly produces tokens. When it becomes a bottleneck, it reduces useful inference output.

Call it the token tax: infrastructure overhead that consumes host resources and quietly taxes the economic output of the node. The answer is not always a faster host processor. At scale, eligible networking, encryption, traffic-management, and security functions can be moved onto programmable infrastructure already sitting in the path.

In F5-commissioned independent testing by The Tolly Group, F5 BIG-IP Next for Kubernetes deployed on NVIDIA BlueField-3 DPUs delivered up to 40% higher token throughput, 61% faster time to first token, and 34% lower request latency than the CPU-hosted alternatives tested.

The financial implication is straightforward. If a fixed infrastructure footprint produces 1.4 times as many sellable tokens within the required service-level objectives, infrastructure cost per token falls by approximately 29%. Revenue per accelerator can also rise where demand exists. That is a more useful business claim than simply saying the network is faster.

Routing became scheduling

Conventional load balancing distributes requests. That works when requests are roughly interchangeable. Large language model inference breaks that assumption because the cost of a request depends on context length, output length, cache state, model selection, queue depth, and current accelerator load.

• A short prompt with a brief completion can be orders of magnitude cheaper than a very long context with a long response.
• A request that reaches a warm key-value cache can avoid expensive prefill re-computation.
• Prefill-heavy and decode-heavy requests can interfere when they land on the same accelerator.
• A model server with eight queued requests and one with none may look equally healthy to a stateless check.

A traffic manager with no visibility into those conditions will make poor placement decisions by construction. The cost appears twice: in latency the user experiences as time to first token, and in recompute the business pays for as duplicated work.

Inference-aware scheduling uses queue depth, cache locality, GPU load, and model state to place each request on the best endpoint.

Routing in an AI factory is therefore no longer only a networking function. It is a scheduling decision that happens in the network. It benefits from the same inputs any scheduler needs: live model-server state, queue depth, accelerator telemetry, cache locality, and runtime signals from the inference framework.

Security is also a cost function

Security in an AI factory is usually discussed as risk: prompt injection, data leakage, agents exceeding their authority, and regulatory exposure. All are real. F5 research found that 88% of organizations report at least one AI-related operational or security challenge.

There is also a unit-economics argument. Requests blocked before model execution avoid unnecessary inference spend. Requests evaluated only after a full response is generated consume compute before policy takes effect. Multiply that by scraping traffic, retry storms, an agent stuck in a loop, or repeated jailbreak attempts, and enforcement placement starts to affect cost as well as risk.

The principle is not to move every control to one place. Some protections require deeper application- or model-level evaluation, and those controls carry their own compute cost. The goal is to apply the least expensive effective control as early as practical, then perform deeper inspection where it adds value.

Agentic traffic sharpens this problem. Agents retry, fan out, and call tools that call models that call other models. They generate machine-speed traffic that human-scale rate limits were not designed to absorb. The failure mode is not always a breach. Sometimes it is simply a bill.

The commercial layer is missing

If an operator cannot count tokens by tenant, model, user, and application, it becomes difficult to price, meter, charge back, enforce fairness, or defend a margin. The infrastructure may have excellent telemetry about power draw and health, while providing limited visibility into revenue attribution.

Multi-tenancy compounds the issue. Shared GPU infrastructure without effective isolation and per-tenant accounting means one tenant's burst can become another tenant's service-level violation. A noisy neighbor on an AI cluster is not merely a performance annoyance. It can become churn.

The inference path provides the most consistent cross-cutting vantage point because it can observe requests and responses across users, applications, models, and tenants. Model servers and billing systems still matter, but the path is where measurement and enforcement can be applied consistently.

The inference service plane

Put these issues together and a third architectural plane comes into focus. The compute plane is the GPUs and accelerators everyone budgets for. The data path covers storage, retrieval, caching, and memory. The inference service plane governs how requests reach accelerators, how they are scheduled against real system state, what is stopped before it consumes unnecessary compute, what is counted, and who is charged.

This layer already exists, but often as an accident of assembled parts: a load balancer here, an API gateway there, a service mesh, a web application firewall, telemetry collectors, and a spreadsheet at the end. It has rarely been designed as a coherent system with shared telemetry, consistent policy, and intentional enforcement points.

The inference service plane connects request routing, policy, security, metering, and observability to the data and compute layers.

F5 BIG-IP Next for Kubernetes can run on the host or on NVIDIA BlueField-3 DPUs. It provides Kubernetes-native traffic management, security, multi-tenant policy, observability, intelligent load balancing, and token governance across the inference path.

At the application layer, F5 AI Gateway adds model and agent governance, routing, caching, metering, security controls, and auditability across prompts, responses, models, and tools.

The architectural direction matters more than any single product: express policy clearly, enforce it where it is most effective, and share telemetry across infrastructure and application layers. That is how operators reduce blind spots and stop separate teams from maintaining divergent controls.

Design the path on purpose

Three questions belong in the next AI infrastructure review. What is the useful token yield of the accelerator capacity already provisioned? How much host processing is consumed by work that does not directly produce tokens? What does a rejected request cost when it reaches model execution before enforcement takes effect?

Across the industry, the conversation is shifting from capacity to conversion—from what infrastructure is owned to what can be reliably extracted from it. The next major improvement in AI economics will not come from faster chips alone or storage alone. It will come from treating the inference path as an economic system.

The compute plane is bought. The data path is being optimized. The inference service plane is still up for grabs.

To learn more, read the AI performance report.

About the Author

Santosh ErramVice President, Global AI Sales and NVIDIA 360 Partnership | F5

More blogs by Santosh Erram