Scope & boundaries
This guide covers the platforms that serve models to live traffic, organized by the unit each one meters — tokens, GPU-seconds, host-hours, output artifacts, or reserved capacity that bills whether you use it or not.
It does not cover which foundation model to license, and on whose terms (Generative AI & LLM Platforms), or the lifecycle around a model once it is running — versioning, drift, retraining (MLOps Platforms).
Executive Summary
Every platform here will serve your model. What separates them is the unit they count, and whether that unit moves with your traffic or with your infrastructure.
The decision in this category is not which platform serves models fastest. It is which one bills on a unit that matches how your traffic actually behaves — because the same workload can cost wildly different amounts depending on whether the meter runs on tokens, GPU-seconds, host-hours, output images, or reserved capacity that bills whether you use it or not.
That reframing matters because the demo never surfaces it. A proof of concept with bursty, low-volume traffic makes serverless token pricing look obvious. The same architecture at steady production volume is where teams discover they are paying retail for capacity they could have reserved, or reserving capacity they use for six hours a day. This guide is organized around that question first and the vendor list second.
Why Inference Serving Is a Line Item Now, Not an Engineering Detail
Inference became a procurement question the moment it stopped being a pilot. While a model is being evaluated, serving it is a developer's afternoon and the cost is noise. Once it is answering customer traffic, the serving layer is a recurring bill with no natural ceiling, owned by whoever signs for cloud spend, and it grows with adoption rather than with headcount.
The pressure is one-directional. Model providers keep releasing capable open-weight models, which makes self-serving viable for workloads that previously had to go to a proprietary API; at the same time the managed platforms keep absorbing the operational work that made self-serving expensive. Both trends push the same decision earlier in the year than most infrastructure teams planned for, and both make a two-year commitment harder to justify than it looks.
There is a second reason this lands on a procurement desk rather than an engineering one. The serving layer is where data residency actually gets decided. A model can be open-weight and a provider can be contractually clean, and the workload still runs wherever the platform's capacity happens to be unless someone specified otherwise. Anyscale lists bring-your-own-cloud among its plan features precisely because that constraint disqualifies most of the market for some buyers — and it is far cheaper to establish before a shortlist than after a pilot.
Which type of AI Model Serving & Inference Platforms fits your organization?
Self-hosting is a real option here in a way it is not in most software categories, because the serving engines are open source and good. vLLM describes itself as delivering “state-of-the-art serving throughput” and is the layer several commercial platforms are built on, so the question is not whether you can run it — you can — but what the managed platforms are actually selling on top of it.
The honest framing of build-versus-buy here is a utilization question, not a capability one. Serving software is free and good; GPUs are neither. A team that can keep hardware busy most of the day will beat retail token pricing, and a team that cannot will pay for idle silicon and an on-call rota to watch it idle. Everything else — autoscaling, batching, routing, upgrades — is work the managed platforms have already done, and the value of that work scales inversely with how much of it you were going to do anyway.
| Approach | What you are buying | What it costs you instead |
|---|---|---|
| Serverless token API | Someone else's capacity, metered by output. No floor, no idle cost, no capacity planning. | Retail rates at steady volume, and a cold start you do not control. |
| Managed dedicated capacity | An endpoint of your own on the vendor's hardware, sized by you. | Paying for the hours the endpoint is warm and idle. |
| Bring-your-own-cloud | The platform's control plane running inside your account, on your committed spend. | Your team owns the cloud relationship, the quota fights, and the region strategy. |
| Self-hosted open engine | Full control of the runtime, the hardware and the upgrade cadence. | The on-call rota, the autoscaler, the observability, and the GPU procurement. |
| Custom silicon | Latency characteristics you cannot get from commodity GPUs. | A narrower model catalog and a supplier with no second source. |
How do you evaluate AI Model Serving & Inference Platforms?
Score a serving platform on what it does when things go wrong, because what it does when things go right is the same everywhere. Every vendor in this category will return a token stream from a popular open-weight model at a reasonable rate. The separation is in cold starts, autoscaling behavior under a spike, what happens to a request mid-generation when an instance is recycled, and whether any of it is visible to you.
One capability deserves separating from the rest, because it changes the arithmetic rather than the experience. Prompt caching reuses computation on repeated prefixes, and DeepInfra publishes a distinctly lower rate for cached input tokens. For a retrieval-augmented application with a long, stable system prompt, that is not a marginal optimization — it is the difference between paying for the same instructions on every request and paying for them once. Whether it applies to you is a property of your prompts, so it is worth measuring before it is worth negotiating.
| Capability | What it does | Buyer translation |
|---|---|---|
| Metering basis | Defines the unit your bill is counted in | The single largest determinant of what you pay. Fireworks AI runs both a per-token and a per-GPU-second basis and describes the second as cheaper “at scale.” |
| Cold-start behavior | How long the first request waits when no instance is warm | Replicate sells “always-on instances” specifically to “eliminate cold start delays” — which prices the problem rather than solving it. |
| Prompt caching | Reuses computation on repeated prefixes | Changes the arithmetic on RAG and long system prompts. DeepInfra publishes a separate, lower rate for cached input tokens. |
| Dedicated endpoints | Isolated capacity rather than a shared pool | Removes noisy-neighbor latency variance and usually removes the per-token discount along with it. |
| Deployment locus | Whether the workload runs in the vendor's account or yours | Anyscale lists bring-your-own-cloud among its plans; most serverless APIs do not offer it at any price. |
| Model catalog breadth | Which weights you can actually run | Constrains you later, not now. Narrow catalogs are the hidden cost of custom silicon. |
| Output-unit billing | Charging per artifact rather than per token | fal bills some media models “by output unit — per second or per video — depending on the model,” which is a different budgeting exercise entirely. |
Which vendors lead in AI Model Serving & Inference Platforms?
The market divides by billing unit, and the camps below are that division. A shortlist that spans two of them is comparing different products, which is fine as long as everyone in the room knows it — and is the usual reason two quotes for “the same thing” differ by an order of magnitude.
Within a camp, four vectors separate vendors, and only one of them shows up in a benchmark. Metering granularity — whether you are billed by the token, the second, the hour or the artifact, and whether idle counts. Escape hatches — whether you can move from shared to dedicated capacity, or into your own cloud, without rewriting the integration. Failure behavior — what happens to an in-flight request when an instance dies, and whether you are told. Catalog breadth — which weights you can run today, which constrains what you can adopt next year rather than what you can ship this quarter.
The honest broker's note on this category is that it is young and consolidating, and nobody's roadmap survives that. Platforms in this market are still being acquired, still repricing, and still discovering which billing basis they actually want to sell; a rate card here has a shorter half-life than a procurement cycle. That argues for two things and against a third. Keep the integration thin enough that moving is a configuration change rather than a project. Negotiate term length down rather than rate down, because the rate will move on its own. And treat any three-year commitment in this category with more suspicion than the discount seems to justify.
| Vendor | Approach | Where it fits |
|---|---|---|
| Anyscale | Bring-your-own-cloud | Organizations that need the workload to stay inside their own cloud account |
| Cerebras | Custom silicon | Interactive products where latency is the thing being bought |
| Baseten | Managed dedicated capacity | Workloads graduating from a shared API to capacity of their own |
| fal | Output-unit platforms | Image and video generation, where the artifact is the billable unit |
| vLLM | Self-hosted engines | Teams with GPUs and the operations practice to keep them busy |
| Fireworks AI | Serverless token APIs | Teams that want both billing bases from one vendor without moving platforms |
One representative of each approach is named here; the category runs to roughly two dozen platforms, and several appear in more than one camp. The camps were written before the vendors were chosen, and no placement here is for sale. Any vendor in this category can speak for themselves in the Spotlight below.
How much should you budget for AI Model Serving & Inference Platforms?
There is no useful single number for what inference costs, so this section does not offer one. What is knowable is how each camp counts, what each one charges for when nothing is happening, and which variables move your bill — and those are enough to model your own figure before you talk to anyone.
The costs that surprise people are not on the rate card. Egress and storage attach to whichever cloud the workload runs in and are invisible until the first full month. Idle capacity is a decision rather than an accident, but it is made by whoever configured the autoscaler rather than whoever signed the contract. Fine-tuning is banded by model size rather than priced per job — Fireworks AI publishes its fine-tuning rates by parameter count — so an experiment that moves up a band costs more than the previous one by a step rather than a slope. And the engineering time to operate any of this is real whether or not it appears in the software budget; it is simply larger in the self-hosted column.
| Metering basis | You are charged for | Idle cost | Where it goes wrong |
|---|---|---|---|
| Per token | Input and output tokens, sometimes at different rates | None | Steady high volume, where retail rates never amortize |
| Cached input token | Repeated prefixes, at a reduced rate | None | Nothing — but only helps if your prompts actually repeat |
| Per GPU-second | Wall-clock time your model occupies a device | None while scaled to zero | Chatty low-volume traffic that keeps waking instances |
| Per host-hour | The instance, warm or not | Full rate | Workloads with long idle stretches |
| Provisioned throughput | Reserved capacity in fixed units | Full rate | Volume you guessed high on, committed a year ahead |
| Per output unit | An image, a second of video, an artifact | None | Nothing, if your product's unit matches the vendor's |
| Self-hosted | GPUs, whether or not they serve a request | Full rate, plus the team | Utilization below roughly the point where the team costs more than the savings |
| Fine-tuning | Training tokens, banded by model size | None | Iterating on a large model, where the band steps up sharply |
Every figure above is a published rate read from the vendor's own pricing page on the date in the sources note. None of it is an estimate of what you will pay.
How long does implementation take for AI Model Serving & Inference Platforms?
Sequence this around measurement, because the only thing that settles the camp question is your own traffic. The phases below front-load the cheap, reversible work and put the commitment last.
Two things are worth doing before week one. Establish whether a residency or in-account constraint applies, because it removes most of the field and everyone would rather know that before building a shortlist. And capture a week of real traffic including the quiet hours — not a synthetic load test, which is generated by someone who already believes the workload is busy.
Capture a representative week of production traffic and replay it against two candidates from different camps. Record the invoice, not the benchmark. Include the quiet hours: they are where per-host and reserved pricing lose.
Deliberately break each candidate. Kill an instance mid-generation, drive a cold start with a user watching, saturate the endpoint past its rate limit. What the platform does here is what you are actually buying, and it is invisible in a demo.
Move one workload, not the portfolio. Keep the abstraction thin enough that the second workload can go somewhere else — most teams end up multi-camp, and the ones who planned for it pay less than the ones who did not.
Put a quarterly review in the calendar with the utilization data attached. The pricing in this category moves faster than annual budget cycles, and the camp that was right at launch volume is frequently wrong at ten times that.
A serving platform is infrastructure: it transports and executes a model rather than deciding anything about a person. The obligations that attach to it are the ones that follow your data — where inference runs, what is retained, and whether prompts are logged — not the ones that attach to the model's outputs. Those sit with the application, and with the model provider upstream.
Classified under the EU AI Act's risk tiers, as they apply to a provider of infrastructure rather than of an AI system placed on the market
What should you ask vendors about AI Model Serving & Inference Platforms?
Take these to the vendor call. Most of them have a right answer the vendor already knows, and the ones that do not are the ones worth the meeting.
-
Does your traffic have a floor you could reserve against?Yes Dedicated capacity or self-hosting. You are paying retail for a load you could have committed.No Serverless token API. Do not buy capacity for traffic that is not there.
-
Does a cold start reach a person?Yes You need always-on capacity, which means paying for idle. Price that in rather than hoping autoscaling hides it.No Scale to zero and take the first-request delay. This is most batch and pipeline work.
-
Must the workload run inside your own cloud account?Yes Bring-your-own-cloud, or self-hosted. This constraint removes most of the market, so establish it before you shortlist.No The full field is open, and the metering basis decides.