Welcome to the second half of the track. Everything up to now has been about producing a good language model. From here on, we’re about running one in the real world. And running a language model, it turns out, is different enough from running a traditional machine learning model that it deserves its own name and its own habits. That world is called LLMOps, and this article is about what actually makes it different from the older, well-established world of MLOps.
If you’ve never heard of MLOps, that’s fine. I’ll frame everything so it makes sense without prior context. If you already know MLOps well, you’ll recognise most of the shape and see where the wrinkles show up. Either way, the goal is the same: to leave you with a clear picture of what changes when the model you’re operating is a language model, and why those changes matter.
The one-sentence version
The bumper-sticker version, before we dig in. MLOps is the discipline of running traditional machine learning models (fraud detectors, recommenders, forecasters) reliably in production. LLMOps is that same discipline, adapted for language models, which turn out to behave differently enough that some of the old habits break and some new ones become essential. Same instincts, updated for a new kind of system.
What stays the same
Let me name what carries over cleanly, because a lot does. If you’re already running traditional ML systems well, you’re not starting from scratch. The core disciplines are the same:
- Version everything. Models, prompts, datasets, configs, and the code around them. When something goes wrong, “which version was live?” needs to be a single command away.
- Test before deploying. Every change gets run against a stable test set (the evaluation habit from the last article) and only ships if it improves things.
- Monitor in production. Something is watching quality, cost, and latency, so you notice problems before your users do.
- Handle failures gracefully. Systems have retries, fallbacks, and clear error paths so a hiccup in one component doesn’t take down the whole thing.
- Automate the boring bits. Deployments, tests, and monitoring aren’t done by hand every time.
If you take those five habits from traditional MLOps and drop them into an LLM system, you’re already miles ahead of a team without them. The question is what those habits look like when the model is a language model, and where they need new tricks. Figure 1 gathers the shifts in one place.
Figure 1: The core operational disciplines are shared, but LLMs bring new realities: prompts are code, outputs are unpredictable and hard to grade, costs are per-token, models can be updated by their provider, and the biggest risks (like hallucination and injection) are different in kind.
What actually changes
Now the interesting bit. Here are the shifts LLMOps has to reckon with that traditional MLOps mostly didn’t.
Prompts are first-class artefacts. In traditional ML, the “model” and the “code around it” were the two things you versioned. In LLM systems, there’s a third thing that quietly determines a huge amount of behaviour: the prompt (system prompt, templates, RAG framing). A prompt change is effectively a behaviour change, so it needs versioning and testing like a model change. This is why the Prompt Engineering track’s “prompts as code” theme keeps coming back.
Outputs are open-ended. A fraud detector returns “fraud” or “not fraud.” A recommender returns a ranked list. An LLM returns free-form text, which is harder to measure automatically. “Was this response good?” isn’t a binary check; it’s a subtle judgement involving quality, tone, faithfulness, and format. That makes evaluation harder and means it needs the habits we’ve been building (test sets, rubrics, model-as-judge for scale, occasional human review).
Everything costs by the token. Traditional ML models cost roughly a fixed amount per prediction. LLMs cost by how much text goes in and out. A long input is more expensive than a short one. A verbose response is more expensive than a terse one. Cost per query becomes a real operational metric, not an afterthought.
Behaviour can change under you. If you use a hosted model, the provider can update it. A model that behaved a certain way last month may behave subtly differently this month, even though you didn’t change anything. This is an LLMOps concern that traditional MLOps mostly didn’t face, and it means your monitoring has to catch drift you didn’t cause.
Risks are different in kind. The failure modes are new. Hallucination (confidently wrong output). Prompt injection (malicious instructions in inputs). Data leakage (the model repeating things it shouldn’t). Sensitive content handling. These aren’t traditional model risks and they need their own guardrails.
Latency is user-visible in a new way. LLMs are slower than most traditional ML models, and they generate output token by token. Which means “how it feels” to a user depends on things like streaming and perceived speed, not just backend latency.
Read those six shifts and you can see the pattern: nothing about the goals of MLOps changed. What changed is that the specific tactics need updating for a new kind of system. If you’re running language models in production, you’ll want habits that address each of the six.
Cutting through the marketing
A quick honest note. “LLMOps” as a term is popular, and there are plenty of tools sold under that banner. Some of them are useful; some are traditional monitoring and testing tools with an LLM sticker on them. The way to cut through the noise is to keep coming back to the actual shifts above. Does the tool help you version and test prompts? Evaluate open-ended outputs? Track cost per query? Detect drift, hallucination, or injection? Those are the real jobs. Any tool that helps with those, from either the old MLOps world or the newer LLMOps world, is useful. Any tool that just re-brands existing capability without addressing the new realities probably isn’t buying you much.
How the pieces fit together in practice
Let me sketch the shape of a well-run LLM system in production, so you can see how the concerns tie together. Figure 2 shows it.
Figure 2: A production LLM system knits together familiar MLOps pieces (deployment, testing, monitoring, versioning) with LLM-specific ones (prompt management, cost tracking, hallucination and injection guardrails, sample-and-review of open-ended outputs). The rest of this track walks through each of these pieces.
There’s the model itself, deployed and served (article 9 in this track, coming next). There are the prompts and templates, versioned alongside code. Every change runs through an evaluation pipeline against a stable test set (article 11). Production traffic is monitored for cost, latency, quality, and drift (article 10). Guardrails filter obvious problems (article 12). And a lightweight ops layer keeps overall cost under control (article 13). Every one of those is a concrete piece we’ll build up.
You don’t need all of it on day one. A serious system can start with just deployment, basic monitoring, and a small evaluation set, and grow from there. The point of listing them together is to give you the shape of what mature looks like, so the pieces you add later slot in without surprise.
The mindset carried forward
Here’s the summary I’d want you to leave with. If you’ve internalised the RAG and Prompt Engineering tracks, you already know a lot of what LLMOps cares about, because those tracks kept nudging you toward the same discipline: test on real cases, ground in real sources, watch cost, log what happened, iterate deliberately. LLMOps is what happens when you make those habits explicit across a whole system and formal across a team. Not more magic. More discipline.
Where we go from here in this track
The rest of the LLMOps half of this track walks through the concrete pieces above, one at a time. Next up: how a language model actually gets served in production, because that’s the piece that turns a fine-tuned checkpoint or an API key into something users can talk to.
Next in the series: Deploying & Serving LLMs, the step where models start creating value.