Fine-Tuning Agentic AI: A Complete Practical Guide

Most teams that fine-tune an agent do one thing really well and then wonder why the agent still falls apart in production. Someone spends two weeks getting a LoRA adapter trained, the loss curve looks great, the demo goes well in the team meeting, and then three days after launch the agent is calling the wrong function, retrying forever, or confidently doing something nobody approved. Nothing about the training run was wrong. The problem is that fine-tuning an agent is not one job, it's at least six, and most guides only walk you through one or two of them.

This guide covers all of it: building a dataset that actually matches how your agent behaves at runtime, picking the right training method and framework for your hardware, tuning what happens after training (which matters just as much as the training itself), teaching the model judgment calls that a single correct answer can't express, getting the model out of your notebook and into something you can actually serve, evaluating it properly, and closing the loop so the agent keeps improving after launch instead of quietly rotting.

We'll use one running example throughout: an internal SRE agent that responds to service alerts. It has three tools available: check_service_health, restart_service, and page_oncall_engineer. Simple on paper. In practice, this is exactly the kind of agent that's easy to get 80% right and genuinely hard to get 99% right, which makes it a good example for showing where things actually go wrong.

What you'll need: Python 3.10 or newer, and pip install peft transformers datasets trl accelerate bitsandbytes. A CUDA GPU is required for anything that actually loads and trains a model. The dataset construction, evaluation, and monitoring code will run fine on a laptop.

Do You Actually Need To Fine-Tune This Agent?

Before writing a single line of training code, it's worth being honest about whether fine-tuning is solving the right problem. A lot of agent teams reach for fine-tuning when what they actually have is a prompting problem or a retrieval problem, and no amount of GPU time fixes those.

Here's a rough way to think about it:

If your problem is missing knowledge, fine-tuning is close to the worst tool for the job. A model trained on data from six months ago will not magically know about a product that launched last week just because you fine-tuned it on unrelated examples. That's what retrieval is for. Fine-tuning earns its keep when the issue is behavioral: the model knows enough to do the task, but it won't reliably format its output, pick the right tool, or apply your specific business judgment without being shown examples of exactly that.

Cost is also part of this decision, and it's less scary than people assume. A QLoRA run on a 7B to 8B model, with a few hundred to a couple thousand well-built examples, typically finishes in a few hours on a single rented GPU. You're not looking at the kind of budget that requires a committee approval. What actually costs time is building a dataset that's worth training on, which is where most of this guide's attention goes.

Step 1: Build A Dataset That Looks Like Real Agent Behavior

This is the step almost everyone underinvests in, and it's also the one with the highest payoff per hour spent. The mistake most tutorials make is showing you a dataset of single, isolated tool calls: one user message, one perfect function call, done. Real agents don't work in single turns. They check something, look at the result, and decide what to do next, sometimes across four or five steps before they're finished.

If you train only on single-shot examples, you're teaching the model a skill it will never get to use in isolation, because in production the model has to decide what to do after seeing a tool's output too, and that's a different skill than picking the first tool.

Here's what a proper multi-turn trajectory looks like for our SRE agent, including a step where the agent has to react to what a tool actually returned instead of guessing blind:

# build_trajectories.py
import json

TOOLS = [
    {
        "name": "check_service_health",
        "description": "Returns current health status and error rate for a named service.",
        "parameters": {"type": "object", "properties": {"service": {"type": "string"}}, "required": ["service"]},
    },
    {
        "name": "restart_service",
        "description": "Restarts a service. Only call this after confirming the service is actually unhealthy.",
        "parameters": {"type": "object", "properties": {"service": {"type": "string"}}, "required": ["service"]},
    },
    {
        "name": "page_oncall_engineer",
        "description": "Pages a human. Use this for anything that a restart won't fix, or anything risky.",
        "parameters": {"type": "object", "properties": {"summary": {"type": "string"}}, "required": ["summary"]},
    },
]

def build_trajectory(alert_text, steps):
    """
    steps is a list of dicts, each one either a tool call the agent made
    or the observation that came back from that tool. This mirrors the
    actual message history the model will see at inference time.
    """
    messages = [
        {"role": "system", "content": "You are an SRE agent. Investigate before you act, and page a human for anything a restart can't safely fix."},
        {"role": "user", "content": alert_text},
    ]
    for step in steps:
        if step["type"] == "tool_call":
            messages.append({
                "role": "assistant",
                "content": None,
                "tool_calls": [{"type": "function", "function": {"name": step["tool"], "arguments": json.dumps(step["args"])}}],
            })
        elif step["type"] == "observation":
            messages.append({"role": "tool", "name": step["tool"], "content": json.dumps(step["result"])})
    return {"messages": messages}

# A trajectory that shows the correct behavior: check first, restart only if unhealthy.
good_trajectory = build_trajectory(
    alert_text="Alert: checkout-service latency spiking above 2s.",
    steps=[
        {"type": "tool_call", "tool": "check_service_health", "args": {"service": "checkout-service"}},
        {"type": "observation", "tool": "check_service_health", "result": {"status": "degraded", "error_rate": 0.12}},
        {"type": "tool_call", "tool": "restart_service", "args": {"service": "checkout-service"}},
    ],
)

# A trajectory that shows the agent correctly refusing to restart, because the
# health check came back clean, meaning the alert is probably a red herring.
escalation_trajectory = build_trajectory(
    alert_text="Alert: checkout-service latency spiking above 2s.",
    steps=[
        {"type": "tool_call", "tool": "check_service_health", "args": {"service": "checkout-service"}},
        {"type": "observation", "tool": "check_service_health", "result": {"status": "healthy", "error_rate": 0.01}},
        {"type": "tool_call", "tool": "page_oncall_engineer", "args": {"summary": "Latency alert fired but health check is clean, likely a monitoring or upstream network issue"}},
    ],
)

Notice the second trajectory. This is the piece that separates a decent tool-calling dataset from a great one: examples where the "obvious" action is wrong, and the correct move is to gather more information or hand off entirely. If every example in your dataset shows the agent taking the action the alert seems to suggest, you're training a model that pattern-matches alert text to actions instead of one that actually investigates.

A few things worth checking before this dataset ever touches a trainer:

• Every tool name and argument matches your real tool schema exactly, including casing.
• At least 15 to 20% of your trajectories show the "don't act yet, gather more data" pattern.
• At least a few trajectories show the agent correctly refusing to escalate for something trivial, so it doesn't learn to page a human for every alert.

For scaling past what you can write by hand, generate synthetic trajectories with a strong model and then filter them with a second pass: have a judge model (or a rule-based checker, which is cheaper and more reliable for structured tool calls) score each trajectory on whether the tool sequence is valid and whether the final action matches the ground truth. Throw out anything that fails validation before it reaches your trainer. Building the filter takes an afternoon. Debugging a model that learned from a badly filtered dataset takes a lot longer.

Step 2: Pick A Training Method That Matches Your Hardware

Once the dataset is solid, the next decision is how you're actually going to update the model's weights. There are three realistic options, and the right one depends entirely on what hardware you have and how much you're willing to babysit the job.

For almost everyone reading this, QLoRA is the sane default. It quantizes the frozen base model to 4-bit precision and trains small low-rank adapter matrices on top, which is the only reason a 7B or 8B model fits comfortably on a single consumer or mid-tier cloud GPU instead of needing a multi-GPU rig.

Where the source you pick matters more than people expect is the training framework sitting on top of PyTorch and PEFT. Three real options dominate right now:

• Unsloth is the fastest and lightest option for single-GPU work. Its custom kernels cut training time by roughly half and VRAM use by up to 70% on supported model families, which matters a lot if you're on one GPU and want fast iteration.
• Axolotl trades a bit of that speed for flexibility. It's YAML-configured, supports full fine-tuning, LoRA, QLoRA, and preference methods like DPO and KTO out of the box, and scales cleanly to multi-GPU setups when you outgrow a single card.
• LLaMA-Factory leans into breadth and ease of use, with a web UI and support for well over a hundred model families, which is handy if your team wants to experiment across several base models without rewriting config files each time.

A reasonable path is to prototype with Unsloth because it's fast to iterate on, then move to Axolotl once you need multi-GPU training or a reproducible YAML pipeline for a production retraining loop.

Here's a QLoRA setup using Unsloth's loader, which handles the 4-bit quantization and adapter wiring in far fewer lines than wiring PEFT and bitsandbytes by hand:

# train_setup.py
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-instruct-bnb-4bit",
    max_seq_length=4096,   # give the agent enough room for multi-turn trajectories
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=8,                    # rank, higher gives more capacity but risks overfitting on small datasets
    lora_alpha=16,
    lora_dropout=0.0,       # Unsloth's fused kernels are optimized for dropout=0
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth",  # extra VRAM savings for longer trajectories
)

The bigger target_modules list compared to a minimal setup is intentional here. Agent trajectories are longer and structurally more varied than single-turn chat, so adapting the MLP projection layers alongside attention tends to give the model more room to learn the tool-selection logic, not just surface-level phrasing.

Step 3: Tune What Happens After Training

Here's the part that gets skipped constantly, and it's a mistake, because a well-trained model can still fail in production purely from bad decisions made at inference time, decisions that have nothing to do with the training run itself.

Temperature is the obvious one. Lower temperature makes tool selection more deterministic, which is usually what you want for the "which function do I call" decision, but you don't have to apply the same temperature to every part of the agent's output. A common pattern is to run tool selection at or near temperature 0, then allow a higher temperature for any free-text explanation the agent writes alongside the call.

The less obvious lever, and one that barely gets mentioned in most fine-tuning writeups, is constraining the shape of the output directly instead of hoping the model gets the JSON right on its own. This is usually called grammar-constrained or schema-constrained decoding, and tools like Outlines, Guidance, and vLLM's built-in guided decoding can force the model's output to match a JSON schema token by token, so it's structurally impossible for the model to emit a malformed tool call in the first place.

# schema_guard.py
from pydantic import BaseModel, ValidationError
import json

class RestartServiceCall(BaseModel):
    service: str

class PageOncallCall(BaseModel):
    summary: str

SCHEMAS = {"restart_service": RestartServiceCall, "page_oncall_engineer": PageOncallCall}

def validate_tool_call(tool_name: str, raw_arguments: str) -> dict:
    """Run this on every model output before it touches a real system.
    Even with constrained decoding upstream, treat this as your last line of defense."""
    if tool_name not in SCHEMAS:
        raise ValueError(f"Unknown tool: {tool_name}")
    try:
        parsed = json.loads(raw_arguments)
    except json.JSONDecodeError as e:
        raise ValueError(f"Malformed JSON from model: {e}")
    validated = SCHEMAS[tool_name](**parsed)
    return validated.model_dump()

Treat schema validation as a safety net you keep even after you've fine-tuned the model well and even if you're using constrained decoding at inference time. Models drift, prompts change, someone updates a tool schema without telling you. A five-line validation function that rejects a bad call before it reaches restart_service in production is cheap insurance against an expensive mistake.

The other runtime dial worth tuning deliberately is the iteration cap: how many tool calls the agent is allowed to make before it's forced to stop and either answer or escalate. Set this too low and the agent gives up on tasks that genuinely need three or four steps. Set it too high and a confused agent can loop for a very long time, burning tokens and, in our SRE example, potentially restarting the same service five times in a row. A retry-with-lower-temperature policy on a failed or invalid call, capped at one or two attempts, is usually a cheaper reliability win than anything you'd get from additional training.

Step 4: Teach Judgment With Preference Optimization

Supervised fine-tuning teaches the model "this is correct." It cannot teach "this is a valid, well-formed action, but it's not the right call given everything else going on," because SFT's loss function only ever sees one labeled right answer per example. That's exactly the gap in our escalation trajectory above: restart_service is a perfectly valid, correctly formatted call, it's simply the wrong judgment call when the health check already came back clean.

Direct Preference Optimization is the standard tool for this: instead of one correct label, you give the model pairs, a chosen response and a rejected one, both plausible, so the model learns the actual ranking rather than just what's technically allowed.

DPO isn't the only option anymore, and depending on the shape of your feedback data, it might not even be the best one:

• DPO needs paired data (chosen vs. rejected for the same prompt). It's the default and it works well when your preference labels are clean.
• KTO works on unpaired binary feedback, meaning a simple thumbs up or thumbs down per example with no need to pair them up. This matters a lot in practice, because production feedback logs (an engineer accepting or overriding an agent's action) naturally arrive unpaired, and reformatting them into pairs is often more work than it's worth.
• ORPO folds supervised fine-tuning and preference alignment into a single training stage, skipping the separate SFT step entirely. It's cheaper to run and a reasonable choice when you're tight on compute budget or iteration time.

For the SRE agent, a KTO-style dataset looks like this, and notice it doesn't require constructing an artificial "wrong" pair for every good example, which is often the hardest part of building a DPO dataset by hand:

# kto_examples.py
import json

def make_labeled_example(prompt_context, tool_call, is_desirable: bool):
    return {
        "prompt": prompt_context,
        "completion": json.dumps(tool_call),
        "label": is_desirable,  # True = good outcome, False = bad outcome, no pairing required
    }

LABELED_EXAMPLES = [
    make_labeled_example(
        prompt_context="Health check on payments-service returned status: healthy, error_rate: 0.02",
        tool_call={"tool": "restart_service", "arguments": {"service": "payments-service"}},
        is_desirable=False,  # restarting a healthy service is a real production incident waiting to happen
    ),
    make_labeled_example(
        prompt_context="Health check on payments-service returned status: healthy, error_rate: 0.02",
        tool_call={"tool": "page_oncall_engineer", "arguments": {"summary": "Alert fired but service reports healthy, needs human review"}},
        is_desirable=True,
    ),
]

There's also a newer angle worth knowing about if you're training for a task where success is automatically checkable, which tool calling often is: GRPO, the reinforcement learning method behind some of the recent reasoning-model gains, trains directly against a verifiable reward instead of a human or judge-model preference. If you can write a function that scores "did the agent pick the right tool and get a valid result," you technically have a reward signal you can train against without collecting preference labels at all. It's a heavier setup than DPO or KTO, but worth knowing it exists as the field moves past preference-only alignment for tasks with a checkable right answer.

Step 5: Merge, Quantize, And Actually Serve The Model

This is the step almost no fine-tuning guide covers, and it's the difference between a model that lives in a training notebook and one that answers real requests. A LoRA or QLoRA adapter isn't deployable by itself in most serving setups. You need to merge it back into the base model weights.

# merge_adapter.py
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

merged = PeftModel.from_pretrained(base_model, "./sre-agent-adapter")
merged = merged.merge_and_unload()  # folds the adapter weights into the base model permanently

merged.save_pretrained("./sre-agent-merged")
tokenizer.save_pretrained("./sre-agent-merged")

Once merged, you'll usually want to quantize again for serving, using something like AWQ or GGUF depending on whether you're deploying with vLLM, TGI, or a local runtime like llama.cpp. Two things trip people up here more than anything else:

1. Chat template mismatch. If the tokenizer's chat template at serving time doesn't exactly match the format your training data used, tool calling accuracy can quietly fall off a cliff even though the underlying weights are fine. Always save and ship the exact tokenizer config you trained against.
2. Tool-calling support in your serving engine. Not every inference server parses function-call syntax out of the box. Check that vLLM (or whatever you're using) is configured with the correct tool-call parser for your model family before assuming a broken response is a model problem.

Step 6: Evaluate It Like Production Depends On It, Because It Does

Two things have to be true before you ship: your agent has to actually get better at the task you trained it for, and it can't have quietly lost general capability in the process, a real and well-documented failure mode called catastrophic forgetting.

For agent-specific evaluation, don't reinvent this from scratch. Tau-bench (built by Sierra) is a solid reference point: it simulates realistic multi-turn conversations between a user, an agent, and a set of domain tools, then scores whether the agent completed the task while following the stated policies. Building something similar, even a scaled-down internal version, gives you a much more honest signal than checking single-turn tool-call accuracy in isolation.

The other piece worth building, and one most teams skip until after their first bad production incident, is a regression suite made entirely of real failures you've actually seen. Every time the agent messes up in production, that failure becomes a permanent test case.

# regression_suite.py
import pytest

REGRESSION_CASES = [
    {
        "alert": "Alert: checkout-service latency spiking above 2s.",
        "health_check_result": {"status": "healthy", "error_rate": 0.01},
        "expected_tool": "page_oncall_engineer",  # this exact case broke in production once
    },
]

def run_agent(alert, health_check_result):
    # stand-in for your actual agent inference call
    ...

@pytest.mark.parametrize("case", REGRESSION_CASES)
def test_no_regression(case):
    result = run_agent(case["alert"], case["health_check_result"])
    assert result["tool"] == case["expected_tool"], (
        f"Regression: expected {case['expected_tool']}, got {result['tool']}"
    )

This one habit, turning every production failure into a permanent test, does more for long-term agent reliability than almost anything else in this guide. It's boring. It's also the thing that actually keeps a fine-tune from regressing quietly six months from now.

Step 7: Close The Loop After Launch

Fine-tuning an agent once and calling it done is how agents slowly get worse over time without anyone noticing until a customer complains. The upstream APIs your tools call change. The distribution of real requests drifts from whatever your training set looked like. The fix isn't a bigger training run, it's a feedback loop.

Log every tool call decision the agent makes, along with enough context to reconstruct why. When a call fails validation, gets overridden by a human, or triggers an escalation that turns out to be wrong, route that case into a queue instead of letting it disappear into your logs.

# failure_logger.py
import json
from datetime import datetime, timezone

def log_for_retraining(alert_context, model_output, outcome: str, human_correction: dict = None):
    """outcome is one of: 'success', 'schema_invalid', 'human_overridden', 'escalation_incorrect'"""
    record = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "context": alert_context,
        "model_output": model_output,
        "outcome": outcome,
        "human_correction": human_correction,
    }
    if outcome != "success":
        with open("retraining_queue.jsonl", "a") as f:
            f.write(json.dumps(record) + "
")

Every few weeks or months, depending on your traffic volume, that queue becomes the seed for your next round of fine-tuning, and it's usually far more valuable per example than your original synthetic dataset, because it's made entirely of cases your current model actually got wrong. This is the flywheel that separates an agent that stays reliable from one that was reliable once, in a demo, a while ago.

Mistakes That Quietly Sink These Projects

A short list of things that show up again and again once an agentic fine-tune is in production:

• Training data that doesn't match the runtime chat template. If your dataset formats tool calls slightly differently than your serving stack expects, you can train a model that looks perfect in evaluation and still fails in production.
• Too many easy examples, not enough edge cases. A dataset full of obvious tool calls teaches the model to pattern-match keywords instead of actually reasoning about the situation.
• Skipping runtime tuning entirely. Teams pour weeks into the training run and then ship with default temperature and no retry policy, then blame the model when it's really an inference-time configuration problem.
• Only testing the target task. A fine-tune that nails tool selection but quietly degrades the model's general reasoning will eventually cause a weird failure somewhere you weren't watching.
• Treating one fine-tuning pass as the finish line. Without a feedback loop, the agent you shipped is the best it will ever be, and it only gets worse from there.

Frequently Asked Questions

Do I need a GPU to fine-tune an agentic model? For any real training run, yes. QLoRA loading in 4-bit precision requires a CUDA-capable GPU, though a single mid-range or rented cloud GPU is usually enough for a 7B to 8B model. Dataset building, evaluation, and monitoring code don't need a GPU at all.

How much training data does an agent actually need? For format and tool-selection behavior, a few hundred well-constructed, validated trajectories will get you further than a few thousand sloppy ones. Volume matters more once you're using synthetic generation to cover edge cases you can't easily write by hand.

Is fine-tuning better than RAG for building agents? They solve different problems and most production agents end up using both. RAG handles missing or changing knowledge. Fine-tuning handles consistent behavior: exact output format, tool-selection judgment, and domain-specific decision-making that a prompt alone won't reliably pin down.

Can I fine-tune a closed model like GPT or Claude for agentic behavior? Some providers offer managed fine-tuning APIs for select models, which handle the training infrastructure for you but give you far less control over the method (LoRA rank, preference optimization technique, and so on) than working with an open-weight model directly.

How often should an agent be retrained? There's no fixed schedule that works for everyone. A better signal is your production failure queue: once it's accumulated enough real, verified failure cases to meaningfully move the needle, that's a good time to retrain, whether that ends up being every few weeks or every few months.

Wrapping Up

Fine-tuning an agent well means treating it as seven connected problems instead of one training run: a dataset that mirrors real multi-turn behavior, a training method that fits your hardware, runtime settings tuned with the same care as training hyperparameters, preference alignment for the judgment calls a single label can't teach, a real path to deployment, evaluation that's honest about both target performance and general capability, and a feedback loop that keeps the whole thing from decaying after launch. Skip any one of these and you'll likely still get something that looks good in a demo. The steps that actually determine whether it survives contact with real traffic are the ones nobody puts in the highlight reel.