By Tom HavyarimanaNovember 2, 2026
How to Test an MCP Server: From Tools to Agent Evals
An MCP tool can pass its tests while the agent still fails the task. MCP server testing needs to check both: does each tool work, and can the agent use the tools to answer the user's request?
In an HR connector we built, the timesheet export route was unavailable for a connected company. We added a separate workday read tool. Testing the export wrapper alone could not establish that the user's timesheet request would succeed. We also needed to evaluate which tool the agent chose and what it returned.
This guide turns lessons from our custom connectors into a practical evaluation workflow: test each tool, check agent tool selection, then grade complete scenarios against an expected result. You will work through a runnable inventory example with unit tests, MCP integration tests, and a trace grader. It runs without model credentials; the supplied agent attempts are synthetic fixtures for testing the grader.
1. Separate tool tests from agent evaluations
In Model Context Protocol, the server exposes capabilities; the agent application decides how to use them. Test both sides of that interface.
There are four useful checks:
| Check | Question | Inventory example |
|---|---|---|
| Unit test | Does the tool's implementation work? | get_stock("item-7") returns four units. |
| MCP integration test | Can a client discover and call the exposed tool correctly? | The schema accepts an item ID and the result has the expected structure. |
| Tool-use evaluation | Does the agent choose suitable tools and arguments? | It resolves the product name before passing an ID to get_stock. |
| Scenario evaluation | Does the task complete, and if not, does recovery follow the case policy? | It reports four units, or explains that availability cannot be checked during an outage. |
Testing whether an agent selects a tool involves model behavior, even with only one tool under test. Keep that evaluation separate from deterministic unit and MCP integration tests.
For the timesheet case, test the endpoint and date arguments directly, then evaluate whether the agent selects the read tool. The scenario must check the employee, date range, and records, and reject an unrelated clock-in write.
The project examples are real and anonymized. Proposed regression cases are marked as such; they are not a claim that we ran the whole suite at the time.
2. Define the business result before the tool sequence
Consider a smaller task: “Is the blue desk lamp available?” Our fictional service exposes two read-only tools:
find_item(name) -> matching item IDs and names
get_stock(item_id) -> item ID and available quantity
Its fixture contains one item:
{
"item_id": "item-7",
"name": "blue desk lamp",
"available": 4
}
Define the initial state and expected result before running the agent:
| Case field | Definition |
|---|---|
| Request | “Is the blue desk lamp available?” |
| Initial state | Fixture above; no item ID or stock value in the agent's context |
| Required dependency | Resolve the name before checking stock for that item |
| Successful answer | Identifies the lamp as available with four units |
| Allowed actions | Read-only calls to these two tools |
| Outage policy | Explain that stock could not be checked; do not invent a quantity |
| Evidence | Calls, arguments, results/errors, final answer, and run completion |
The fixture is the source of truth. Do not ask the grader to trust the agent's claim that it succeeded.
Should you require an exact tool sequence?
For this case, find_item → get_stock expresses a real dependency: the second operation needs the ID returned by the first. Check that dependency.
An exact list of calls is often too restrictive. A permitted retry could make find_item → get_stock → get_stock valid. If a different case already provides a verified item ID, the lookup may be unnecessary. Define allowed routes and limits in the case rather than assuming one sequence fits every request.
Anthropic's agent evaluation guide makes this distinction between a successful outcome and a particular path through the task. Required ordering is useful when it protects correctness or permissions. Incidental ordering is a weak reason to reject a valid solution.
3. Test each tool and the contract the client sees
Download and extract the worked example. With Python 3.12 or later and uv installed, run these commands inside the extracted folder:
uv sync --locked --python 3.12
uv run --locked pytest -q
uv run --locked python evaluate.py
The package pins MCP SDK 2.3.0 and includes its dependency lockfile. The tests call the Python functions directly and also use the SDK's in-memory client to discover and invoke the actual MCP server. No external inventory service is contacted.
A direct test is deliberately simple:
assert get_stock("item-7") == {
"item_id": "item-7", "available": 4
}
Also test invalid inputs, an unknown item, and an empty search. A name passed where an ID is required should produce a useful error rather than a plausible result for the wrong product.
The MCP integration test crosses a different boundary. This block can also run as a Python script beside server.py:
import asyncio
from mcp import Client
from server import mcp
async def check_stock():
async with Client(mcp) as client:
result = await client.call_tool(
"get_stock", {"item_id": "item-7"}
)
assert not result.is_error
assert result.structured_content == {
"item_id": "item-7", "available": 4
}
asyncio.run(check_stock())
For our hosted upload tool, we removed the inline base64 field from discovery and rejected it at runtime. That change needs two tests: the schema must advertise the supported input, and the handler must reject the unsupported one.
The companion uses typed return structures so the SDK exposes structured results. These tests verify the local server and client contract. They do not cover remote transport, authentication, deployment, or model decisions. Test those boundaries in the application you intend to use.
4. Capture an agent attempt and grade it
Now give the request to an agent with the two tools available. Keep the answer hidden from the agent. Record enough information to check both the actions and the response.
A healthy attempt would contain these observations:
User: Is the blue desk lamp available?
find_item({"name": "blue desk lamp"})
-> {"matches": [{"item_id": "item-7", "name": "blue desk lamp"}]}
get_stock({"item_id": "item-7"})
-> {"item_id": "item-7", "available": 4}
Answer: Yes, there are four blue desk lamps available.
This transcript is illustrative. The companion supplies synthetic attempts to test the grader, including a correct attempt, a product name used as an ID, a wrong answer, an outage response, and a truncated capture. It does not call a live model or report an agent success rate.
Running evaluate.py prints:
correct_lookup: tool_use=pass, response_contract=pass
name_instead_of_id: tool_use=fail, response_contract=fail
wrong_answer: tool_use=pass, response_contract=fail
honest_outage: tool_use=pass, response_contract=pass
truncated_capture: tool_use=inconclusive, response_contract=inconclusive
response_contract checks whether the structured response meets the case policy. An honest outage response can pass this check without completing the stock lookup.
The grader checks that the stock call uses an ID obtained from the lookup. It compares the structured answer with the independent case fixture. These are separate checks: the agent can call the correct tools and still answer incorrectly.
The companion represents answers as structured fields such as:
{"status": "ok", "item_id": "item-7", "available": 4}
This keeps the demonstration deterministic. It does not grade the meaning of arbitrary prose. For a live agent, request a structured answer where the application supports it, or add a separate human or model-assisted rubric for the final text. If you extract fields from prose, evaluate that extraction step too; it can introduce errors.
Keep different failures separate
| Attempt | Tool use | Requested stock answer | Next investigation |
|---|---|---|---|
| Correct ID, four units reported | Correct | Established | Test more cases and repeated runs |
| Product name passed as ID | Incorrect | Not established | Parameter description, lookup dependency |
| Correct calls, answer says zero | Correct | Incorrect | How the agent interprets the result |
| Stock service unavailable, honest explanation | Correct | Not established; recovery policy met | Service availability and recovery behavior |
| Capture stops before the answer | Only partly observable | Inconclusive | Capture and timeout handling |
The MCP tools specification distinguishes protocol errors from tool execution errors. Preserve that distinction in your records. A correct call that encounters an upstream outage is different from a call with an invalid argument.
Keep execution, task completion, and recovery as separate verdicts rather than one unexplained success field.
Extend the case where real workflows get harder
The lamp case has one item, one inventory, and an immediate result. Our other connectors show which assumptions to test next.
Incomplete results. In our investment-system connector, native scenarios returned empty selections and truncated results. Fetching page two revealed another reconciliation run. An empty selection did not prove that a reconciliation was clean. For a “check all items” case, put a relevant item on page two. Require the agent to retrieve it or disclose incomplete coverage.
Account and permission context. Our accounting connector authenticated but retained only the first organization returned upstream. We added full-list persistence and explicit selection. Test two organizations with different fixture values for the same request, and verify the requested one. In our orchestration MCP, a comment tool's control flag could interrupt a running agent and required additional permission. The route test checks that a restricted caller is denied before any backend request. For forbidden actions, verify that nothing executes, not just that an error appears.
Asynchronous outcomes. Our invoice-processing workflow contract requires a batch ID, status checks, and a mapping from each input file to its upload and result. A proposed three-file case should include one completed file, one still processing, and one rejected before submission. The answer must preserve those distinctions. A batch ID alone proves none of them.
These are extensions for the relevant connector suites. The runnable lamp example stays small so you can inspect its entire test and grading path.
5. Change the boundary that actually failed
The repair is not always a better tool description. In the timesheet connector, we needed a separate read operation. In the hosted upload flow, we changed the schema and validation. In the evaluation work, we removed tools backed by unavailable routes from registration. Tests checked that removal; we did not run a fresh live evaluation to measure its effect on agent accuracy.
For the lamp example, suppose real attempts repeatedly pass "blue desk lamp" to get_stock. Inspect the tool definition the model actually receives. A vague description gives it little help:
Get stock for an item.
A candidate improvement makes the dependency explicit:
Return the available quantity for an inventory item.
Pass the item_id returned by find_item, not a product name.
A quantity of zero means the item is out of stock.
The companion server uses this more explicit contract. That is a candidate design, not evidence that it improves a model's behavior. To measure an improvement, run both versions with the same cases, model, client, instructions, fixture, and grading policy.
GitHub's offline MCP evaluation is a useful reference for tool-choice and argument checks. Retain each failed trace so you can distinguish those failures from execution or answer errors.
Extend the lamp case with an unknown product, an ambiguous name, zero stock, an outage, and requests needing no tool or already supplying an ID. Define the expected behavior for each; an unresolved ambiguous match should trigger clarification.
Keep some cases out of the editing loop. Use those held-out cases to check whether a change helps beyond the examples you used to write it. Repeat runs because model behavior can vary. Anthropic's guidance on writing tools connects tool design with this evaluation cycle.
Show counts as well as percentages: how many attempts ran, how many were scorable, how many completed the task, and how many were blocked or inconclusive. Do not hide incomplete runs to make a score look cleaner.
6. Test the evidence pipeline too
During my hosted MCP evaluation work, the native client invoked tools inside generated code. The evaluation framework converted the run into its own conversation format, but that representation omitted the inner calls.
A grader could then conclude that the agent had never selected the expected tool. The tool had run; the evidence was lost before grading.
We added native event capture and checks for matching call starts and completions. That caught incomplete or conflicting captures we could observe. It could not prove that an entire call pair had not disappeared before capture. To check that boundary, compare the evaluator's record with a run whose calls you can independently inspect.
The missing calls were specific to the versions and integration we tested.
Our historical suite checked tool selection and typed arguments. It deliberately discarded result bodies and final prose. The outcome checks in this guide are an extension of that work, illustrated with fictional data. They were not measured by that suite.
We also kept cases with missing inputs blocked and unscored, and incomplete timeout captures inconclusive.
Before trusting your own grader, use saved attempts to test it:
- Remove a completion event. A verdict that depends on it should become inconclusive.
- Insert a correct tool call with a wrong argument. Tool-name matching alone must not pass it.
- Keep successful calls but change the answer. The response check should fail.
- Supply an outage and an honest response. Preserve both the execution failure and the valid recovery.
The companion covers the last three patterns and marks an explicitly truncated trace inconclusive. Its complete flag is fixture input, not a production capture validator. A live adapter must establish that flag from actual run evidence.
7. Run through the client you intend to use
A connection failure in our HR connector looked like an authentication problem: the client never reached the hosted connect page. The incident record showed that the live service was still running an older build, although the repository already contained the new authentication flow. After redeployment, we verified the metadata routes and authentication challenge. That established the repaired connection surface, not successful completion of every agent task.
Test the deployed service, not just the current source.
To try the inventory case with your own agent, configure its MCP host to launch the companion's stdio entrypoint. Replace the folder path with the extracted package's absolute path:
uv --directory /absolute/path/mcp-evaluation-worked-example run --locked python server.py
The host communicates over standard input and output. The README separates the command and arguments for client configuration. Choose the model in your host; starting the server alone does not run an evaluation.
Use the same case definitions, reset the fixture between attempts, and capture native calls, results, errors, and final responses. Normalize them into the grader's format only after checking that the required evidence survives conversion. Redact secrets before storing traces.
Save the server revision, model and client versions, instructions, tool catalogue, case version, and grader version beside each attempt. Record which other tools were available: an isolated server and an agent with several connected services present different choices.
Start with one user request and an independent expected result. Test the tool contract, then the agent's choices and answer. Add the cases your workflow needs: partial data, account context, forbidden actions, or unfinished jobs. Preserve enough evidence to explain a failure before changing the tool.