Adding MCP problem type basic sampler infrastructure by guidorc · Pull Request #1628 · WebFuzzing/EvoMaster

guidorc · 2026-07-04T16:18:59Z

Summary

Adding the basic search infrastructure for blackbox fuzzing of Model Context Protocol (MCP) servers. The approach is: connect to the target server, discover its capabilities via the standard MCP handshake, automatically construct a mutable input representation (gene tree) for each tool and resource from their JSON Schema definitions, then use a search algorithm to evolve a sequences of calls that maximise coverage; reaching each capability and obtaining a non-error response

Components

MCP client (HttpMcpClient, McpClient, McpDTOs)

An HTTP-based client that implements the MCP JSON-RPC protocol: initialization handshake, tool invocation (tools/call), static resource reads (resources/read), and capability discovery (tools/list, resources/list, resources/templates/list).

Action model (McpAction, McpToolCallAction, McpResourceReadAction)

Two concrete action types mirror the two MCP interaction patterns: calling a named tool with structured arguments, and reading a resource by URI (including template URIs with mutable parameters).

Individual (McpIndividual)

A test case represented as an ordered sequence of McpActions, integrating with EvoMaster's existing EnterpriseIndividual and mutation infrastructure.

Sampler (McpSampler)

On startup, connects to the target MCP server, performs the handshake, and discovers all tools and resources. Each tool's JSON Schema input definition is converted into an ObjectGene tree used by the search engine to mutate arguments. Pre-built single-action individuals ensure every capability is exercised at least once before random search begins.

Fitness function (McpBlackBoxFitness)

Evaluates individuals by executing their actions sequentially and scoring coverage targets based on observable protocol signals: 1.0 for a successful tool call or resource read, 0.5 for a server-reported error (partial credit keeps the search gradient useful). Exceptions break the action sequence early.

McpBlackBoxModule

Wires all the above into EvoMaster's dependency injection container and registers the problem type in the main entry point.

Follow Up

McpTestCaseWriter — the test output pipeline currently uses NoTestCaseWriter, so generated test files have empty bodies. A dedicated writer that emits executable MCP
client calls is needed.
Format-aware gene building — string fields carrying semantic formats (date-time, email, uuid) are built as plain StringGene. The sampler needs a three-layer cascade
(explicit JSON Schema format → type switch → name-based heuristics) to generate valid values for these fields, matching the approach used by the REST builder.
SortingHelper / TestCaseNamingStrategyFactory — McpIndividual is not yet registered in the test sorting and naming strategies, causing a handled warning during output.

@PostConstruct

Introduces a new problem domain for blackbox testing of MCP (Model Context Protocol) servers, following the same architecture as the existing GraphQL and RPC problem domains. New components under core/src/main/kotlin/.../problem/mcp/: - McpAction (abstract base), McpToolCallAction (tools/call), McpResourceReadAction (resources/read), McpInputParam, McpUriParam - McpIndividual: chromosome wrapping sequences of MCP actions - McpCallResult: per-action execution record storing isError flag - client/McpClient (interface), client/HttpMcpClient (JSON-RPC 2.0 over HTTP with pagination), client/McpDTOs - service/McpSampler: @PostConstruct discovery via tools/list, resources/list, resources/templates/list; gene tree building from JSON Schema; random and smart (ad-hoc) sampling - service/McpFitness (abstract base), service/McpBlackBoxFitness: per-action scoring using isError flag and successful reads as signals - service/McpBlackBoxModule: Guice bindings mirroring GraphQLBlackBoxModule Unit tests cover action construction, individual copy/mutation, and HttpMcpClient JSON-RPC parsing via WireMock stubs.

- Add ProblemType.MCP (experimental) to EMConfig enum - Add bbTargetUrl validation for MCP black-box mode - Route ProblemType.MCP to McpBlackBoxModule in Main.init() - Add getAlgorithmKeyMcp() supporting RANDOM, MIO, MOSA, WTS, SMARTS - Dispatch MCP algorithm key in Main.run() - Bind NoTestCaseWriter in McpBlackBoxModule EvoMaster can now be invoked with: --blackBox true --problemType MCP --bbTargetUrl <url>

@PostConstruct

Connection failures during @PostConstruct were surfacing as an uncaught InvocationTargetException labelled as an EvoMaster bug. Wrap the listTools() call so a failed connection to the MCP server produces a clean, user-readable SutProblemException instead.

Servers may return 400/404/405 for MCP methods they don't support (e.g., resources/list on a tools-only server). Previously this caused an IOException from HttpURLConnection.inputStream on non-2xx responses, crashing the sampler during initialization. Extracted openConnection() helper and made post() return null on 400/404/405, which callers treat as an empty/error result.

* Setting up MCP Problem Type Skeleton * remove db related size from MCP individual * Fix for resource read action id Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * feedback * adjust resource read action parameters --------- Co-authored-by: guido-rodriguez_sfemu <guido.rodriguez@salesforce.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Pgarrett · 2026-07-27T23:30:57Z

@jgaleotti can you review please?

jgaleotti · 2026-07-28T12:47:01Z

+    private fun postNotification(method: String, params: Map<String, Any?>) {
+        val response = sendJsonRpc(method, params, id = null, acceptEventStream = false)
+        // Discard the response to complete the HTTP exchange and release the connection
+        try { response.close() } catch (_: Exception) {}


should exceptions go unnoticed here? Shouldn't this be logged?

Here we are only silencing a failure while closing the connection for an already received response. If the actual request fails that will be propagated and fail the initialization. WIll add a log for tracing purposes.

jgaleotti · 2026-07-28T12:47:44Z

+            val response = post(method, params)
+                ?: throw IllegalStateException("$method request failed: received null response")
+
+            val result = response["result"] as? Map<String, Any?>


should we have "result" as a constant?

I don't think we gain much in this case

jgaleotti · 2026-07-28T12:48:09Z

+            throw IllegalStateException("MCP initialize handshake returned empty body")
+        }
+        // Send the required follow-up notification (fire-and-forget)
+        postNotification("notifications/initialized", emptyMap())


should this string be a constant value?

jgaleotti · 2026-07-28T12:48:21Z

+            "method" to method,
+            "params" to params
+        )
+        id?.let { payload["id"] = it }


should "id" be a constant?

Same as for "response"

jgaleotti · 2026-07-28T12:49:21Z

+            val type = item["type"] as? String
+                ?: throw IllegalStateException("Tool: $name response content item is missing required 'type' field")
+            when (type) {
+                "text" -> McpTextToolContent(item["text"] as? String ?: "")


plenty of constant strings are used here, should they be constant values?

Extracted MCP specific types to constants

guido-rodriguez_sfemu and others added 15 commits May 28, 2026 22:11

Add missing header

badf32d

Add MCP initialization handshake

cf1c967

Merge branch 'master' into feature/mcp-blackbox-sampler

af0cdd2

merge skeleton setup

999c3e0

Fix for MCP uriTemplate

5aea547

store MCP-Session-Id on inititalize

38ef921

Merge branch 'master' into feature/mcp-blackbox-sampler

b4a4384

Adding JavaDocs

2b0e604

remove supress unchecked cast annotations

11df7b1

Merge branch 'master' into feature/mcp-blackbox-sampler

ef80412

guidorc changed the base branch from master to external-pr-mcp July 4, 2026 16:37

Merge branch 'external-pr-mcp' into feature/mcp-blackbox-sampler

e2820c6

guidorc changed the base branch from external-pr-mcp to master July 4, 2026 16:41

Merge branch 'master' into feature/mcp-blackbox-sampler

5cf695e

guidorc requested a review from Pgarrett July 7, 2026 00:57

Merge branch 'master' into feature/mcp-blackbox-sampler

e210728

Pgarrett requested changes Jul 12, 2026

View reviewed changes

guidorc changed the title ~~Adding MCP problem type basic fuzzing infrastructure~~ Adding MCP problem type basic sampler infrastructure Jul 13, 2026

guido-rodriguez_sfemu and others added 4 commits July 12, 2026 23:02

Adopt Jersey client

d240e23

Extract constants into shared file

c501ad1

use JsonNode data type for tools input schema

ed9906d

use MediaType constants

d4c0c34

guidorc force-pushed the feature/mcp-blackbox-sampler branch from c7cc742 to d4c0c34 Compare July 13, 2026 03:05

guidorc added 2 commits July 13, 2026 20:13

remove redundant HTTP status check

4e6b786

refactor MCP list response navigation

14b9b8d

guidorc added 2 commits July 13, 2026 21:50

Validate tool result includes type

76948f5

general feedback

9517441

guidorc requested a review from Pgarrett July 14, 2026 02:36

Pgarrett reviewed Jul 18, 2026

View reviewed changes

Comment thread core/src/main/kotlin/org/evomaster/core/problem/mcp/McpConst.kt

Pgarrett requested changes Jul 18, 2026

View reviewed changes

guidorc and others added 3 commits July 25, 2026 11:06

Merge branch 'master' into feature/mcp-blackbox-sampler

02d462b

feedback

12cdd69

add test coverage

6527f6e

Pgarrett approved these changes Jul 27, 2026

View reviewed changes

Pgarrett requested review from Copilot and jgaleotti July 27, 2026 23:30

Copilot started reviewing on behalf of Pgarrett July 27, 2026 23:30 View session

Pgarrett removed the request for review from Copilot July 27, 2026 23:30

jgaleotti requested changes Jul 28, 2026

View reviewed changes

guidorc added 2 commits July 30, 2026 22:37

Update HttpMcpClient

ac1fd1e

remove input schema object builder

5f6f876

guidorc requested a review from jgaleotti July 31, 2026 01:57

Uh oh!

Conversation

guidorc commented Jul 4, 2026

Summary

Components

MCP client (HttpMcpClient, McpClient, McpDTOs)

Action model (McpAction, McpToolCallAction, McpResourceReadAction)

Individual (McpIndividual)

Sampler (McpSampler)

Fitness function (McpBlackBoxFitness)

McpBlackBoxModule

Follow Up

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pgarrett commented Jul 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants