mcp-validator

MCP Validator

A testing suite and reference implementation for the Model Context Protocol (MCP).

Summary

The MCP Validator provides a comprehensive environment for testing and validating MCP server implementations. It includes reference implementations and a testing framework to ensure compliance with the MCP specification.

Latest Protocol Support: The validator now supports MCP protocol version 2025-06-18, which includes major enhancements like structured tool output, OAuth 2.1 authentication, elicitation support, removal of JSON-RPC batching, and enhanced security features.

🔐 OAuth 2.1 Authentication Support

The validator includes comprehensive OAuth 2.1 authentication support for the 2025-06-18 protocol version, providing a complete framework for secure MCP implementations.

Authentication Features

• OAuth 2.1 Compliance: Full RFC 6749, RFC 6750, and OAuth 2.1 draft compliance
• Bearer Token Support: Proper Bearer token extraction and validation
• WWW-Authenticate Headers: Standards-compliant 401 responses with authentication challenges
• Resource Server Capabilities: Complete OAuth 2.1 resource server implementation
• MCP-Protocol-Version Headers: Protocol version negotiation via HTTP headers
• Security Headers: CORS, Origin validation, and DNS rebinding attack prevention

OAuth 2.1 Configuration

The reference HTTP server supports OAuth 2.1 authentication through environment variables:

# Enable OAuth 2.1 authentication
export MCP_OAUTH_ENABLED=true
export MCP_OAUTH_INTROSPECTION_URL=https://auth.example.com/oauth/introspect
export MCP_OAUTH_REQUIRED_SCOPES=mcp:read,mcp:write
export MCP_OAUTH_REALM=mcp-server

# Optional: Configure resource indicators (RFC 8707)
export MCP_OAUTH_RESOURCE_INDICATORS=https://api.example.com/mcp

Authentication Testing

Test OAuth 2.1 compliance with the built-in authentication test suite:

# Test OAuth 2.1 features (framework validation)
python mcp_testing/scripts/http_compliance_test.py --debug

# Test with authentication enabled
MCP_OAUTH_ENABLED=true python ref_http_server/reference_mcp_server.py --port 8088 &
python mcp_testing/scripts/http_compliance_test.py --server-url http://localhost:8088

# Test Bearer token handling
curl -H "Authorization: Bearer your-token-here" \
     -H "MCP-Protocol-Version: 2025-06-18" \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"client_info":{"name":"Test"},"client_capabilities":{"protocol_versions":["2025-06-18"]}}}' \
     http://localhost:8088/messages

🚀 2025-06-18 Protocol Features

Core Enhancements

• Structured Tool Output: Tools return structured responses with content, isError, and structuredContent fields
• Enhanced Tool Schema: Tools use inputSchema (renamed from parameters), title, and outputSchema fields
• JSON-RPC Batching Removal: Batch requests are properly rejected for 2025-06-18 protocol
• Elicitation Support: Framework for user interaction and data collection
• Protocol Version Headers: HTTP requests include MCP-Protocol-Version header

Security Improvements

• OAuth 2.1 Authentication: Complete authentication framework
• Enhanced Error Handling: Improved error responses with security considerations
• Origin Validation: DNS rebinding attack prevention
• Session Security: Secure session management with per-session protocol versions

STDIO Compliance Testing

The validator includes a comprehensive testing suite for STDIO-based MCP servers.

Running STDIO Tests

# Run compliance tests for the STDIO server (latest protocol)
python -m mcp_testing.scripts.compliance_report --server-command "python ref_stdio_server/stdio_server_2025_03_26.py" --protocol-version 2025-06-18

# Run with previous protocol versions
python -m mcp_testing.scripts.compliance_report --server-command "python ref_stdio_server/stdio_server_2025_03_26.py" --protocol-version 2025-03-26

STDIO Test Coverage

The STDIO compliance tests verify:

1. Protocol Initialization
2. Tools Functionality
   • Basic tools (echo, add)
   • Async tools (sleep) for 2025-03-26+ versions
   • Structured tool output (2025-06-18)
3. Error Handling
4. Protocol Version Negotiation
5. Advanced Features (2025-06-18)
   • Elicitation support
   • Enhanced validation
   • JSON-RPC batching restrictions

Testing Different STDIO Server Types

The validator supports testing any STDIO-based MCP server, whether it's run directly from a command or installed via pip. Here's how to test different types of servers:

Direct Command Testing

For servers that run directly from a Python file or command:

# Test a local Python file (latest protocol)
python -m mcp_testing.scripts.compliance_report --server-command "python path/to/your/server.py" --protocol-version 2025-06-18

# Test with specific timeouts
python -m mcp_testing.scripts.compliance_report --server-command "python path/to/server.py" --protocol-version 2025-06-18 --test-timeout 30 --tools-timeout 15

# Focus on tools testing with dynamic discovery
python -m mcp_testing.scripts.compliance_report --server-command "python path/to/server.py" --protocol-version 2025-06-18 --test-mode tools --dynamic-only

# Test with previous protocol versions for compatibility
python -m mcp_testing.scripts.compliance_report --server-command "python path/to/server.py" --protocol-version 2025-03-26

Testing Pip-Installed Servers

For servers installed via pip (like mcp-server-fetch):

# Ensure you're in the correct virtual environment
source .venv/bin/activate

# Install the server and dependencies
pip install your-mcp-server  # Replace with actual package name

# Run compliance tests
python -m mcp_testing.scripts.compliance_report --server-command "python -m your_server_module" --protocol-version 2024-11-05

# Run tools-only tests
python -m mcp_testing.scripts.compliance_report --server-command "python -m your_server_module" --protocol-version 2024-11-05 --test-mode tools

# example brave search server
BRAVE_API_KEY=api-key python -m mcp_testing.scripts.compliance_report --server-command "npx -y @modelcontextprotocol/server-brave-search" --protocol-version 2024-11-05 

Test Configuration Options

Common options for both types:

• --test-mode tools: Focus on testing tool functionality
• --dynamic-only: Automatically discover and test available tools
• --test-timeout 30: Set timeout for regular tests (seconds)
• --tools-timeout 15: Set timeout for tool-specific tests (seconds)
• --required-tools tool1,tool2: Specify required tools to test
• --skip-tests test1,test2: Skip specific tests
• --skip-async: Skip async tool testing

Note: Tool-related tests that timeout are treated as non-critical, allowing testing to continue.

Test Reports

Each test run generates a detailed report containing:

• Server information (command, protocol version)
• Test execution timestamp
• Test duration
• Success rate
• Detailed results for each test case
• Server capabilities
• Session information

Running Tests

You can run different types of tests using module-style commands:

# Basic interaction test (latest protocol)
python -m mcp_testing.scripts.basic_interaction --server-command "python -m mcp_server_fetch" --protocol-version 2025-06-18

# Compliance tests with tools-only mode
python -m mcp_testing.scripts.compliance_report --server-command "python -m mcp_server_fetch" --protocol-version 2025-06-18 --test-mode tools

# Set custom timeouts for tools tests vs. other tests
python -m mcp_testing.scripts.compliance_report --server-command "python -m mcp_server_fetch" --protocol-version 2025-06-18 --test-timeout 30 --tools-timeout 15

# Test only specific functionality
python -m mcp_testing.scripts.compliance_report --server-command "/path/to/server" --protocol-version 2025-06-18 --test-mode tools

# Skip async tests if experiencing hanging issues
python -m mcp_testing.scripts.compliance_report --server-command "/path/to/server" --protocol-version 2025-06-18 --skip-async

# HTTP debug output
python -m mcp_testing.scripts.http_test --server-url http://localhost:8000/mcp --debug

# Test backward compatibility with older protocols
python -m mcp_testing.scripts.compliance_report --server-command "python -m mcp_server_fetch" --protocol-version 2024-11-05

Note: Tool-related tests that timeout are treated as non-critical, allowing testing to continue.

HTTP Compliance Testing

The validator includes a comprehensive compliance testing suite for HTTP-based MCP servers with full 2025-06-18 protocol support.

Running HTTP Tests

# Start the reference HTTP server (runs on port 8088)
python ref_http_server/reference_mcp_server.py

# Run compliance tests and generate a detailed report
python -m mcp_testing.scripts.http_compliance_test --output-dir reports

# Test with OAuth 2.1 authentication enabled
MCP_OAUTH_ENABLED=true python ref_http_server/reference_mcp_server.py --port 8088 &
python -m mcp_testing.scripts.http_compliance_test --server-url http://localhost:8088 --debug

HTTP Test Coverage

The HTTP compliance test suite verifies:

1. Protocol Initialization

   • 2025-06-18 protocol negotiation
   • Session management with protocol-specific features
   • MCP-Protocol-Version header handling

2. Tools Functionality

   • Structured tool output (2025-06-18)
   • Legacy tool responses (older protocols)
   • Enhanced tool schema validation

3. Error Handling

   • Enhanced error responses (2025-06-18)
   • OAuth 2.1 authentication errors
   • Proper HTTP status codes

4. Batch Request Processing

   • Batch request restrictions (2025-06-18)
   • Legacy batch support (older protocols)

5. Session Management

   • Per-session protocol versions
   • Session security and validation

6. Protocol Negotiation

   • Multi-version support (2024-11-05, 2025-03-26, 2025-06-18)
   • Intelligent version selection

7. Authentication & Security

   • OAuth 2.1 Bearer token validation
   • WWW-Authenticate header compliance
   • CORS and origin validation

8. Ping Utility

   • Connection testing and validation

Production Deployment

For production deployments with 2025-06-18 protocol:

# Enable HTTPS (required for OAuth 2.1 in production)
export MCP_TLS_CERT_FILE=/path/to/cert.pem
export MCP_TLS_KEY_FILE=/path/to/key.pem

# Configure OAuth 2.1 authentication
export MCP_OAUTH_ENABLED=true
export MCP_OAUTH_INTROSPECTION_URL=https://auth.example.com/oauth/introspect
export MCP_OAUTH_REQUIRED_SCOPES=mcp:read,mcp:write

# Configure CORS for web clients
export MCP_CORS_ORIGINS=https://myapp.example.com,https://anotherapp.example.com

# Enable rate limiting and security features
export MCP_RATE_LIMIT_ENABLED=true
export MCP_MAX_REQUESTS_PER_MINUTE=100

# Start the server
python ref_http_server/reference_mcp_server.py --port 443 --production

Testing Scripts Overview

The following scripts are available in mcp_testing/scripts/:

Active and Maintained

• http_compliance_test.py: Primary script for HTTP server testing with 2025-06-18 support (7/7 tests passing)
• compliance_report.py: Primary script for STDIO server testing with 2025-06-18 support (enhanced test coverage)

Supporting Scripts mixed working/in progress

• basic_interaction.py: Simple tool for testing basic server functionality
• http_test.py: Lower-level HTTP testing utilities with OAuth 2.1 support
• http_compliance.py: Core HTTP compliance testing logic
• http_compliance_report.py: Report generation for HTTP tests
• run_stdio_tests.py: Lower-level STDIO testing utilities
• session_test.py: Session management testing utilities

📋 Protocol Version Support

Protocol Version	Status	Features
2025-06-18	✅ Full Support	Structured tool output, OAuth 2.1, elicitation, no batching
2025-03-26	✅ Full Support	Async tools, enhanced capabilities, batch support
2024-11-05	✅ Full Support	Basic MCP functionality, legacy compatibility

🧪 Test Results Summary

Current test coverage for 2025-06-18 protocol:

✅ HTTP Compliance Test: 7/7 tests passed
✅ OAuth 2.1 Framework: 6/6 tests passed  
✅ Protocol Features: 7/7 tests passed
✅ Multi-Protocol Support: 3/3 versions supported
✅ Backward Compatibility: 100% maintained
✅ Security Features: Authentication, CORS, Origin validation

📚 Additional Resources

• JSON Schema: Complete schemas available in schema/ directory
• Specification Documents: Full specification files in specification/ directory
• Reference Implementations: HTTP and STDIO servers in ref_http_server/ and ref_stdio_server/
• Test Reports: Generated reports available in reports/ directory

License

SPDX-License-Identifier: AGPL-3.0-or-later
Copyright (c) 2025 Scott Wilcox