scorecard-mcp

Scorecard MCP Server

Connect Claude to Scorecard's AI evaluation platform through natural language conversations.
Test, measure, and improve your AI systems without switching between tools.

Quick Start

Add to your MCP client configuration:

{
  "mcpServers": {
    "scorecard": {
      "command": "npx",
      "args": ["mcp-remote", "https://app.scorecard.io/api/mcp"]
    }
  }
}

You'll authenticate with your Scorecard account on first use via Clerk OAuth.

What You Can Do

Ask Claude to help with AI evaluation tasks:

• "Create a new project for evaluating my chatbot"
• "Set up test cases for customer service scenarios"
• "Configure accuracy and helpfulness metrics"
• "Run an evaluation against my latest model"
• "Show me the performance results"

Available Operations

• Projects: Create and manage evaluation projects
• Test Sets: Build comprehensive test suites
• Test Cases: Add and organize individual test scenarios
• Metrics: Configure custom evaluation criteria
• Systems: Manage AI system configurations and versions
• Runs: Execute evaluations and analyze results

Technical Details

Built using the https://modelcontextprotocol.io/specification/2025-06-18/changelog on
Scorecard's Next.js frontend:

• Clerk OAuth for secure authentication
• JWT tokens passed to Scorecard's backend
• Auto-generated MCP tools from OpenAPI spec
• Deployed on Vercel's edge infrastructure

Security

• OAuth 2.0 authentication through Clerk
• Access limited to your authenticated Scorecard account
• Tokens passed through but never stored

Transform Scorecard into a conversational AI evaluation assistant - comprehensive model
testing through natural conversation.

Special thanks to Dustin Moore for his engineering leadership in developing this MCP implementation.