mcp-evals
A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.
GitHub Stars
92
User Rating
Not Rated
Favorites
0
Views
24
Forks
8
Issues
5
Installation
Difficulty
BeginnerEstimated Time
5-10 minutes
Installation
Installation
Prerequisites
Please specify required software and versions:Node.js: 18.0.0 or higher
npm: 8.0.0 or higher
Installation Steps
1. Clone Repository
bash
git clone https://github.com/mclenhard/mcp-evals.git
cd mcp-evals
2. Install Dependencies
bash
npm install
3. Configure as GitHub Action
Add the following to your workflow file:yaml
name: Run MCP Evaluations
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
evaluate:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm install
- name: Run MCP Evaluations
uses: mclenhard/mcp-evals@v1.0.9
with:
evals_path: 'src/evals/evals.ts'
server_path: 'src/index.ts'
openai_api_key: ${{ secrets.OPENAI_API_KEY }}
model: 'gpt-4'