CompletionKit

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

CommunityOnlineAPI key requiredGlobalFreeRead-only

What it can do

    What data it sees

    Do you need an account

    An API key from the service settings is required

    Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

    How to connect

    How to connect CompletionKit to Claude

    The server address to paste into Claude settings:

    https://completionkit.com/mcp

    1. Open Claude (claude.ai or the desktop app).
    2. Go to Settings → Connectors.
    3. Click Add custom connector.
    4. Paste the server address copied below into Remote MCP server URL and click Add.
    5. In a chat, click + → Connectors and switch the new connector on.

    Custom connectors are available on Free, Pro, Max, Team and Enterprise plans (Free is limited to one). On Team and Enterprise an organization Owner adds the connector first under Organization settings → Connectors.

    Authorization

    After you click Add, a sign-in window for the service opens. Sign in with your own account and approve access. Claude never sees your password.

    How to connect CompletionKit to ChatGPT

    The server address to paste into ChatGPT settings:

    https://completionkit.com/mcp

    1. Open ChatGPT in a browser (chatgpt.com). A Plus, Pro, Business, Enterprise or Edu plan is required.
    2. Turn on developer mode once: Settings → Apps → Advanced settings → Developer mode.
    3. Open Settings → Connectors and click Create.
    4. Fill in the form: Name (anything), Description (one line about what the service does), MCP server URL (copy it below).
    5. Under Authentication choose OAuth if the service requires sign-in, otherwise None. Click Create.
    6. In a new chat open + → Apps/Connectors and enable the connector.

    OpenAI has renamed this section before (Connectors → Apps/Plugins). If the label differs, search settings for "developer mode". On Business/Enterprise workspaces an admin must allow custom connectors first.

    Authorization

    On first use ChatGPT opens the service's sign-in window. Sign in and approve access.

    How to connect CompletionKit to Cursor

    The server address to paste into Cursor settings:

    https://completionkit.com/mcp

    Fastest: click Open in Cursor below and confirm the prompt.

    Manually:

    1. In Cursor open Settings → Cursor Settings → MCP and click Add new global MCP server.
    2. Paste the JSON copied below into ~/.cursor/mcp.json (per project: .cursor/mcp.json in the repo root).
    3. Save the file. The server appears in the MCP list; authorize it there if asked.

    Authorization

    If the service needs sign-in, an authorize button appears next to the server in the MCP list.

    Server tool list (54)

    Raw names from tools/list. Only developers need these.

    prompts_listList all prompts
    prompts_getGet a prompt by ID
    prompts_createCreate a prompt
    prompts_updateUpdate a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.
    prompts_deleteDelete a prompt
    prompts_publishPublish a prompt version, making it the current version
    prompts_suggest_improvementSuggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).
    runs_listList all runs
    runs_getGet a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.
    runs_createCreate a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.
    runs_updateUpdate a run
    runs_deleteDelete a run
    runs_generateStart a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.
    runs_regradeRe-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.
    runs_rerunCreate and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.
    runs_retry_failuresRe-run only the failed responses of a run, optionally limited to specific response ids via "only".
    responses_listList responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "score_asc" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.
    responses_getGet a specific response
    datasets_listList all datasets
    datasets_getGet a dataset by ID
    datasets_createCreate a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.
    datasets_updateUpdate a dataset
    datasets_deleteDelete a dataset
    datasets_create_from_urlCreate a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.
    metrics_listList all metrics
    metrics_getGet a metric by ID
    metrics_createCreate a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.
    metrics_updateUpdate a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.
    metrics_deleteDelete a metric
    metrics_suggest_variantsAsk the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.
    metric_groups_listList all metric groups
    metric_groups_getGet a metric group by ID
    metric_groups_createCreate a metric group
    metric_groups_updateUpdate a metric group
    metric_groups_deleteDelete a metric group
    metric_versions_listList every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.
    metric_versions_publishPublish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.
    metric_versions_dismissDestroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as current instead.
    provider_credentials_listList all provider credentials (API keys are not exposed)
    provider_credentials_getGet a provider credential by ID (API key is not exposed)
    provider_credentials_createCreate a provider credential
    provider_credentials_updateUpdate a provider credential
    provider_credentials_deleteDelete a provider credential
    tags_listList all tags
    tags_getGet a tag by ID
    tags_createCreate a tag. Color is auto-assigned.
    tags_updateRename a tag.
    tags_deleteDelete a tag. Removes the tag from every linked metric, prompt, run, and dataset.
    agreements_listList agreements. Filter by run_id, response_id, metric_id, or created_by.
    agreements_createUpsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.
    judges_replayCreate a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.
    judges_compareCompare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.
    promptfoo_importImport a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.
    usage_getGet this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.