CompletionKit

CompletionKit: прогон промптов на датасете и оценка ответов LLM-судьёй по шкале 1–5

От сообществаРаботаетНужен API-ключГлобальныйБесплатноТолько чтение

Что умеет

  • Создавать и публиковать промпты
  • Собирать датасеты вручную или из URL
  • Запускать прогоны промптов на датасете
  • Пересчитывать оценки и повторять неудачные ответы
  • Просматривать метрики и результаты прогонов

Какие данные видит

Доступ по API-ключу к промптам, датасетам и результатам прогонов аккаунта пользователя.

Нужен ли аккаунт

Нужен API-ключ из настроек сервиса

CompletionKit — MCP-сервер для тестирования промптов: ассистент запускает промпт на выбранном датасете и получает оценку каждого ответа от LLM-судьи. Можно создавать и публиковать промпты, собирать датасеты, запускать прогоны, пересчитывать оценки и повторять неудачные запуски.

Как подключить

Как подключить CompletionKit к Claude

Адрес сервера, который нужно вставить в настройках Claude:

https://completionkit.com/mcp

  1. Откройте Claude (сайт claude.ai или приложение для компьютера).
  2. Перейдите в Настройки → Коннекторы (в английском интерфейсе: Settings → Connectors).
  3. Нажмите Добавить пользовательский коннектор (Add custom connector).
  4. Вставьте адрес сервера, скопированный ниже, в поле URL удалённого MCP-сервера и нажмите Добавить.
  5. В чате нажмите + → Коннекторы и включите новый коннектор.

Пользовательские коннекторы доступны на тарифах Free, Pro, Max, Team и Enterprise. На Free можно добавить только один. В Team и Enterprise коннектор сначала добавляет владелец организации в Настройки организации → Коннекторы.

Авторизация

После нажатия Добавить откроется окно входа в сервис. Войдите под своим аккаунтом и подтвердите доступ. Пароль Claude не видит.

Как подключить CompletionKit к ChatGPT

Адрес сервера, который нужно вставить в настройках ChatGPT:

https://completionkit.com/mcp

  1. Откройте ChatGPT в браузере (chatgpt.com). Нужен тариф Plus, Pro, Business, Enterprise или Edu.
  2. Один раз включите режим разработчика: Настройки → Приложения → Дополнительные настройки → Режим разработчика (Settings → Apps → Advanced settings → Developer mode).
  3. Откройте Настройки → Коннекторы и нажмите Создать (Create).
  4. Заполните форму: Название (любое), Описание (одна фраза, что делает сервис), URL MCP-сервера (скопируйте ниже).
  5. В поле Аутентификация выберите OAuth, если сервис требует вход, иначе Нет. Нажмите Создать.
  6. В новом чате откройте + → Приложения/Коннекторы и включите коннектор.

Название раздела у OpenAI менялось (Connectors → Apps/Plugins). Если пункт называется иначе, ищите «режим разработчика» в настройках. В рабочих пространствах Business/Enterprise администратор должен разрешить пользовательские коннекторы.

Авторизация

При первом использовании ChatGPT откроет окно входа в сервис. Войдите и подтвердите доступ.

Как подключить CompletionKit к Cursor

Адрес сервера, который нужно вставить в настройках Cursor:

https://completionkit.com/mcp

Быстрый способ: нажмите кнопку Открыть в Cursor ниже, Cursor сам предложит добавить сервер.

Вручную:

  1. В Cursor откройте Settings → Cursor Settings → MCP и нажмите Add new global MCP server.
  2. Вставьте JSON, скопированный ниже, в открывшийся файл ~/.cursor/mcp.json (для одного проекта: .cursor/mcp.json в корне проекта).
  3. Сохраните файл. Сервер появится в списке MCP; при необходимости пройдите авторизацию по кнопке рядом с ним.

Авторизация

Если сервис требует вход, рядом с сервером в списке MCP появится кнопка авторизации.

Список инструментов сервера (54)

Технические названия из tools/list. Нужны только разработчикам.

prompts_listList all prompts
prompts_getGet a prompt by ID
prompts_createCreate a prompt
prompts_updateUpdate a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.
prompts_deleteDelete a prompt
prompts_publishPublish a prompt version, making it the current version
prompts_suggest_improvementSuggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).
runs_listList all runs
runs_getGet a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.
runs_createCreate a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.
runs_updateUpdate a run
runs_deleteDelete a run
runs_generateStart a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.
runs_regradeRe-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.
runs_rerunCreate and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.
runs_retry_failuresRe-run only the failed responses of a run, optionally limited to specific response ids via "only".
responses_listList responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "score_asc" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.
responses_getGet a specific response
datasets_listList all datasets
datasets_getGet a dataset by ID
datasets_createCreate a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.
datasets_updateUpdate a dataset
datasets_deleteDelete a dataset
datasets_create_from_urlCreate a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.
metrics_listList all metrics
metrics_getGet a metric by ID
metrics_createCreate a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.
metrics_updateUpdate a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.
metrics_deleteDelete a metric
metrics_suggest_variantsAsk the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.
metric_groups_listList all metric groups
metric_groups_getGet a metric group by ID
metric_groups_createCreate a metric group
metric_groups_updateUpdate a metric group
metric_groups_deleteDelete a metric group
metric_versions_listList every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.
metric_versions_publishPublish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.
metric_versions_dismissDestroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as current instead.
provider_credentials_listList all provider credentials (API keys are not exposed)
provider_credentials_getGet a provider credential by ID (API key is not exposed)
provider_credentials_createCreate a provider credential
provider_credentials_updateUpdate a provider credential
provider_credentials_deleteDelete a provider credential
tags_listList all tags
tags_getGet a tag by ID
tags_createCreate a tag. Color is auto-assigned.
tags_updateRename a tag.
tags_deleteDelete a tag. Removes the tag from every linked metric, prompt, run, and dataset.
agreements_listList agreements. Filter by run_id, response_id, metric_id, or created_by.
agreements_createUpsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.
judges_replayCreate a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.
judges_compareCompare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.
promptfoo_importImport a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.
usage_getGet this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.