agentcheck

Synthetic checks, nightly regression replay and model-drift alerts for AI agents

CommunityOnlineAPI key requiredGlobalFreeRead-only

What it can do

    What data it sees

    Do you need an account

    An API key from the service settings is required

    Synthetic checks, nightly regression replay and model-drift alerts for AI agents

    How to connect

    How to connect agentcheck to Claude

    The server address to paste into Claude settings:

    https://agentwares-agentcheck.vercel.app/api/mcp

    1. Open Claude (claude.ai or the desktop app).
    2. Go to Settings → Connectors.
    3. Click Add custom connector.
    4. Paste the server address copied below into Remote MCP server URL and click Add.
    5. In a chat, click + → Connectors and switch the new connector on.

    Custom connectors are available on Free, Pro, Max, Team and Enterprise plans (Free is limited to one). On Team and Enterprise an organization Owner adds the connector first under Organization settings → Connectors.

    Authorization

    After you click Add, a sign-in window for the service opens. Sign in with your own account and approve access. Claude never sees your password.

    How to connect agentcheck to ChatGPT

    The server address to paste into ChatGPT settings:

    https://agentwares-agentcheck.vercel.app/api/mcp

    1. Open ChatGPT in a browser (chatgpt.com). A Plus, Pro, Business, Enterprise or Edu plan is required.
    2. Turn on developer mode once: Settings → Apps → Advanced settings → Developer mode.
    3. Open Settings → Connectors and click Create.
    4. Fill in the form: Name (anything), Description (one line about what the service does), MCP server URL (copy it below).
    5. Under Authentication choose OAuth if the service requires sign-in, otherwise None. Click Create.
    6. In a new chat open + → Apps/Connectors and enable the connector.

    OpenAI has renamed this section before (Connectors → Apps/Plugins). If the label differs, search settings for "developer mode". On Business/Enterprise workspaces an admin must allow custom connectors first.

    Authorization

    On first use ChatGPT opens the service's sign-in window. Sign in and approve access.

    How to connect agentcheck to Cursor

    The server address to paste into Cursor settings:

    https://agentwares-agentcheck.vercel.app/api/mcp

    Fastest: click Open in Cursor below and confirm the prompt.

    Manually:

    1. In Cursor open Settings → Cursor Settings → MCP and click Add new global MCP server.
    2. Paste the JSON copied below into ~/.cursor/mcp.json (per project: .cursor/mcp.json in the repo root).
    3. Save the file. The server appears in the MCP list; authorize it there if asked.

    Authorization

    If the service needs sign-in, an authorize button appears next to the server in the MCP list.

    Server tool list (8)

    Raw names from tools/list. Only developers need these.

    agentcheck_get_statusCurrent status of a public monitored target: overall state, uptime over 24h/7d/30d, last check time, last nightly scores, open incidents and the badge/status URLs. Use the owner (GitHub login) and target slug from the status page URL https://agentwares-agentcheck.vercel.app/<owner>/<slug>. No API key needed.
    agentcheck_get_pricingMachine-readable pricing for agentcheck: tiers with monthly USD price, target limits, check interval and features, plus per-run add-ons. Same data as /pricing.json. No API key needed.
    agentcheck_create_targetEnroll something to monitor: an http endpoint (JSON or OpenAI-style chat), a remote MCP server (Streamable HTTP url) or an A2A agent (origin with /.well-known/agent-card.json). Pass `checks` to create checks in the same call (POST /api/v1/probe proposes three). Returns the target id, the public status page, the badge SVG URL and a README snippet. The first check runs on the next minute tick; call agentcheck_run_now to run immediately. Free tier: 1 target, hourly; Starter+: 5-minute checks. Requires an API key.
    agentcheck_add_checkAdd a check to one of your targets. A check runs an input (http path/prompt, mcp tool call, a2a message) on a schedule and judges the answer with a golden: exact, contains, regex and json_schema cost nothing; rubric and baseline use the LLM judge (baseline = same outcome as the last known-good answer). Returns the check id. Requires an API key.
    agentcheck_run_nowRun every check of one of your targets immediately (outside the schedule) and return pass/fail per check with latency, judge cost and any incident opened or closed. Use it right after enrolling, or to confirm a fix. Requires an API key.
    agentcheck_recordRecord one production interaction with your agent (the prompt or messages, the final answer, the tools it called, optionally the model) as a trace on a target. Call it from your agent or from a proxy in front of it after each task; promote a good trace with agentcheck_promote_trace to replay it nightly and catch regressions. Requires an API key.
    agentcheck_promote_traceTurn an imported or recorded trace into a replayable check whose golden is the recorded outcome (tool sequence + final-answer rubric). The check runs daily and in the nightly replay (Pro). Returns the check. Requires an API key.
    agentcheck_list_incidentsOpen and recent incidents across your targets (or one target): when they opened/closed, the failing check and the cause. Requires an API key.