Nonobench
An open-source benchmark of how well LLMs solve nonogram puzzles, from 5x5 to 20x20.
What it can do
- Get Leaderboard: Models ranked by accuracy; defaults to all effort levels for compatibility.
- List Providers: Provider ids, names, families and variant counts.
- List Families: Model families, available efforts and best variants.
What data it sees
Do you need an account
No: the server works without sign-in
An open-source benchmark of how well LLMs solve nonogram puzzles, from 5x5 to 20x20. Query every result through a REST API or MCP server.
How to connect
How to connect Nonobench to Claude
The server address to paste into Claude settings:
https://nonobench.run.tools
- Open Claude (claude.ai or the desktop app).
- Go to Settings → Connectors.
- Click Add custom connector.
- Paste the server address copied below into Remote MCP server URL and click Add.
- In a chat, click + → Connectors and switch the new connector on.
Custom connectors are available on Free, Pro, Max, Team and Enterprise plans (Free is limited to one). On Team and Enterprise an organization Owner adds the connector first under Organization settings → Connectors.
Authorization
After you click Add, a sign-in window for the service opens. Sign in with your own account and approve access. Claude never sees your password.
How to connect Nonobench to ChatGPT
The server address to paste into ChatGPT settings:
https://nonobench.run.tools
- Open ChatGPT in a browser (chatgpt.com). A Plus, Pro, Business, Enterprise or Edu plan is required.
- Turn on developer mode once: Settings → Apps → Advanced settings → Developer mode.
- Open Settings → Connectors and click Create.
- Fill in the form: Name (anything), Description (one line about what the service does), MCP server URL (copy it below).
- Under Authentication choose OAuth if the service requires sign-in, otherwise None. Click Create.
- In a new chat open + → Apps/Connectors and enable the connector.
OpenAI has renamed this section before (Connectors → Apps/Plugins). If the label differs, search settings for "developer mode". On Business/Enterprise workspaces an admin must allow custom connectors first.
Authorization
On first use ChatGPT opens the service's sign-in window. Sign in and approve access.
How to connect Nonobench to Cursor
The server address to paste into Cursor settings:
https://nonobench.run.tools
Fastest: click Open in Cursor below and confirm the prompt.
Manually:
- In Cursor open Settings → Cursor Settings → MCP and click Add new global MCP server.
- Paste the JSON copied below into
~/.cursor/mcp.json(per project:.cursor/mcp.jsonin the repo root). - Save the file. The server appears in the MCP list; authorize it there if asked.
Authorization
If the service needs sign-in, an authorize button appears next to the server in the MCP list.
Server tool list (11)
Raw names from tools/list. Only developers need these.
| get_leaderboard | Models ranked by accuracy; defaults to all effort levels for compatibility. |
| list_providers | Provider ids, names, families and variant counts. |
| list_families | Model families, available efforts and best variants. |
| compare_models | Side-by-side core overall and per-size accuracy, cost, latency and token results for model or family names. |
| get_model_results | Accuracy, cost, latency and token use for one model, broken down by grid size. |
| list_puzzles | The benchmark puzzles with their ids and row/column clues. |
| get_puzzle | One puzzle, including the clue text models were prompted with. The reference solution is only included on request; some puzzles have several valid solutions. |
| check_solution | Check a grid against a puzzle's clues, using the same rule as the benchmark grader. Reports which rows and columns do not match. |
| get_puzzle_results | Per-model outcomes for one puzzle. Answers are omitted unless requested. |
| get_model_puzzles | Which puzzles one model solved, missed, timed out on, or has not run. |
| list_runs | Individual benchmark runs (one model on one puzzle), optionally with the raw prompt and model output. |