Crawlab Mcp

What is Crawlab MCP? Crawlab MCP is a Model Context Protocol (MCP) server for Crawlab that exposes Crawlab functionality (spider, task and file management, resource access) to AI applications and LLM-driven clients. It provides a standardized MCP Server/Client bridge so LLMs can generate tool calls that are executed against a Crawlab instance.

How to use Crawlab MCP?

• Install: pip install -e . (or build/run with the provided Dockerfile).
• Configure: copy .env.example to .env and set CRAWLAB_API_BASE_URL and CRAWLAB_API_TOKEN to point at your Crawlab instance.
• Run server: python server.py or use the CLI (crawlab_mcp-mcp server) or run the Docker image (docker run -p 8000:8000 --env-file .env crawlab-mcp-server).
• Run client: crawlab_mcp-mcp client SERVER_URL or connect an AI client (e.g., Claude Desktop) by adding the MCP server URL.
• Use: send natural-language requests to your AI client; the LLM will produce tool calls that the MCP client sends to the MCP server, which performs Crawlab API operations and returns results.

Key features

• Spider management: create, read, update, delete spiders (create_spider, get_spider, update_spider, delete_spider).
• Task management: run, cancel, restart tasks; fetch logs (run_spider, cancel_task, restart_task, get_task_logs).
• File management: list and read spider files, save file contents (get_spider_files, get_spider_file, save_spider_file).
• Resource endpoints: standardized resources (spiders, tasks) exposed to LLM tooling.
• MCP architecture: client/server pattern that integrates with arbitrary LLM providers for natural-language-driven automation.
• Deployment options: local Python, Docker, Docker Compose integration for Crawlab stacks.

Common use cases

1. Natural-language management of scraping workflows (create/run spiders via chat).
2. Automating periodic crawls and orchestrating distributed runs across nodes.
3. Inspecting and editing spider code/files through an AI assistant.
4. Rapid debugging and log inspection using LLM-assisted queries.
5. Embedding Crawlab controls inside AI apps, dashboards, or chatbots.

FAQ

• Do I need a running Crawlab instance?

Yes. The MCP server proxies requests to an existing Crawlab API; you must provide the Crawlab base URL and an API token.

• Which LLMs are supported?

MCP is LLM-agnostic: it works with any LLM client that can call the MCP client and emit tool calls (examples include Claude, OpenAI-based clients).

• How is authentication handled?

The MCP server uses your Crawlab API token (configured in .env) to authenticate with the Crawlab API. Secure the MCP endpoint in your deployment.

• Can I run it in Docker/compose?

Yes. The repo includes Dockerfile examples and a sample docker-compose service snippet to integrate the MCP server with an existing Crawlab compose setup.

• Is it production-ready?

The project provides deployment options (Docker, Compose) but production readiness depends on your security, scaling and HA requirements; ensure you secure the MCP endpoint and credentials before production use.