Md Webcrawl Mcp

What is Md Webcrawl MCP?

Md Webcrawl MCP is a Python-based web crawler that implements the Model Context Protocol (MCP) to extract website content and save it as Markdown. It maps website structure and links, supports batch processing, and is designed to integrate with FastMCP for running as a crawl server.

How to use Md Webcrawl MCP?

• Install: clone the repo and install dependencies: ‘pip install -r requirements.txt’.
• Configure: set environment variables such as OUTPUT_PATH (default output dir), MAX_CONCURRENT_REQUESTS (default 5), and REQUEST_TIMEOUT (default 30s).
• Run with FastMCP: install/run the server via FastMCP (examples: ‘fastmcp install server.py’ or run with ‘fastmcp run /path/to/server.py’ and provide env vars).
• Example commands:
  • Extract content: ‘mcp call extract_content --url ‘https://example.com’ --output_path ‘example.md’’
  • Create index from linked content: ‘mcp call scan_linked_content --url ‘https://example.com’ | mcp call create_index --content_map - --output_path ‘index.md’’
• Development: use ‘fastmcp dev server.py --with-editable .’ for live development. Use the MCP inspector tool for debugging.

Key features

• Save crawled content as Markdown files
• Map site structure and internal links
• Batch processing of multiple URLs
• Configurable output directory and request settings (concurrency, timeout)
• FastMCP / MCP integration for running and calling handlers
• Debugging support via Model Context Protocol inspector

Use cases

1. Archiving website content as Markdown for knowledge bases
2. Building content indexes for search or LLM retrieval augmentation
3. Extracting and migrating content during site rebuilds or CMS migrations
4. SEO analysis and link-structure mapping
5. Automated data collection for research and content analysis

FAQ

• What are the requirements?

  • Python 3.7+, FastMCP, and dependencies from requirements.txt.

• Where are crawled files saved?

  • Files are saved in the directory specified by OUTPUT_PATH (defaults to ./output or your configured path).

• How do I change concurrency or timeouts?

  • Configure MAX_CONCURRENT_REQUESTS and REQUEST_TIMEOUT environment variables.

• How do I run it with FastMCP?

  • Use ‘fastmcp install server.py’ to install or ‘fastmcp run /path/to/server.py’ with appropriate env vars.

• Is there a way to debug crawls?

  • Yes — the project recommends using the MCP inspector (https://modelcontextprotocol.io/docs/tools/inspector) and FastMCP’s dev mode.

• What license is this under?

  • Distributed under the MIT License.