# Tools > Every listed resource, editors’ picks first. The filters below can be combined. - Search: append `?q=` to the URL. Separate several keywords with spaces; all of them must match. Matches name, repository, tags, professions, description and README. - Filter by type: append `?type=` to the URL. - Filter by profession: append `?profession=` to the URL. Separate several ids with commas; a resource matches when it suits any of them. Ids are listed at https://hebbmind.com/api/v1/professions - Filter by tag: append `?tag=` to the URL. Separate several tags with commas; a resource must carry all of them. Tags are listed at https://hebbmind.com/api/v1/tags Resources for Full-stack / Indie or ML / AI, Tagged `web-scraping`: 33 items. > - Full-stack / Indie: General software work: coding-agent workflows, code review, debugging and Git collaboration. > - ML / AI: Model training, evaluation and LLM application development. - [Browser Use](https://hebbmind.com/cli/browser-use.md): [CLI] Open-source browser agent project. Its browser-use CLI lets Claude Code, Codex and other agents drive your logged-in local Chrome with Python snippets, or connect to a cloud browser; also ships a Python agent library. The cloud browser needs an API key. - [Browser Use](https://hebbmind.com/skill/browser-use.md): [Skill] Controls your local Chrome over CDP to navigate, click, type, take screenshots and scrape data, reusing logged-in sessions. Official browser-use skill; install the browser-use CLI with uv first. - [Browser Use MCP](https://hebbmind.com/mcp/browser-use-mcp.md): [MCP] Local MCP server built into Browser Use: lets agents navigate, click, type and extract page content, or hand multi-step tasks to the built-in browser agent. Bring your own OpenAI or Anthropic API key. - [Scrapling](https://hebbmind.com/cli/scrapling.md): [CLI] Adaptive web scraping framework: scrapling extract turns a page into Markdown, text or HTML in one command, with CSS selectors and browser rendering for dynamic pages, plus an MCP server (needs the ai extra). Needs Python 3.10+ and a browser install on first use. - [Scrapling](https://hebbmind.com/mcp/scrapling.md): [MCP] MCP server built into the adaptive Python scraping framework Scrapling: supports plain HTTP requests, browser rendering and stealth fetching, and uses CSS selectors to pass only the target content to the AI, saving tokens and filtering out prompt injection. - [Fetch MCP Server](https://hebbmind.com/mcp/fetch.md): [MCP] Official MCP reference server that fetches web pages and converts them to Markdown for the model, reading long pages in chunks from a given offset. Zero config, but it can reach internal network addresses, so use it with care. - [Crawl4AI](https://hebbmind.com/cli/crawl4ai.md): [CLI] Open-source, LLM-friendly web crawler that turns pages into clean Markdown for RAG and agents. Ships the crwl CLI and supports deep crawling and question-driven data extraction (extraction needs an LLM key). Free to self-host. - [career-ops](https://hebbmind.com/plugin/career-ops.md): [Plugin] Job search system that runs in your local AI coding CLI: scans job portals, scores roles with a structured A-H evaluation and a 1-5 rating, flags fake listings, tailors ATS resumes and cover letters, and tracks applications. Never submits for you. Needs Node.js. - [agent-browser](https://hebbmind.com/plugin/agent-browser.md): [Plugin] Browser automation plugin from Vercel Labs: agents drive Chrome through accessibility tree snapshots and element refs to navigate, click, fill forms and take screenshots, for web testing and exploratory QA. Install the agent-browser CLI first (npm, Homebrew or Cargo). - [agent-browser](https://hebbmind.com/cli/agent-browser.md): [CLI] Browser automation CLI built for AI agents in native Rust: open pages, click and fill forms by refs from accessibility tree snapshots, read text and take screenshots, with compact snapshots and output limits to save context. From Vercel Labs; reuses local Chrome or installs one. - [Lightpanda Browser](https://hebbmind.com/cli/browser.md): [CLI] Headless browser written from scratch in Zig for AI agents and automation, using less memory and running faster than headless Chrome in official benchmarks. fetch turns pages into Markdown, and it can serve CDP or MCP. Telemetry is on by default (an env var disables it); Windows needs WSL. - [@agent-infra/mcp-server-browser](https://hebbmind.com/mcp/browser.md): [MCP] Browser automation MCP server open-sourced by ByteDance in the UI-TARS-desktop repo: drives pages through Puppeteer accessibility data with an optional vision mode, running locally over stdio or remotely over SSE or Streamable HTTP. - [OpenCLI](https://hebbmind.com/cli/opencli.md): [CLI] Turns websites and Electron desktop apps into deterministic CLIs, with built-in adapters for Reddit, Hacker News, Bilibili, Zhihu, Xiaohongshu and more, and lets agents operate any page through your signed-in Chrome. Needs its companion browser extension. - [Skyvern MCP](https://hebbmind.com/mcp/skyvern-ai-mcp.md): [MCP] Skyvern's official MCP: drives a browser with vision LLMs to complete multi-step tasks like filling forms, downloading files and web research. Connect to Skyvern Cloud (OAuth or API key), or self-host Skyvern and run it as a local process. - [Skill Seekers](https://hebbmind.com/cli/skill-seekers.md): [CLI] Turns 18 kinds of sources, including doc sites, GitHub repos, PDFs and videos, into skill packages for Claude and other platforms, or exports RAG chunks and vector store data. Optional MCP server, and it can call a local agent like Claude Code to enrich content. Needs Python 3.10+. - [Apify MCP Server](https://hebbmind.com/mcp/apify-mcp-server.md): [MCP] Lets agents call thousands of ready-made scrapers and automation Actors from Apify Store to pull data from social media, search engines, maps, e-commerce sites and more. Maintained by Apify; needs an Apify account (OAuth) or API token. - [Defuddle](https://hebbmind.com/cli/defuddle.md): [CLI] Extracts the main content of a web page, stripping comments, sidebars, headers and footers, and outputs clean HTML or Markdown with metadata like title, author and publish date, saving tokens when agents read the web. Built for Obsidian Web Clipper by kepano; still in development. - [Firecrawl MCP Server](https://hebbmind.com/mcp/firecrawl-mcp-server.md): [MCP] Scrapes web pages into Markdown or structured data, with full-site crawling, sitemaps, web search and interaction with dynamic pages. Maintained by Firecrawl; the cloud needs an API key (self-hosting is possible), and the keyless hosted endpoint only offers scrape, search and parse. - [Trafilatura](https://hebbmind.com/cli/trafilatura.md): [CLI] Python library and CLI for extracting main web content: fetches pages, crawls via sitemaps and RSS feeds, strips headers, footers and other noise, and outputs TXT, Markdown, JSON, XML and more with metadata. Also processes downloaded HTML offline. - [ExecuteAutomation Playwright MCP Server](https://hebbmind.com/mcp/mcp-playwright.md): [MCP] Community Playwright browser automation MCP from ExecuteAutomation (not Microsoft's official one): navigate, click, fill forms, screenshot and scrape page text, plus send HTTP requests, assert network responses and record actions into test code, with 143 device emulations built in. - [Exa MCP Server](https://hebbmind.com/mcp/exa-mcp-server.md): [MCP] Gives agents Exa web search and page content fetching, with an optional multi-step research agent and advanced filtered search. Maintained by Exa; the hosted service works anonymously with limits, and signing in or adding an API key raises them. - [Browserbase Skills](https://hebbmind.com/skill/browserbase-skills.md): [Skill] Browserbase's official skills: drive a local or cloud browser with the browse CLI for web scraping and search, turning site traffic into an OpenAPI spec, adversarial UI testing and competitor research. Cloud features need a Browserbase account. - [Tavily MCP Server](https://hebbmind.com/mcp/tavily-mcp.md): [MCP] Four tools for real-time web search, page content extraction, site mapping and full-site crawling, available as a remote server or run locally. Maintained by Tavily; needs an API key or OAuth sign-in, and a free account is available. - [Open-WebSearch](https://hebbmind.com/mcp/open-websearch.md): [MCP] Multi-engine web search MCP with no API key, supporting Bing, DuckDuckGo, Brave, Baidu, Sogou, CSDN, Juejin and more; also fetches page content, CSDN articles and GitHub READMEs. Ships a CLI and a local server too. - [DuckDuckGo Search MCP Server](https://hebbmind.com/mcp/duckduckgo-mcp-server.md): [MCP] Web search through DuckDuckGo plus fetching and parsing page content, with built-in rate limiting and LLM-friendly result formatting. No API key needed; runs with a single uvx command. - [daymade Claude Code Skills](https://hebbmind.com/plugin/claude-code-skills.md): [Plugin] A plugin marketplace with dozens of plugins: an enhanced skill-creator, scrapers for Bilibili, Douban and Feishu docs, China A-share news and US stock fundamentals, WeCom notifications, deep research and fact-checking, with docs in English and Chinese. - [SearXNG MCP Server](https://hebbmind.com/mcp/mcp-searxng.md): [MCP] Web search and page reading for agents through a self-hosted or trusted SearXNG metasearch instance, with no paid search API key. Requires a SearXNG instance with JSON output enabled. - [Jina AI MCP Server](https://hebbmind.com/mcp/jina-ai-mcp.md): [MCP] Jina AI's official remote MCP: converts web pages and PDFs to Markdown, searches the web, arXiv, SSRN and images, and offers screenshots, reranking and semantic dedup. Reading tools work without a key (rate-limited); search and rerank need a Jina API key. - [Firecrawl](https://hebbmind.com/skill/firecrawl.md): [Skill] Official Firecrawl skill: drives the Firecrawl CLI to search, scrape, map site URLs, batch-crawl, interact with pages and monitor changes, returning LLM-ready Markdown. Search and scrape have a rate-limited free tier; crawling and monitoring need an account or self-hosted instance. - [Firecrawl CLI](https://hebbmind.com/cli/firecrawl-cli.md): [CLI] Firecrawl's official CLI: scrape, search, map and crawl sites and interact with pages in a browser from the terminal, with Markdown output; can also install companion skills for coding agents. Needs a Firecrawl account or API key, or point it at a self-hosted instance. - [Tavily Agent Skills](https://hebbmind.com/skill/tavily-ai-skills.md): [Skill] Official Tavily skills: let agents search the web, extract page content, discover site URLs, crawl in bulk and run cited deep research through the Tavily CLI (tvly). Install the CLI first; search and extract have a limited no-login tier, other features need sign-in or an API key. - [Firecrawl Plugin for Claude Code](https://hebbmind.com/plugin/firecrawl-claude-plugin.md): [Plugin] Firecrawl's official plugin: brings the Firecrawl CLI in as skills so Claude Code can search the web, scrape pages to Markdown, map site links or crawl whole sites. Install firecrawl-cli and set an API key first; self-hosting supported. - [Browser Use](https://hebbmind.com/plugin/browser-use.md): [Plugin] Browser Use's official plugin: connects the Browser Use MCP so Claude can drive your local Chrome or a cloud browser to browse, extract data, fill forms, test sites and take screenshots. Needs uv; cloud browsers need sign-in or an API key.