# Tools > Every listed resource, editors’ picks first. The filters below can be combined. - Search: append `?q=` to the URL. Separate several keywords with spaces; all of them must match. Matches name, repository, tags, professions, description and README. - Filter by type: append `?type=` to the URL. - Filter by profession: append `?profession=` to the URL. Separate several ids with commas; a resource matches when it suits any of them. Ids are listed at https://hebbmind.com/api/v1/professions - Filter by tag: append `?tag=` to the URL. Separate several tags with commas; a resource must carry all of them. Tags are listed at https://hebbmind.com/api/v1/tags Tagged `web-scraping`: 52 items. - [Browser Use](https://hebbmind.com/ru/cli/browser-use.md): [CLI] Open-source browser agent project. Its browser-use CLI lets Claude Code, Codex and other agents drive your logged-in local Chrome with Python snippets, or connect to a cloud browser; also ships a Python agent library. The cloud browser needs an API key. - [Browser Use](https://hebbmind.com/ru/skill/browser-use.md): [Skill] Controls your local Chrome over CDP to navigate, click, type, take screenshots and scrape data, reusing logged-in sessions. Official browser-use skill; install the browser-use CLI with uv first. - [Browser Use MCP](https://hebbmind.com/ru/mcp/browser-use-mcp.md): [MCP] Local MCP server built into Browser Use: lets agents navigate, click, type and extract page content, or hand multi-step tasks to the built-in browser agent. Bring your own OpenAI or Anthropic API key. - [Fetch MCP Server](https://hebbmind.com/ru/mcp/fetch.md): [MCP] Official MCP reference server that fetches web pages and converts them to Markdown for the model, reading long pages in chunks from a given offset. Zero config, but it can reach internal network addresses, so use it with care. - [Scrapling](https://hebbmind.com/ru/mcp/scrapling.md): [MCP] MCP server built into the adaptive Python scraping framework Scrapling: supports plain HTTP requests, browser rendering and stealth fetching, and uses CSS selectors to pass only the target content to the AI, saving tokens and filtering out prompt injection. - [Scrapling](https://hebbmind.com/ru/cli/scrapling.md): [CLI] Adaptive web scraping framework: scrapling extract turns a page into Markdown, text or HTML in one command, with CSS selectors and browser rendering for dynamic pages, plus an MCP server (needs the ai extra). Needs Python 3.10+ and a browser install on first use. - [Crawl4AI](https://hebbmind.com/ru/cli/crawl4ai.md): [CLI] Open-source, LLM-friendly web crawler that turns pages into clean Markdown for RAG and agents. Ships the crwl CLI and supports deep crawling and question-driven data extraction (extraction needs an LLM key). Free to self-host. - [career-ops](https://hebbmind.com/ru/plugin/career-ops.md): [Plugin] Job search system that runs in your local AI coding CLI: scans job portals, scores roles with a structured A-H evaluation and a 1-5 rating, flags fake listings, tailors ATS resumes and cover letters, and tracks applications. Never submits for you. Needs Node.js. - [Competitor Profiling](https://hebbmind.com/ru/skill/competitor-profiling.md): [Skill] Give it competitor URLs and it scrapes homepages, pricing, feature pages and third-party reviews, adds SEO and traffic data, and writes standardized profiles of positioning, features, pricing, customers and differentiators. Needs Firecrawl and DataForSEO MCPs with API keys. - [agent-browser](https://hebbmind.com/ru/cli/agent-browser.md): [CLI] Browser automation CLI built for AI agents in native Rust: open pages, click and fill forms by refs from accessibility tree snapshots, read text and take screenshots, with compact snapshots and output limits to save context. From Vercel Labs; reuses local Chrome or installs one. - [agent-browser](https://hebbmind.com/ru/plugin/agent-browser.md): [Plugin] Browser automation plugin from Vercel Labs: agents drive Chrome through accessibility tree snapshots and element refs to navigate, click, fill forms and take screenshots, for web testing and exploratory QA. Install the agent-browser CLI first (npm, Homebrew or Cargo). - [@agent-infra/mcp-server-browser](https://hebbmind.com/ru/mcp/browser.md): [MCP] Browser automation MCP server open-sourced by ByteDance in the UI-TARS-desktop repo: drives pages through Puppeteer accessibility data with an optional vision mode, running locally over stdio or remotely over SSE or Streamable HTTP. - [Lightpanda Browser](https://hebbmind.com/ru/cli/browser.md): [CLI] Headless browser written from scratch in Zig for AI agents and automation, using less memory and running faster than headless Chrome in official benchmarks. fetch turns pages into Markdown, and it can serve CDP or MCP. Telemetry is on by default (an env var disables it); Windows needs WSL. - [OpenCLI](https://hebbmind.com/ru/cli/opencli.md): [CLI] Turns websites and Electron desktop apps into deterministic CLIs, with built-in adapters for Reddit, Hacker News, Bilibili, Zhihu, Xiaohongshu and more, and lets agents operate any page through your signed-in Chrome. Needs its companion browser extension. - [Skyvern MCP](https://hebbmind.com/ru/mcp/skyvern-ai-mcp.md): [MCP] Skyvern's official MCP: drives a browser with vision LLMs to complete multi-step tasks like filling forms, downloading files and web research. Connect to Skyvern Cloud (OAuth or API key), or self-host Skyvern and run it as a local process. - [monolith](https://hebbmind.com/ru/cli/monolith.md): [CLI] Bundles a whole web page with its CSS, images, fonts and scripts into a single HTML file that opens offline exactly as it looked, good for archiving pages or handing them to an agent for offline analysis. It does not run JavaScript, so pre-render dynamic pages with a headless browser. - [Skill Seekers](https://hebbmind.com/ru/cli/skill-seekers.md): [CLI] Turns 18 kinds of sources, including doc sites, GitHub repos, PDFs and videos, into skill packages for Claude and other platforms, or exports RAG chunks and vector store data. Optional MCP server, and it can call a local agent like Claude Code to enrich content. Needs Python 3.10+. - [Chrome MCP Server](https://hebbmind.com/ru/mcp/mcp-chrome.md): [MCP] A Chrome extension that hands the browser you use every day to your agent, keeping your logins and settings, with screenshots, network monitoring, bookmarks and history, and semantic search over tab content. Install the extension and mcp-chrome-bridge first. - [Apify MCP Server](https://hebbmind.com/ru/mcp/apify-mcp-server.md): [MCP] Lets agents call thousands of ready-made scrapers and automation Actors from Apify Store to pull data from social media, search engines, maps, e-commerce sites and more. Maintained by Apify; needs an Apify account (OAuth) or API token. - [Defuddle](https://hebbmind.com/ru/cli/defuddle.md): [CLI] Extracts the main content of a web page, stripping comments, sidebars, headers and footers, and outputs clean HTML or Markdown with metadata like title, author and publish date, saving tokens when agents read the web. Built for Obsidian Web Clipper by kepano; still in development. - [Firecrawl MCP Server](https://hebbmind.com/ru/mcp/firecrawl-mcp-server.md): [MCP] Scrapes web pages into Markdown or structured data, with full-site crawling, sitemaps, web search and interaction with dynamic pages. Maintained by Firecrawl; the cloud needs an API key (self-hosting is possible), and the keyless hosted endpoint only offers scrape, search and parse. - [Trafilatura](https://hebbmind.com/ru/cli/trafilatura.md): [CLI] Python library and CLI for extracting main web content: fetches pages, crawls via sitemaps and RSS feeds, strips headers, footers and other noise, and outputs TXT, Markdown, JSON, XML and more with metadata. Also processes downloaded HTML offline. - [Summarize](https://hebbmind.com/ru/cli/summarize.md): [CLI] One command extracts or summarizes web pages, PDFs, YouTube videos, podcasts and media files, with JSON output. Summaries can use many model APIs, local models or a logged-in Codex or Claude CLI; plain extraction needs no key. Also has a browser sidebar extension. - [ExecuteAutomation Playwright MCP Server](https://hebbmind.com/ru/mcp/mcp-playwright.md): [MCP] Community Playwright browser automation MCP from ExecuteAutomation (not Microsoft's official one): navigate, click, fill forms, screenshot and scrape page text, plus send HTTP requests, assert network responses and record actions into test code, with 143 device emulations built in. - [Exa MCP Server](https://hebbmind.com/ru/mcp/exa-mcp-server.md): [MCP] Gives agents Exa web search and page content fetching, with an optional multi-step research agent and advanced filtered search. Maintained by Exa; the hosted service works anonymously with limits, and signing in or adding an API key raises them. - [Social Media Research Skills](https://hebbmind.com/ru/skill/social-media-research-skills.md): [Skill] ScrapeCreators' official skills: mine public TikTok, Instagram, YouTube, Reddit and X data for viral content, customer voice in comments, competitor accounts and ad libraries, and creator vetting, with sourced reports. Needs a ScrapeCreators key (100 free credits, then pay as you go). - [Browserbase Skills](https://hebbmind.com/ru/skill/browserbase-skills.md): [Skill] Browserbase's official skills: drive a local or cloud browser with the browse CLI for web scraping and search, turning site traffic into an OpenAPI spec, adversarial UI testing and competitor research. Cloud features need a Browserbase account. - [Dembrandt](https://hebbmind.com/ru/cli/dembrandt.md): [CLI] Extracts any website's logo, colors, fonts, spacing, shadows and motion in one command, exporting W3C design tokens, a Tailwind theme, DESIGN.md or a brand guide PDF, plus WCAG contrast checks. Good for competitor and brand audits. Needs Node.js 18+; run dembrandt install-browser first. - [Markdownify MCP Server](https://hebbmind.com/ru/mcp/markdownify-mcp.md): [MCP] Converts PDF, Word, Excel, PowerPoint, images, audio, web pages and YouTube transcripts into Markdown that agents can read and cite. Built on Microsoft MarkItDown; the Docker image lacks audio transcription and image recognition, so build locally for full features. - [Bright Data MCP](https://hebbmind.com/ru/mcp/brightdata-mcp.md): [MCP] Bright Data's official MCP: web search (Google, Bing, Yandex), any page to Markdown, structured data extraction from e-commerce, social and other platforms, and remote browser control. Needs an API token; includes 5,000 free requests a month. - [shot-scraper](https://hebbmind.com/ru/cli/shot-scraper.md): [CLI] Simon Willison's website screenshot CLI: batch screenshots from YAML, record demo videos, export PDF and HTML, run JavaScript in pages to extract data, and dump accessibility trees. Built on Playwright; install a browser on first use. - [Apify Agent Skills](https://hebbmind.com/ru/skill/apify-agent-skills.md): [Skill] Apify's official skill collection: picks from 130+ curated Actors to scrape public data from social networks, maps, e-commerce sites and more, and can also build, package and deploy Actors. Needs an Apify account; a free tier is available. - [Tavily MCP Server](https://hebbmind.com/ru/mcp/tavily-mcp.md): [MCP] Four tools for real-time web search, page content extraction, site mapping and full-site crawling, available as a remote server or run locally. Maintained by Tavily; needs an API key or OAuth sign-in, and a free account is available. - [Open-WebSearch](https://hebbmind.com/ru/mcp/open-websearch.md): [MCP] Multi-engine web search MCP with no API key, supporting Bing, DuckDuckGo, Brave, Baidu, Sogou, CSDN, Juejin and more; also fetches page content, CSDN articles and GitHub READMEs. Ships a CLI and a local server too. - [DuckDuckGo Search MCP Server](https://hebbmind.com/ru/mcp/duckduckgo-mcp-server.md): [MCP] Web search through DuckDuckGo plus fetching and parsing page content, with built-in rate limiting and LLM-friendly result formatting. No API key needed; runs with a single uvx command. - [Python SEO Analyzer](https://hebbmind.com/ru/cli/python-seo-analyzer.md): [CLI] Crawls an entire site, counts words and keywords per page, and checks technical SEO such as titles, meta descriptions, image alt text, H1s and social tags. Outputs JSON by default or an HTML report for agents to act on. Needs Python; the optional AI content review needs an API key. - [daymade Claude Code Skills](https://hebbmind.com/ru/plugin/claude-code-skills.md): [Plugin] A plugin marketplace with dozens of plugins: an enhanced skill-creator, scrapers for Bilibili, Douban and Feishu docs, China A-share news and US stock fundamentals, WeCom notifications, deep research and fact-checking, with docs in English and Chinese. - [SearXNG MCP Server](https://hebbmind.com/ru/mcp/mcp-searxng.md): [MCP] Web search and page reading for agents through a self-hosted or trusted SearXNG metasearch instance, with no paid search API key. Requires a SearXNG instance with JSON output enabled. - [Web Clone](https://hebbmind.com/ru/skill/web-clone.md): [Skill] Clones websites: find the real source first or run browser recon, pick a path for static, React/Vue/Next or WebGL/Canvas/Three.js sites, strip trackers, check licenses, and verify against the original before swapping in your content. In Chinese; script paths assume the author's machine. - [Jina AI MCP Server](https://hebbmind.com/ru/mcp/jina-ai-mcp.md): [MCP] Jina AI's official remote MCP: converts web pages and PDFs to Markdown, searches the web, arXiv, SSRN and images, and offers screenshots, reranking and semantic dedup. Reading tools work without a key (rate-limited); search and rerank need a Jina API key. - [Firecrawl](https://hebbmind.com/ru/skill/firecrawl.md): [Skill] Official Firecrawl skill: drives the Firecrawl CLI to search, scrape, map site URLs, batch-crawl, interact with pages and monitor changes, returning LLM-ready Markdown. Search and scrape have a rate-limited free tier; crawling and monitoring need an account or self-hosted instance. - [Firecrawl CLI](https://hebbmind.com/ru/cli/firecrawl-cli.md): [CLI] Firecrawl's official CLI: scrape, search, map and crawl sites and interact with pages in a browser from the terminal, with Markdown output; can also install companion skills for coding agents. Needs a Firecrawl account or API key, or point it at a self-hosted instance. - [Knowledge Absorber](https://hebbmind.com/ru/skill/studyanalysis-skills.md): [Skill] Turns web pages, long PDFs, Word files, images or code into tutor-style study notes: Markdown notes plus searchable HTML knowledge cards with concept breakdowns and mind maps. Needs Python 3.8+, plus Chrome for some sites; the optional tutor chat needs an OpenAI-compatible API key. - [YouTube Transcript Server](https://hebbmind.com/ru/mcp/mcp-server-youtube-transcript.md): [MCP] Fetches captions and transcripts for YouTube videos and Shorts, with language selection and fallback, optional timestamps and sponsor segments filtered by default. No API key for transcripts; the optional visual analysis tool needs a TwelveLabs API key and a direct video URL. - [Learn This (Tapestry Skills)](https://hebbmind.com/ru/skill/learn-this.md): [Skill] Give it a YouTube video, web article or PDF link and it extracts the transcript or text, then builds a five-round Ship-Learn-Next action plan saved as Markdown, for self-study. Needs CLI tools such as yt-dlp and pdftotext; videos default to English subtitles. - [Tavily Agent Skills](https://hebbmind.com/ru/skill/tavily-ai-skills.md): [Skill] Official Tavily skills: let agents search the web, extract page content, discover site URLs, crawl in bulk and run cited deep research through the Tavily CLI (tvly). Install the CLI first; search and extract have a limited no-login tier, other features need sign-in or an API key. - [Firecrawl Plugin for Claude Code](https://hebbmind.com/ru/plugin/firecrawl-claude-plugin.md): [Plugin] Firecrawl's official plugin: brings the Firecrawl CLI in as skills so Claude Code can search the web, scrape pages to Markdown, map site links or crawl whole sites. Install firecrawl-cli and set an API key first; self-hosting supported. - [Letterboxd Diary](https://hebbmind.com/ru/skill/letterboxd-diary.md): [Skill] Fetches recently watched films from a Letterboxd member's public diary RSS feed and renders them as a compact Markdown list, with filters by date or count and ratings. Asks for the username on first run and saves it to an env var or memory file if you agree. - [Firecrawl Competitive Intel](https://hebbmind.com/ru/skill/firecrawl-competitive-intel.md): [Skill] Official Firecrawl skill that scrapes competitors' pricing, feature and changelog pages, extracts plan prices, feature lists and recent changes, and produces a sourced competitive-intel report with change alerts, easy to rerun on a schedule. Needs a Firecrawl API key and its CLI or MCP. - [Firecrawl SEO Audit](https://hebbmind.com/ru/skill/firecrawl-seo-audit.md): [Skill] Official Firecrawl skill: crawls a site and its key pages, checks titles, descriptions, heading structure, internal links and keyword opportunities, compares top-ranking competitors, and outputs a prioritized SEO audit. Needs a Firecrawl API key (free: 1,000 credits/month) and its CLI. Next page: https://hebbmind.com/ru/tools.md?page=2&tag=web-scraping