Tagged “web-scraping”

Open-source browser agent project. Its browser-use CLI lets Claude Code, Codex and other agents drive your logged-in local Chrome with Python snippets, or connect to a cloud browser; also ships a Python agent library. The cloud browser needs an API key.

Controls your local Chrome over CDP to navigate, click, type, take screenshots and scrape data, reusing logged-in sessions. Official browser-use skill; install the browser-use CLI with uv first.

Local MCP server built into Browser Use: lets agents navigate, click, type and extract page content, or hand multi-step tasks to the built-in browser agent. Bring your own OpenAI or Anthropic API key.

Official MCP reference server that fetches web pages and converts them to Markdown for the model, reading long pages in chunks from a given offset. Zero config, but it can reach internal network addresses, so use it with care.

MCP server built into the adaptive Python scraping framework Scrapling: supports plain HTTP requests, browser rendering and stealth fetching, and uses CSS selectors to pass only the target content to the AI, saving tokens and filtering out prompt injection.

Adaptive web scraping framework: scrapling extract turns a page into Markdown, text or HTML in one command, with CSS selectors and browser rendering for dynamic pages, plus an MCP server (needs the ai extra). Needs Python 3.10+ and a browser install on first use.

Open-source, LLM-friendly web crawler that turns pages into clean Markdown for RAG and agents. Ships the crwl CLI and supports deep crawling and question-driven data extraction (extraction needs an LLM key). Free to self-host.

Job search system that runs in your local AI coding CLI: scans job portals, scores roles with a structured A-H evaluation and a 1-5 rating, flags fake listings, tailors ATS resumes and cover letters, and tracks applications. Never submits for you. Needs Node.js.

Give it competitor URLs and it scrapes homepages, pricing, feature pages and third-party reviews, adds SEO and traffic data, and writes standardized profiles of positioning, features, pricing, customers and differentiators. Needs Firecrawl and DataForSEO MCPs with API keys.

Browser automation CLI built for AI agents in native Rust: open pages, click and fill forms by refs from accessibility tree snapshots, read text and take screenshots, with compact snapshots and output limits to save context. From Vercel Labs; reuses local Chrome or installs one.

Browser automation plugin from Vercel Labs: agents drive Chrome through accessibility tree snapshots and element refs to navigate, click, fill forms and take screenshots, for web testing and exploratory QA. Install the agent-browser CLI first (npm, Homebrew or Cargo).

Browser automation MCP server open-sourced by ByteDance in the UI-TARS-desktop repo: drives pages through Puppeteer accessibility data with an optional vision mode, running locally over stdio or remotely over SSE or Streamable HTTP.

Headless browser written from scratch in Zig for AI agents and automation, using less memory and running faster than headless Chrome in official benchmarks. fetch turns pages into Markdown, and it can serve CDP or MCP. Telemetry is on by default (an env var disables it); Windows needs WSL.

Turns websites and Electron desktop apps into deterministic CLIs, with built-in adapters for Reddit, Hacker News, Bilibili, Zhihu, Xiaohongshu and more, and lets agents operate any page through your signed-in Chrome. Needs its companion browser extension.

Skyvern's official MCP: drives a browser with vision LLMs to complete multi-step tasks like filling forms, downloading files and web research. Connect to Skyvern Cloud (OAuth or API key), or self-host Skyvern and run it as a local process.

Bundles a whole web page with its CSS, images, fonts and scripts into a single HTML file that opens offline exactly as it looked, good for archiving pages or handing them to an agent for offline analysis. It does not run JavaScript, so pre-render dynamic pages with a headless browser.

Turns 18 kinds of sources, including doc sites, GitHub repos, PDFs and videos, into skill packages for Claude and other platforms, or exports RAG chunks and vector store data. Optional MCP server, and it can call a local agent like Claude Code to enrich content. Needs Python 3.10+.

A Chrome extension that hands the browser you use every day to your agent, keeping your logins and settings, with screenshots, network monitoring, bookmarks and history, and semantic search over tab content. Install the extension and mcp-chrome-bridge first.

Lets agents call thousands of ready-made scrapers and automation Actors from Apify Store to pull data from social media, search engines, maps, e-commerce sites and more. Maintained by Apify; needs an Apify account (OAuth) or API token.

Extracts the main content of a web page, stripping comments, sidebars, headers and footers, and outputs clean HTML or Markdown with metadata like title, author and publish date, saving tokens when agents read the web. Built for Obsidian Web Clipper by kepano; still in development.

Scrapes web pages into Markdown or structured data, with full-site crawling, sitemaps, web search and interaction with dynamic pages. Maintained by Firecrawl; the cloud needs an API key (self-hosting is possible), and the keyless hosted endpoint only offers scrape, search and parse.

Python library and CLI for extracting main web content: fetches pages, crawls via sitemaps and RSS feeds, strips headers, footers and other noise, and outputs TXT, Markdown, JSON, XML and more with metadata. Also processes downloaded HTML offline.

One command extracts or summarizes web pages, PDFs, YouTube videos, podcasts and media files, with JSON output. Summaries can use many model APIs, local models or a logged-in Codex or Claude CLI; plain extraction needs no key. Also has a browser sidebar extension.

Community Playwright browser automation MCP from ExecuteAutomation (not Microsoft's official one): navigate, click, fill forms, screenshot and scrape page text, plus send HTTP requests, assert network responses and record actions into test code, with 143 device emulations built in.