Data engineering向けのツール

Converts PDF, Word, Excel, PowerPoint, HTML, images, audio and more into Markdown that keeps headings, lists and tables, ready for LLMs to read and analyze. Maintained by Microsoft; requires Python 3.10+.

In-process analytical SQL database. Its CLI queries CSV, Parquet and JSON files directly with SQL, no import or server needed, so agents can sample and aggregate large files. Officially maintained.

Google's open-source database MCP server with ready-made query and SQL tools for dozens of sources, including PostgreSQL, MySQL, BigQuery, Spanner and MongoDB. Define controlled custom queries in tools.yaml.

Baidu PaddlePaddle's official OCR and document parsing tool: the paddleocr command reads text from images and parses PDFs and scans into Markdown or JSON, supporting 100+ languages with models that run locally.

Astral's extremely fast Python package and project manager written in Rust, replacing pip, pipx, poetry, pyenv and more. Supports lockfiles and multiple Python versions, and uvx runs Python tools without installing them.

MCP server built into the adaptive Python scraping framework Scrapling: supports plain HTTP requests, browser rendering and stealth fetching, and uses CSS selectors to pass only the target content to the AI, saving tokens and filtering out prompt injection.

Adaptive web scraping framework: scrapling extract turns a page into Markdown, text or HTML in one command, with CSS selectors and browser rendering for dynamic pages, plus an MCP server (needs the ai extra). Needs Python 3.10+ and a browser install on first use.

Open-source, LLM-friendly web crawler that turns pages into clean Markdown for RAG and agents. Ships the crwl CLI and supports deep crawling and question-driven data extraction (extraction needs an LLM key). Free to self-host.

Document parser from OpenDataLab that turns PDFs, images and Office files into Markdown or JSON, handling tables, formulas and complex layouts well, with page-by-page reading and citation anchors for agents. Parses locally by default; needs Python 3.10+.

Classic open-source OCR engine and CLI that reads text from images and outputs plain text, hOCR, searchable PDF or TSV. Supports 100+ languages; install the language data for each language you need.

Parses PDF, DOCX, PPTX, XLSX, HTML, images and audio, recovering layout, reading order, tables and formulas, and exports Markdown, HTML or lossless JSON. Started by IBM Research, now hosted by the LF AI & Data Foundation; runs fully locally.

'rsync for cloud storage': sync and copy files across dozens of providers such as S3, Google Drive, OneDrive and Alibaba Cloud OSS, with encryption and mounting. Set up remotes with rclone config before use.

Lightweight multi-protocol command-line downloader that fetches in parallel segments from multiple sources over HTTP(S), FTP, SFTP, BitTorrent and Metalink. Resumes downloads, reads URLs in bulk from text files and offers a JSON-RPC interface for scripts or agents.

Data engineering plugin from the wshobson/agents marketplace: a data engineer subagent, a data pipeline command, and skills for Airflow DAGs, dbt transformations, Spark tuning and data quality, to help build ETL and data warehouses.

MLOps plugin from the wshobson/agents marketplace: data scientist, ML engineer and MLOps engineer subagents and a /ml-pipeline command that runs the pipeline in phases from data analysis and training to deployment and monitoring, plus ML pipeline and recommender system skills.

Cross-platform shell built around structured data: pipelines pass tables instead of plain text, and you can open and query JSON, CSV, TOML and more directly. Default builds include nu --mcp, which serves it to agents as an MCP server.

Converts PDFs, images, PPTX, DOCX, XLSX, EPUB and more into Markdown, JSON or HTML, preserving tables, formulas and layout, with optional LLM help for higher accuracy. Needs Python 3.10+ and PyTorch; large-scale commercial use of the model weights needs a paid license.

Connects Claude Code to the Pyright language server for static type checking and code intelligence in Python. Install pyright first with npm, pip or pipx. Official Anthropic plugin.

Headless browser written from scratch in Zig for AI agents and automation, using less memory and running faster than headless Chrome in official benchmarks. fetch turns pages into Markdown, and it can serve CDP or MCP. Telemetry is on by default (an env var disables it); Windows needs WSL.

Lightweight command-line JSON processor that slices, filters, maps and transforms JSON with concise expressions and no runtime dependencies. A staple for agents parsing API responses and logs.

Data analysis plugin: writes queries in each SQL dialect, explores datasets, runs statistical tests, charts with Python and builds interactive HTML dashboards, then reviews methods and conclusions before sharing. Connects to Snowflake, BigQuery and other warehouses. Official from Anthropic.

Anthropic's official data skill that guides agents to write, translate and optimize SQL across Snowflake, BigQuery, Databricks, PostgreSQL and other warehouse dialects, including CTEs, window functions and analytical aggregations.

Manages BigQuery datasets, tables and jobs with the bq and gcloud CLIs, runs SQL queries and loads data, with references for client libraries, MCP and Terraform. Google's official skill; requires a Google Cloud project.

Google's official skill library, focused on Google Cloud: 100+ skills for gcloud, Cloud Run, GKE, BigQuery, IAM, monitoring and logging, and the Well-Architected Framework, plus Ads and Analytics.