Tools

In-process analytical SQL database. Its CLI queries CSV, Parquet and JSON files directly with SQL, no import or server needed, so agents can sample and aggregate large files. Officially maintained.

Adaptive web scraping framework: scrapling extract turns a page into Markdown, text or HTML in one command, with CSS selectors and browser rendering for dynamic pages, plus an MCP server (needs the ai extra). Needs Python 3.10+ and a browser install on first use.

Open-source, LLM-friendly web crawler that turns pages into clean Markdown for RAG and agents. Ships the crwl CLI and supports deep crawling and question-driven data extraction (extraction needs an LLM key). Free to self-host.

Command-line edition of the open-source OpenBB investment research platform: query stock prices, financial statements, macroeconomic and other financial data from the terminal, and automate data pulls with Routine scripts. Officially maintained; some sources need API keys.

Cross-platform shell built around structured data: pipelines pass tables instead of plain text, and you can open and query JSON, CSV, TOML and more directly. Default builds include nu --mcp, which serves it to agents as an MCP server.

Turns websites and Electron desktop apps into deterministic CLIs, with built-in adapters for Reddit, Hacker News, Bilibili, Zhihu, Xiaohongshu and more, and lets agents operate any page through your signed-in Chrome. Needs its companion browser extension.

Alibaba's open-source offline data sync tool: JSON job configs batch-sync between heterogeneous sources like MySQL, Oracle, PostgreSQL, Hive, HDFS and MaxCompute, run with python datax.py. Needs JDK 1.8+ and Python; download and unpack the official toolkit.

Command-line framework for warehouse modeling and ELT transformations: write models in SQL and let dbt handle dependencies, data tests and documentation. v2 is rewritten in Rust as a single binary. Maintained by dbt Labs; needs a data warehouse connection.

PostgreSQL command-line client with smart autocompletion and syntax highlighting. Compatible with psql environment variables, and -c runs a statement and exits, so agents can query databases non-interactively.

Simon Willison's LLM CLI and Python library: call OpenAI, Claude, Gemini and local models from the terminal, with conversations logged to SQLite automatically, plus embeddings, structured extraction, tool calling and a plugin system.

MySQL terminal client with smart autocompletion and syntax highlighting, also compatible with MariaDB and TiDB. Supports --execute and batch mode with CSV or table output, so agents can run SQL directly.

Publishes SQLite databases as browsable, queryable web pages and JSON APIs, with SQL queries, metadata and plugins, and deploys to Cloud Run and other platforms with datasette publish. Great for data exploration and data journalism. Needs Python 3.10+.

Reads the exact position of every character, line and table cell in digital PDFs: the CLI exports layout-preserving text or CSV/JSON details, and the Python API extracts tables and form fields, ideal for invoices, statements and reports. Runs locally with Python; scans need OCR first.

Universal psql-style SQL command line: one tool to connect to PostgreSQL, MySQL, SQLite, SQL Server, Oracle and dozens of other SQL and NoSQL databases, with support for copying data between databases.

Processes CSV, TSV, JSON and other structured data like awk, sed, cut, join and sort, but by field name, for cleaning, aggregation, format conversion and pretty-printing. The command is mlr; written in Go with no runtime dependencies.

Linter and formatter for 20+ SQL dialects that auto-fixes most issues and handles Jinja templating and dbt projects (dbt needs a plugin), suited to warehouse code review and CI.

Terminal tool for exploring and wrangling tabular data: opens dozens of formats including CSV, JSON, Excel and SQLite for filtering, sorting, pivots and frequency counts. --batch mode replays actions headlessly or converts file formats.

Query cloud resources and SaaS APIs live with SQL: plugins map AWS, Azure, GCP, GitHub, Kubernetes and other services to tables you can select from directly, no ETL needed. A single binary with Postgres built in.

Converts and two-way syncs Jupyter notebooks with plain Python scripts or Markdown, so agents can edit research and teaching notebooks like ordinary text and diffs stay readable. Needs a local Python environment; install with pip.

Python library and CLI for extracting main web content: fetches pages, crawls via sitemaps and RSS feeds, strips headers, footers and other noise, and outputs TXT, Markdown, JSON, XML and more with metadata. Also processes downloaded HTML offline.

Veteran suite of CSV command-line tools: in2csv converts Excel and JSON to CSV, csvcut, csvgrep and csvsort filter and sort, csvstat computes statistics, and csvsql runs SQL directly against CSV files.

Posit's open-source scientific and technical publishing system built on Pandoc: embed Python, R or Julia code in Markdown or Jupyter notebooks, then quarto render produces HTML, PDF, Word, slides, websites and books.

Draws bar charts, histograms, line charts, scatter plots and box plots in the terminal from CSV/TSV on stdin, so you can glance at data distributions at the end of a shell pipeline. The command is uplot.

Knownsec's open-source Python-use agent: describe a task in plain language and it writes and runs Python on your machine, e.g. tidying Excel files, summarizing attendance or screening resumes. Needs Python 3.11+ and a model API key such as DeepSeek; code runs locally, so back up important files.