Etiqueta: model-evaluation

Guides the agent through writing or improving a skill: clarify intent, draft SKILL.md, run scored evals with and without the skill, then tune the description so it triggers accurately. Official Anthropic skill.

Creates or improves Agent Skills from scratch: writes eval cases to test them, runs benchmarks with variance analysis, and tunes descriptions for more accurate triggering. Maintained by Anthropic.

CLI and library for evaluating and red-teaming LLM apps: compare prompts and model outputs in bulk, scan for security vulnerabilities and plug into CI/CD, with evals running locally. Now part of OpenAI and still open source under MIT.

Context and harness engineering skills: context fundamentals, degradation diagnosis and compression, multi-agent architectures, memory systems, tool design, LLM-as-judge evaluation and self-improvement loops. Focused on principles and design patterns, not tied to any agent platform.

Context and harness engineering skills covering context compression and optimization, multi-agent architectures, memory systems, tool design and LLM-as-judge evaluation, activated automatically by task. Aimed at building production-grade agent systems.

ModelScope's official training framework for LLMs and multimodal models: the swift command covers pretraining, fine-tuning, RL, inference, evaluation, quantization and deployment for 600+ text and 400+ multimodal models, with Megatron parallelism and a web UI. Usually needs a GPU.

Plugin marketplace for AI research with nearly 100 skills in 23 categories, from ideation, fine-tuning and post-training to distributed training, inference serving, evaluation and paper writing, coordinated end to end by the autoresearch skill.

Nearly 100 skills spanning the AI research workflow: ideation and literature review, fine-tuning and post-training, distributed training, inference serving, evaluation, interpretability and paper writing, tied together by an autoresearch orchestration skill.

Connects to the Arize Phoenix observability platform so agents can query LLM traces and sessions, manage prompts, browse datasets and read experiment evaluation results. Maintained by Arize; needs a self-hosted or cloud Phoenix instance.

Hugging Face's official skills: manage Hub models and datasets with the hf CLI, fine-tune language and vision models, run model evaluations, and build Gradio apps deployed to Spaces. Needs an HF account; cloud training on HF Jobs requires a paid plan.

Lets agents work with the Hugging Face Hub through the hf CLI: search and download models and datasets, manage repos and Spaces, and run training and evaluation jobs on cloud GPUs, with more workflow skills installable as needed. Maintained by Hugging Face.

NVIDIA's open-source LLM vulnerability scanner that probes for prompt injection, jailbreaks, data leakage, hallucination and harmful content, supporting OpenAI, Hugging Face, Bedrock and most REST-accessible models. Scanning cloud models needs their API keys.

Open-source LLM evaluation platform: one opencompass command picks the model and datasets and runs the evaluation across hundreds of benchmarks. Works with local Hugging Face models and API models, with vLLM and LMDeploy for faster inference.

Google's official skills for building ADK agents: dev workflow, scaffolding, the ADK Python API, evaluation with LLM judges and adaptive rubrics, deployment to Agent Runtime, Cloud Run or GKE, publishing and observability. Works with agents-cli; local dev needs only an AI Studio key.

LangChain's official skill collection: framework selection, quickstarts, middleware, persistence, human-in-the-loop approval, RAG and eval design across LangChain, LangGraph and Deep Agents, in Python and TypeScript. Also installable as a Claude Code plugin. Still in early development.

The standard machine translation scorer: computes BLEU, chrF and TER with significance tests and downloads public test sets such as WMT, to compare engines or prompts. Needs Python 3.9+ and reference translations; automatic metrics are a rough screen, so final copy still needs human review.

Unbabel's machine-translation quality metric: comet-score rates translations per sentence and overall, XCOMET marks error spans and severity, and CometKiwi works without a reference. Good for spot-checking MT. Needs Python 3.8+; models download from Hugging Face, some need a license; GPU helps.

Comet's official Opik MCP: query traces and spans of LLM apps in chat, log scores and comments, and manage datasets, experiments and prompt versions. Works with Opik Cloud (API key or browser sign-in) and self-hosted open-source Opik.

Microsoft Translator's translation quality tool: GPT scores each sentence using MQM error categories or a direct score; --list_mqm_errors shows error types and severity. Needs line-aligned source and target files, Python and an OpenAI or Azure OpenAI key (paid); a translator should still review.

MLflow's official skills plugin: lets coding agents add MLflow Tracing to LLM apps, analyze failed traces, build evaluation datasets and scorers, and query token and latency metrics, closing the instrument, evaluate, iterate loop. Needs MLflow 3.8+.