GitHub Copilot Evals: Power-User Guide to /eval, Vally and Run Costs

Last updated: November 17, 2026

GitHub Copilot Evals is a public preview tool, released November 17, 2026, that turns a task into a repeatable test for your Copilot skills, custom agents and MCP tools. You create an eval, approve each run, see which checks failed and compare runs after a change. It works in VS Code, Copilot CLI and the GitHub Copilot app.

Contents

What GitHub released

GitHub announced in its changelog on November 17, 2026 that GitHub Copilot Evals is now in public preview for Copilot Individual, Copilot Business and Copilot Enterprise plans. GitHub defines an eval as a repeatable test: a realistic task, a description of a good result, and checks on both the result and how the agent got there.

The point is regression testing. AI output changes between runs, so one good answer does not prove a customization works. After editing a skill, agent or MCP tool, rerun the same eval and compare.

If you already build custom workflows, this pairs naturally with Copilot Automations and the plugin system covered in our Copilot CLI 1.0.111 guide.

What is in the plugin

The github/evals repository README lists these components:

Component What it does
Six skills create, review, improve, run, compare and setup, callable in plain language
/eval commands /eval setup, /eval create, /eval review, /eval improve, /eval run, /eval compare in Copilot CLI and the Copilot app
Evals Canvas Side panel to browse project assets, draft evals, inspect YAML and read saved run reports
MCP server A 15-tool copilot-evals server bundled with the plugin
Automation blueprints Three .automation.md definitions: eval-health-review, eval-smoke-run and eval-skill-compare

What you can test, according to the extension documentation: skills defined in SKILL.md, custom agents defined in *.agent.md files, and MCP tools.

How to install it

The Copilot Evals getting-started guide recommends Node.js 24 or later, Vally 0.19.0 or later, Copilot CLI 1.0.85 or later and Git 2.28.0 or later. The README notes the VS Code extension needs only Node.js 22.14.0. Install steps differ by surface:

Where you use Copilot How to install Notes
VS Code Install GitHub.copilot-evals (publisher GitHub) from the Visual Studio Marketplace VS Code 1.138.0 or later, local trusted project; run “GitHub Copilot Evals: Open Evals Canvas”
VS Code Agents window Install the extension in the default profile, enable chat.canvases.enabled Do not also enable the standalone plugin in the same session
Copilot CLI copilot plugin marketplace add github/evals, then copilot plugin install eval@github-copilot-evals /eval commands and Canvas need experimental mode
GitHub Copilot app Customize > Browse > Add > Manage marketplaces, enter github/evals, install eval Start a fresh session, then /eval help

In the CLI, the six skills work without experimental mode. To get the native /eval commands and the Canvas, the guide says to turn on CLI-wide experimental mode with /experimental and start a fresh conversation. For the app itself, see our GitHub Copilot app guide.

How to create and run your first eval

The README walks through this sequence:

  1. Open a local project in Copilot CLI or VS Code, trust the folder and sign in to GitHub Copilot.
  2. Enter /eval create, or open the Evals Canvas, and pick a skill, custom agent or MCP tool to test. You can also just ask, “Help me create evals for my skills.”
  3. Describe what a passing result looks like. For a release-notes skill, GitHub’s example checks that only completed work appears and input files stay unchanged.
  4. Review the proposed test cases, fixtures and checks, then save the eval.
  5. Run /eval run or choose Review & run in the Canvas. Review the execution effects and token or credit costs, then approve the run.
  6. Inspect the report with /eval review or the Results tab to see passing checks, outputs and the conversation trace.
  7. After editing your skill or agent, run the eval again and use /eval compare against the previous saved result.

Running needs the local eval runner. In the Canvas, go to Settings > Advanced > Install eval runner, or run “GitHub Copilot Evals: Set up test runner” in VS Code. Use /eval setup to configure eval folders and test suites. The compare skill can also run one task with and without a skill.

Runs, costs and saved results

  • Saving and running are separate. File validation and reading or comparing saved results make no model calls.
  • Every run needs approval. Each eval run or grader test shows its execution effects and token or credit costs first. Opening or installing Evals never approves a run.
  • No spending cap. The getting-started guide says Evals does not enforce a total token or credit cap, so watch large suites and repeated trials.
  • Local execution. Setup scripts, tools, graders and AI judges run under your normal local permissions.
  • Results stay in your project. Reports are written to .copilot\evals\results. Review them before sharing.

For how Copilot picks and bills models, see our Copilot Fusion guide.

Copilot Evals compared with running Vally directly

GitHub says Copilot Evals uses Vally, Microsoft’s extensible local eval tool, as its engine. Its pipeline is stimulus, executor, trajectory (every turn, tool call and token), graders and score.

Question GitHub Copilot Evals Vally on its own
Main user Copilot users testing their skills, agents and MCP tools Teams evaluating any AI project
How you author evals Ask Copilot or use /eval create and the Canvas Configure executors and graders yourself
What it can run Copilot skills, custom agents, local Node.js stdio MCP servers Any MCP server, skillset or CLI agent with an executor, or your own
Where it runs VS Code, Copilot CLI, Copilot app Your dev workflow, CI or an isolated environment

Copilot Evals is the guided front end; Vally is the general engine. Teams testing agents through an SDK, as in our Cursor Agent SDK guide, may prefer Vally directly for CI.

Limitations and privacy

  • MCP scope. One local Node.js stdio MCP server per run. Python and remote MCP servers are not currently supported.
  • VS Code scope. Bundled skills are unavailable in remote, web and untrusted workspaces.
  • A pass is not a guarantee. GitHub says a passing check does not mean the AI will get it right every time.
  • Telemetry. Plugin telemetry is on by default but excludes code, prompts and outputs. Set COPILOT_EVALS_TELEMETRY=off to disable it.
  • License. The code is source available under the GitHub Limited Use Source License, not open source.

Frequently asked questions

What is GitHub Copilot Evals?

GitHub Copilot Evals is a preview plugin and VS Code extension that lets you create, run and compare repeatable tests for your Copilot skills, custom agents and MCP tools. GitHub released it in public preview on November 17, 2026, and it uses Microsoft’s Vally eval engine.

Which Copilot plans can use Copilot Evals?

GitHub says Copilot Evals is in public preview for Copilot Individual, Copilot Business and Copilot Enterprise plans. Features in preview can change before general availability.

Does running an eval cost Copilot credits?

It can. The getting-started guide says chat and eval runs may use credits, and each run asks you to approve its costs, tools and execution effects. Evals does not enforce a total token or credit cap. Saving, validating and comparing saved results make no model calls.

How do I install Copilot Evals in Copilot CLI?

Run copilot plugin marketplace add github/evals, then copilot plugin install eval@github-copilot-evals, and start a fresh conversation in your project. The skills work right away; the /eval commands and Evals Canvas need CLI experimental mode.

Can Copilot Evals test any MCP server?

No. The extension page says it can evaluate one local Node.js stdio MCP server per run, with one task per fresh server start. Python MCP servers and remote MCP servers are not currently supported.

Sources

Image: GitHub Copilot Evals illustration from the GitHub Changelog, used for editorial purposes.

Related guide: MAI Code 2 guide to Flash, Copilot pricing and access.

Related answers

Leave a Comment