{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.13.0"
    },
    "blog_metadata": {
      "topic": "How Copilot Studio’s New Evaluation Experience Could Change Agent Governance",
      "slug": "how-copilot-studio-s-new-evaluation-experience-could-change-",
      "generated_by": "LinkedIn Post Generator + Azure OpenAI",
      "generated_at": "2026-08-20T22:48:10.000Z"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# How Copilot Studio’s New Evaluation Experience Could Change Agent Governance\n",
        "\n",
        "This notebook turns the blog post into a hands-on validation workflow focused on agent governance, release gates, and regression detection. The core idea is simple: governance is not just policy documentation, it is evidence that a changed agent still meets the bar before promotion to production."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install pandas matplotlib"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "from datetime import datetime\n",
        "\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 1: Score agent test cases for groundedness and policy compliance\n",
        "\n",
        "This example validates a small set of agent test runs against a groundedness threshold and a policy compliance flag. It demonstrates the blog’s point that release decisions should be based on measurable evidence, not a green checkmark or a good demo."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Python: score a small set of agent test cases for groundedness and policy compliance\n",
        "\n",
        "test_runs = [\n",
        "    {\"id\": \"T1\", \"grounded\": 0.96, \"policy_ok\": True},\n",
        "    {\"id\": \"T2\", \"grounded\": 0.71, \"policy_ok\": True},\n",
        "    {\"id\": \"T3\", \"grounded\": 0.88, \"policy_ok\": False},\n",
        "]\n",
        "\n",
        "threshold = 0.85\n",
        "results = []\n",
        "\n",
        "for run in test_runs:\n",
        "    status = \"PASS\" if run[\"grounded\"] >= threshold and run[\"policy_ok\"] else \"FAIL\"\n",
        "    results.append({**run, \"threshold\": threshold, \"status\": status})\n",
        "    print(f'{run[\"id\"]}: grounded={run[\"grounded\"]:.2f}, policy_ok={run[\"policy_ok\"]} -> {status}')\n",
        "\n",
        "df = pd.DataFrame(results)\n",
        "display(df)\n",
        "\n",
        "summary = df[\"status\"].value_counts().rename_axis(\"status\").reset_index(name=\"count\")\n",
        "display(summary)\n",
        "\n",
        "ax = df.plot(kind=\"bar\", x=\"id\", y=\"grounded\", legend=False, title=\"Groundedness by Test Run\", color=[\"#2ca02c\" if s == \"PASS\" else \"#d62728\" for s in df[\"status\"]])\n",
        "ax.axhline(threshold, linestyle=\"--\", color=\"black\", label=f\"threshold={threshold}\")\n",
        "ax.set_ylabel(\"Groundedness\")\n",
        "ax.set_ylim(0, 1.05)\n",
        "ax.legend()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 2: Fail a release gate when evaluation metrics drop below governance thresholds\n",
        "\n",
        "The original post included a PowerShell release gate. Here, the same governance logic is implemented in Python so it can be executed directly in this notebook. The goal is to show that promotion should be blocked automatically when key thresholds are not met."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Python equivalent of the PowerShell release gate example\n",
        "\n",
        "evaluation = {\n",
        "    \"Groundedness\": 0.82,\n",
        "    \"Safety\": 0.97,\n",
        "    \"PolicyPass\": True,\n",
        "}\n",
        "\n",
        "groundedness_threshold = 0.85\n",
        "safety_threshold = 0.95\n",
        "\n",
        "release_blocked = (\n",
        "    evaluation[\"Groundedness\"] < groundedness_threshold\n",
        "    or evaluation[\"Safety\"] < safety_threshold\n",
        "    or not evaluation[\"PolicyPass\"]\n",
        ")\n",
        "\n",
        "print(\"Evaluation payload:\")\n",
        "print(json.dumps(evaluation, indent=2))\n",
        "print()\n",
        "\n",
        "if release_blocked:\n",
        "    print(\"Release blocked: evaluation gate failed.\")\n",
        "else:\n",
        "    print(\"Release approved: governance checks passed.\")\n",
        "\n",
        "reasons = []\n",
        "if evaluation[\"Groundedness\"] < groundedness_threshold:\n",
        "    reasons.append(f\"Groundedness below threshold ({evaluation['Groundedness']:.2f} < {groundedness_threshold:.2f})\")\n",
        "if evaluation[\"Safety\"] < safety_threshold:\n",
        "    reasons.append(f\"Safety below threshold ({evaluation['Safety']:.2f} < {safety_threshold:.2f})\")\n",
        "if not evaluation[\"PolicyPass\"]:\n",
        "    reasons.append(\"PolicyPass is False\")\n",
        "\n",
        "print(\"\\nGate analysis:\")\n",
        "if reasons:\n",
        "    for reason in reasons:\n",
        "        print(f\"- {reason}\")\n",
        "else:\n",
        "    print(\"- All governance thresholds satisfied\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 3: Compare two agent versions to detect regression before promotion\n",
        "\n",
        "This example checks whether a candidate version regresses materially against a baseline. That aligns with the blog’s warning that agents often fail after knowledge changes or broader rollout, not during the initial demo."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Python: compare two agent versions to detect regression before promotion\n",
        "\n",
        "baseline = {\"version\": \"v1\", \"task_success\": 0.91, \"grounded\": 0.89}\n",
        "candidate = {\"version\": \"v2\", \"task_success\": 0.93, \"grounded\": 0.81}\n",
        "\n",
        "regression = candidate[\"grounded\"] < baseline[\"grounded\"] - 0.05\n",
        "if regression:\n",
        "    print(f'Block {candidate[\"version\"]}: groundedness regression detected')\n",
        "else:\n",
        "    print(f'Promote {candidate[\"version\"]}: no material regression')\n",
        "\n",
        "comparison = pd.DataFrame([baseline, candidate]).set_index(\"version\")\n",
        "display(comparison)\n",
        "\n",
        "comparison.plot(kind=\"bar\", figsize=(8, 4), title=\"Baseline vs Candidate Agent Metrics\")\n",
        "plt.ylim(0, 1.05)\n",
        "plt.ylabel(\"Score\")\n",
        "plt.show()\n",
        "\n",
        "metric_deltas = {\n",
        "    \"task_success_delta\": candidate[\"task_success\"] - baseline[\"task_success\"],\n",
        "    \"grounded_delta\": candidate[\"grounded\"] - baseline[\"grounded\"],\n",
        "}\n",
        "print(\"Metric deltas:\")\n",
        "for k, v in metric_deltas.items():\n",
        "    print(f\"- {k}: {v:+.2f}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 4: Export a lightweight audit record for agent evaluation reviews\n",
        "\n",
        "A governed release process needs an audit trail: who evaluated the agent, when it was evaluated, what the metrics were, and whether it was approved. This example creates a lightweight JSON audit record that can be attached to a release packet."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Python equivalent of the PowerShell audit export example\n",
        "\n",
        "audit = {\n",
        "    \"AgentName\": \"HR-Policy-Agent\",\n",
        "    \"EvaluatedOn\": datetime.now().isoformat(timespec=\"seconds\"),\n",
        "    \"Groundedness\": 0.91,\n",
        "    \"Safety\": 0.98,\n",
        "    \"Approved\": True,\n",
        "}\n",
        "\n",
        "output_path = \"agent-eval-audit.json\"\n",
        "with open(output_path, \"w\", encoding=\"utf-8\") as f:\n",
        "    json.dump(audit, f, indent=2)\n",
        "\n",
        "print(f\"Saved audit trail to {output_path}\")\n",
        "print(json.dumps(audit, indent=2))\n",
        "\n",
        "with open(output_path, \"r\", encoding=\"utf-8\") as f:\n",
        "    loaded = json.load(f)\n",
        "\n",
        "print(\"\\nReloaded audit record:\")\n",
        "print(json.dumps(loaded, indent=2))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 5: Model the governance flow from test case to release decision\n",
        "\n",
        "The blog included a Mermaid diagram showing the evaluation loop. Since Mermaid is not executable Python, this cell recreates the same workflow as structured data and a simple text-based flow so you can validate the governance stages programmatically."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Python representation of the governance flow shown in the Mermaid diagram\n",
        "\n",
        "flow_steps = [\n",
        "    \"Prompt / Test Case\",\n",
        "    \"Agent in Copilot Studio\",\n",
        "    \"New Evaluation Experience\",\n",
        "    \"Metrics: groundedness, safety, task success\",\n",
        "    \"Governance Gate\",\n",
        "]\n",
        "\n",
        "pass_path = [\"Promote to production\"]\n",
        "fail_path = [\"Block release and review\", \"Refine prompts, tools, policies\", \"Agent in Copilot Studio\"]\n",
        "\n",
        "print(\"Primary flow:\")\n",
        "for i, step in enumerate(flow_steps, start=1):\n",
        "    print(f\"{i}. {step}\")\n",
        "\n",
        "print(\"\\nIf gate passes:\")\n",
        "for step in pass_path:\n",
        "    print(f\"- {step}\")\n",
        "\n",
        "print(\"\\nIf gate fails:\")\n",
        "for step in fail_path:\n",
        "    print(f\"- {step}\")\n",
        "\n",
        "nodes = pd.DataFrame(\n",
        "    {\n",
        "        \"stage\": [\n",
        "            \"Prompt / Test Case\",\n",
        "            \"Agent in Copilot Studio\",\n",
        "            \"New Evaluation Experience\",\n",
        "            \"Metrics\",\n",
        "            \"Governance Gate\",\n",
        "            \"Promote to production\",\n",
        "            \"Block release and review\",\n",
        "            \"Refine prompts, tools, policies\",\n",
        "        ],\n",
        "        \"type\": [\n",
        "            \"input\",\n",
        "            \"system\",\n",
        "            \"system\",\n",
        "            \"measurement\",\n",
        "            \"decision\",\n",
        "            \"outcome\",\n",
        "            \"outcome\",\n",
        "            \"remediation\",\n",
        "        ],\n",
        "    }\n",
        ")\n",
        "\n",
        "display(nodes)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "This notebook validated the blog’s main governance argument with executable Python examples: score test scenarios, enforce release thresholds, detect regressions, and preserve an audit trail. Together, these patterns make evaluation operational and help separate authorship, evidence review, and approval decisions.\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "- Expand the test set to include escalation behavior, unsafe responses, tool-use boundaries, and authorization-sensitive paths.\n",
        "- Add a failure taxonomy such as content gap, instruction conflict, tool failure, authorization issue, and grader disagreement.\n",
        "- Integrate these checks into CI/CD so failed evaluation evidence can actually block production promotion.\n",
        "- Define role ownership clearly across authors, evaluators, and approvers."
      ]
    }
  ]
}