{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.13.0" }, "blog_metadata": { "topic": "From Pilot to Production: A Governance Checklist for Microsoft Foundry Agents", "slug": "from-pilot-to-production-a-governance-checklist-for-microsof", "generated_by": "LinkedIn Post Generator + Azure OpenAI", "generated_at": "2026-07-23T23:11:10.455Z" } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# From Pilot to Production: A Governance Checklist for Microsoft Foundry Agents\n", "\n", "A successful Foundry agent pilot proves usefulness, but production readiness requires enforceable governance. This notebook turns the blog post into hands-on validation steps you can run locally to test identity, tool permissions, escalation evidence, observability artifacts, and phased rollout controls.\n", "\n", "The focus is practical: convert governance expectations into simple Python checks, review artifacts, and release gates that can be automated." ] }, { "cell_type": "code", "metadata": {}, "source": [ "%pip install -q pandas pydantic" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import json\n", "from dataclasses import dataclass, asdict\n", "from datetime import datetime, timezone\n", "from typing import List, Dict, Any\n", "\n", "import pandas as pd" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Promotion flow and governance gate shape\n", "\n", "The blog emphasizes that production should be treated as a gate, not a passive checklist. This cell captures the pilot-to-production flow in a structured form so teams can validate that failed reviews loop back into remediation before rollout." ] }, { "cell_type": "code", "metadata": {}, "source": [ "promotion_flow = {\n", " \"nodes\": [\n", " \"Pilot Agent\",\n", " \"Governance Intake\",\n", " \"Policy-as-Code Gate\",\n", " \"Phased Rollout\",\n", " \"Remediate Gaps\",\n", " \"Production Approval\",\n", " \"Continuous Monitoring\",\n", " \"Incident Review + Re-evaluation\",\n", " ],\n", " \"edges\": [\n", " (\"Pilot Agent\", \"Governance Intake\"),\n", " (\"Governance Intake\", \"Policy-as-Code Gate\"),\n", " (\"Policy-as-Code Gate\", \"Phased Rollout\", \"Pass\"),\n", " (\"Policy-as-Code Gate\", \"Remediate Gaps\", \"Fail\"),\n", " (\"Remediate Gaps\", \"Governance Intake\"),\n", " (\"Phased Rollout\", \"Production Approval\"),\n", " (\"Production Approval\", \"Continuous Monitoring\"),\n", " (\"Continuous Monitoring\", \"Incident Review + Re-evaluation\"),\n", " ],\n", "}\n", "\n", "print(json.dumps(promotion_flow, indent=2))\n", "\n", "required_loop = (\"Remediate Gaps\", \"Governance Intake\") in [e[:2] for e in promotion_flow[\"edges\"]]\n", "print({\"has_remediation_loop\": required_loop, \"gate_present\": \"Policy-as-Code Gate\" in promotion_flow[\"nodes\"]})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Policy-as-code gate for production promotion\n", "\n", "This example implements the blog's core idea: promotion should fail automatically when ownership, approved tools, evaluation evidence, runbook coverage, or rollout constraints are missing. The logic is intentionally simple so it can be embedded into CI/CD or release workflows." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Policy-as-code gate for promoting a Foundry agent to production\n", "nagent = {\n", " \"name\": \"claims-triage-agent\",\n", " \"owner\": \"platform-ops@contoso.com\",\n", " \"approved_tools\": [\"search\", \"ticketing\"],\n", " \"requested_tools\": [\"search\"],\n", " \"evaluation\": {\"groundedness\": 0.93, \"safety\": 0.98, \"evidence_uri\": \"https://review/evals/42\"},\n", " \"incident_runbook\": \"https://wiki/runbooks/claims-triage\",\n", " \"rollout\": {\"strategy\": \"phased\", \"pilot_users\": 25, \"max_blast_radius_pct\": 10},\n", "}\n", "\n", "errors = []\n", "if not agent.get(\"owner\"):\n", " errors.append(\"Missing accountable owner\")\n", "if not set(agent[\"requested_tools\"]).issubset(set(agent[\"approved_tools\"])):\n", " errors.append(\"Requested tools are not approved\")\n", "if agent[\"evaluation\"][\"groundedness\"] < 0.90 or agent[\"evaluation\"][\"safety\"] < 0.95:\n", " errors.append(\"Evaluation thresholds not met\")\n", "if not agent[\"evaluation\"].get(\"evidence_uri\"):\n", " errors.append(\"Missing evaluation evidence\")\n", "if not agent.get(\"incident_runbook\"):\n", " errors.append(\"Missing incident runbook\")\n", "if agent[\"rollout\"][\"strategy\"] != \"phased\" or agent[\"rollout\"][\"max_blast_radius_pct\"] > 10:\n", " errors.append(\"Rollout criteria not compliant\")\n", "\n", "print({\"agent\": agent[\"name\"], \"decision\": \"PROMOTE\" if not errors else \"BLOCK\", \"errors\": errors})\n", "\n", "# Negative test case\n", "candidate = {\n", " **agent,\n", " \"requested_tools\": [\"search\", \"email\"],\n", " \"incident_runbook\": \"\",\n", "}\n", "\n", "candidate_errors = []\n", "if not candidate.get(\"owner\"):\n", " candidate_errors.append(\"Missing accountable owner\")\n", "if not set(candidate[\"requested_tools\"]).issubset(set(candidate[\"approved_tools\"])):\n", " candidate_errors.append(\"Requested tools are not approved\")\n", "if candidate[\"evaluation\"][\"groundedness\"] < 0.90 or candidate[\"evaluation\"][\"safety\"] < 0.95:\n", " candidate_errors.append(\"Evaluation thresholds not met\")\n", "if not candidate[\"evaluation\"].get(\"evidence_uri\"):\n", " candidate_errors.append(\"Missing evaluation evidence\")\n", "if not candidate.get(\"incident_runbook\"):\n", " candidate_errors.append(\"Missing incident runbook\")\n", "if candidate[\"rollout\"][\"strategy\"] != \"phased\" or candidate[\"rollout\"][\"max_blast_radius_pct\"] > 10:\n", " candidate_errors.append(\"Rollout criteria not compliant\")\n", "\n", "print({\"agent\": candidate[\"name\"], \"decision\": \"PROMOTE\" if not candidate_errors else \"BLOCK\", \"errors\": candidate_errors})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Minimal governance checklist model\n", "\n", "The blog argues that every production agent needs explicit evidence and named ownership. This dataclass provides a compact review packet model you can extend for internal governance workflows." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Minimal governance checklist model for a Foundry agent review packet\n", "from dataclasses import dataclass\n", "\n", "@dataclass\n", "class GovernanceChecklist:\n", " owner: str\n", " data_classification: str\n", " approved_tools: list[str]\n", " eval_evidence_uri: str\n", " incident_runbook_uri: str\n", " rollout_strategy: str\n", "\n", "checklist = GovernanceChecklist(\n", " owner=\"platform-ops@contoso.com\",\n", " data_classification=\"Confidential\",\n", " approved_tools=[\"search\", \"ticketing\"],\n", " eval_evidence_uri=\"https://review/evals/42\",\n", " incident_runbook_uri=\"https://wiki/runbooks/claims-triage\",\n", " rollout_strategy=\"phased\",\n", ")\n", "\n", "print(checklist)\n", "print(asdict(checklist))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Endpoint and tool inventory artifact\n", "\n", "Tool sprawl is one of the main production risks described in the post. This Python version of the operational evidence script inventories endpoints, access methods, approved tools, and sign-off targets so reviewers can validate what the agent can actually reach." ] }, { "cell_type": "code", "metadata": {}, "source": [ "agent_inventory = {\n", " \"Name\": \"claims-triage-agent\",\n", " \"Environment\": \"prod\",\n", " \"Endpoints\": [\n", " {\"Name\": \"Inference\", \"Url\": \"https://foundry.contoso.com/agents/claims/invoke\", \"Access\": \"ManagedIdentity\"},\n", " {\"Name\": \"Search\", \"Url\": \"https://search.contoso.com/indexes/claims\", \"Access\": \"RBAC\"},\n", " {\"Name\": \"Ticketing\", \"Url\": \"https://api.contoso.com/tickets\", \"Access\": \"ServicePrincipal\"},\n", " ],\n", " \"ApprovedTools\": [\"search\", \"ticketing\"],\n", " \"Reviewers\": [\"security@contoso.com\", \"ops@contoso.com\"],\n", "}\n", "\n", "artifact = {\n", " \"AgentName\": agent_inventory[\"Name\"],\n", " \"Environment\": agent_inventory[\"Environment\"],\n", " \"EndpointCount\": len(agent_inventory[\"Endpoints\"]),\n", " \"Endpoints\": agent_inventory[\"Endpoints\"],\n", " \"ApprovedTools\": agent_inventory[\"ApprovedTools\"],\n", " \"SignOffTargets\": agent_inventory[\"Reviewers\"],\n", " \"GeneratedAtUtc\": datetime.now(timezone.utc).isoformat(),\n", "}\n", "\n", "print(json.dumps(artifact, indent=2))\n", "\n", "inventory_df = pd.DataFrame(agent_inventory[\"Endpoints\"])\n", "print(inventory_df)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Review workflow sequence as executable state transitions\n", "\n", "The original post includes a sequence diagram showing how an agent owner, policy gate, security, operations, and production approval interact. This cell converts that sequence into a simple ordered event log that can be validated programmatically." ] }, { "cell_type": "code", "metadata": {}, "source": [ "sequence_events = [\n", " {\"from\": \"Agent Owner\", \"to\": \"Policy Gate\", \"action\": \"Submit agent metadata + eval evidence\"},\n", " {\"from\": \"Policy Gate\", \"to\": \"Policy Gate\", \"action\": \"Validate owner, tools, runbook, rollout\"},\n", " {\"from\": \"Policy Gate\", \"to\": \"Agent Owner\", \"action\": \"Fail with remediation items\"},\n", " {\"from\": \"Policy Gate\", \"to\": \"Security Review\", \"action\": \"Request access/tool sign-off\"},\n", " {\"from\": \"Security Review\", \"to\": \"Operations\", \"action\": \"Review endpoints and environment\"},\n", " {\"from\": \"Operations\", \"to\": \"Policy Gate\", \"action\": \"Operational evidence approved\"},\n", " {\"from\": \"Policy Gate\", \"to\": \"Production\", \"action\": \"Approve phased promotion\"},\n", " {\"from\": \"Production\", \"to\": \"Agent Owner\", \"action\": \"Production enabled with monitoring\"},\n", "]\n", "\n", "sequence_df = pd.DataFrame(sequence_events)\n", "print(sequence_df)\n", "\n", "required_actions = {\n", " \"Submit agent metadata + eval evidence\",\n", " \"Validate owner, tools, runbook, rollout\",\n", " \"Approve phased promotion\",\n", "}\n", "print({\"required_actions_present\": required_actions.issubset(set(sequence_df[\"action\"]))})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Compact review artifact for audit and sign-off\n", "\n", "The blog recommends generating a compact artifact that proves owner, environment, sign-offs, and evidence locations. This is useful for release records, audit trails, and incident reconstruction." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Create a compact review artifact for audit and sign-off workflows\n", "import json\n", "from datetime import datetime, timezone\n", "\n", "artifact = {\n", " \"agent\": \"claims-triage-agent\",\n", " \"environment\": \"prod\",\n", " \"owner\": \"platform-ops@contoso.com\",\n", " \"signoffs\": [\"security\", \"operations\", \"product\"],\n", " \"evidence\": {\n", " \"eval_report\": \"https://review/evals/42\",\n", " \"runbook\": \"https://wiki/runbooks/claims-triage\",\n", " \"endpoint_inventory\": \"artifact://endpoint-inventory.json\",\n", " },\n", " \"generated_at_utc\": datetime.now(timezone.utc).isoformat(),\n", "}\n", "\n", "print(json.dumps(artifact, indent=2))\n", "\n", "required_evidence = {\"eval_report\", \"runbook\", \"endpoint_inventory\"}\n", "print({\"artifact_complete\": required_evidence.issubset(set(artifact[\"evidence\"].keys()))})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Phased rollout validation\n", "\n", "The post stresses that rollout should begin with a tightly bounded cohort and explicit rollback readiness. This cell validates canary-style rollout settings before production traffic is enabled." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Validate phased rollout settings before enabling production traffic\n", "rollout = {\n", " \"stage\": \"canary\",\n", " \"traffic_percent\": 5,\n", " \"success_rate\": 0.992,\n", " \"rollback_ready\": True,\n", " \"approver\": \"release-manager@contoso.com\",\n", "}\n", "\n", "allowed = (\n", " rollout[\"stage\"] in {\"canary\", \"ring1\", \"ring2\"} and\n", " rollout[\"traffic_percent\"] <= 10 and\n", " rollout[\"success_rate\"] >= 0.99 and\n", " rollout[\"rollback_ready\"] and\n", " bool(rollout[\"approver\"])\n", ")\n", "\n", "print({\"rollout_ok\": allowed, \"rollout\": rollout})\n", "\n", "bad_rollout = {\n", " \"stage\": \"full\",\n", " \"traffic_percent\": 100,\n", " \"success_rate\": 0.97,\n", " \"rollback_ready\": False,\n", " \"approver\": \"\",\n", "}\n", "\n", "bad_allowed = (\n", " bad_rollout[\"stage\"] in {\"canary\", \"ring1\", \"ring2\"} and\n", " bad_rollout[\"traffic_percent\"] <= 10 and\n", " bad_rollout[\"success_rate\"] >= 0.99 and\n", " bad_rollout[\"rollback_ready\"] and\n", " bool(bad_rollout[\"approver\"])\n", ")\n", "\n", "print({\"rollout_ok\": bad_allowed, \"rollout\": bad_rollout})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Release failure on missing governance evidence\n", "\n", "The blog's final automation example fails a release when required evidence is incomplete. This Python equivalent removes politics from the decision by enforcing owner, tool approval, evaluation evidence, runbook presence, and phased rollout." ] }, { "cell_type": "code", "metadata": {}, "source": [ "release = {\n", " \"Owner\": \"platform-ops@contoso.com\",\n", " \"ApprovedTools\": [\"search\", \"ticketing\"],\n", " \"RequestedTools\": [\"search\"],\n", " \"EvaluationEvidence\": \"https://review/evals/42\",\n", " \"IncidentRunbook\": \"https://wiki/runbooks/claims-triage\",\n", " \"RolloutStrategy\": \"phased\",\n", "}\n", "\n", "def validate_release(release: dict) -> list[str]:\n", " errors = []\n", " if not release.get(\"Owner\"):\n", " errors.append(\"Missing owner\")\n", " if any(tool not in release.get(\"ApprovedTools\", []) for tool in release.get(\"RequestedTools\", [])):\n", " errors.append(\"Unapproved tools requested\")\n", " if not release.get(\"EvaluationEvidence\"):\n", " errors.append(\"Missing evaluation evidence\")\n", " if not release.get(\"IncidentRunbook\"):\n", " errors.append(\"Missing incident runbook\")\n", " if release.get(\"RolloutStrategy\") != \"phased\":\n", " errors.append(\"Rollout must be phased\")\n", " return errors\n", "\n", "errors = validate_release(release)\n", "if errors:\n", " raise RuntimeError(\"Governance gate failed: \" + \"; \".join(errors))\n", "else:\n", " print(\"Governance gate passed\")\n", "\n", "failing_release = {\n", " \"Owner\": \"\",\n", " \"ApprovedTools\": [\"search\"],\n", " \"RequestedTools\": [\"search\", \"ticketing\"],\n", " \"EvaluationEvidence\": \"\",\n", " \"IncidentRunbook\": \"\",\n", " \"RolloutStrategy\": \"full\",\n", "}\n", "\n", "print({\"failing_release_errors\": validate_release(failing_release)})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Team self-assessment scorecard\n", "\n", "To close the loop, this notebook includes a simple scorecard based on the blog's 1-to-5 maturity framing. Rate each control area and compute an overall production-gate score." ] }, { "cell_type": "code", "metadata": {}, "source": [ "scorecard = {\n", " \"identity_and_access\": 4,\n", " \"tool_permissions\": 3,\n", " \"human_escalation\": 4,\n", " \"observability_and_audit\": 3,\n", " \"incident_response_and_rollback\": 5,\n", "}\n", "\n", "avg_score = sum(scorecard.values()) / len(scorecard)\n", "\n", "if avg_score < 2:\n", " maturity = \"demo-driven chaos\"\n", "elif avg_score < 3:\n", " maturity = \"pilot-heavy, weak production controls\"\n", "elif avg_score < 4:\n", " maturity = \"partially governed\"\n", "elif avg_score < 4.75:\n", " maturity = \"production-capable with gaps\"\n", "else:\n", " maturity = \"strong enforceable production gate\"\n", "\n", "print({\"scorecard\": scorecard, \"average_score\": round(avg_score, 2), \"maturity\": maturity})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Next Steps\n", "\n", "Use these cells as a starter governance harness for Microsoft Foundry agents and adjacent agent platforms. Extend the checks with your real identity model, endpoint inventory, evaluation datasets, approval workflow, and CI/CD release gates.\n", "\n", "Before promoting any agent, confirm you can explain its identity path, approved tools, escalation route, telemetry evidence, rollback plan, and named incident ownership. If you cannot produce that packet on demand, the pilot should not pass the production gate." ] } ] }