{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.13.0"
    },
    "blog_metadata": {
      "topic": "Fabric Data Agents Choosing Query Language by Context: The Governance Implications",
      "slug": "fabric-data-agents-choosing-query-language-by-context-the-go",
      "generated_by": "LinkedIn Post Generator + Azure OpenAI",
      "generated_at": "2026-07-10T13:40:21.191Z"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Fabric Data Agents Choosing Query Language by Context: The Governance Implications\n",
        "\n",
        "This notebook turns the blog post into a hands-on validation workflow focused on governance for Fabric data agents. The core idea is that once an agent can choose among semantic models, KQL, OneLake, and external sources, governance must cover not just data access but also the routing logic that selects the execution path.\n",
        "\n",
        "We'll build small review artifacts, score risk by context, package audit evidence, and simulate workspace, connection, and activity posture using Python."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install pandas"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "import csv\n",
        "from dataclasses import dataclass, asdict\n",
        "from pathlib import Path\n",
        "import pandas as pd"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Governance routing model\n",
        "\n",
        "The blog argues that the governed object is no longer only the dataset or source. It is also the policy that determines which source gets used for a given class of question.\n",
        "\n",
        "Below is the routing model from the post, preserved as text for notebook reference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "mermaid_flowchart = r'''\n",
        "flowchart TD\n",
        "    A[User prompt to Fabric Data Agent] --> B{Context detected}\n",
        "    B -->|Business metrics| C[DAX via semantic model]\n",
        "    B -->|Operational telemetry| D[KQL via KQL database]\n",
        "    B -->|External enrichment| E[Connector or external source]\n",
        "    C --> F[Policy checks: model sensitivity, RLS, endorsements]\n",
        "    D --> G[Policy checks: retention, cluster access, query scope]\n",
        "    E --> H[Policy checks: approved connector, egress, secrets]\n",
        "    F --> I[Governance evidence and audit trail]\n",
        "    G --> I\n",
        "    H --> I\n",
        "'''\n",
        "\n",
        "print(mermaid_flowchart)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Review workflow sequence\n",
        "\n",
        "This sequence shows a practical review loop: enumerate dependencies, map controls by dependency type, and export evidence for audit and architecture review."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "mermaid_sequence = r'''\n",
        "sequenceDiagram\n",
        "    participant Admin\n",
        "    participant ReviewScript\n",
        "    participant Fabric\n",
        "    participant AuditStore\n",
        "    Admin->>ReviewScript: Run governance inventory\n",
        "    ReviewScript->>Fabric: Enumerate agents and dependencies\n",
        "    Fabric-->>ReviewScript: Models, KQL DBs, external sources\n",
        "    ReviewScript->>ReviewScript: Map controls by dependency type\n",
        "    ReviewScript->>AuditStore: Export evidence package\n",
        "    AuditStore-->>Admin: Review-ready CSV/JSON artifacts\n",
        "'''\n",
        "\n",
        "print(mermaid_sequence)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 1: Build a dependency inventory\n",
        "\n",
        "Start with a simple inventory of what each agent can reach. If you cannot enumerate an agent's dependencies, you do not yet understand its governance surface."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from dataclasses import dataclass, asdict\n",
        "import json\n",
        "\n",
        "@dataclass\n",
        "class Dependency:\n",
        "    agent: str\n",
        "    kind: str\n",
        "    name: str\n",
        "    workspace: str\n",
        "    control_family: str\n",
        "\n",
        "deps = [\n",
        "    Dependency(\"SalesCopilot\", \"semantic_model\", \"RevenueModel\", \"Finance\", \"RLS+Sensitivity\"),\n",
        "    Dependency(\"OpsAnalyst\", \"kql_database\", \"PlantTelemetry\", \"Operations\", \"Retention+Access\"),\n",
        "    Dependency(\"VendorRisk\", \"external_source\", \"ServiceNowAPI\", \"Risk\", \"Connector+Egress\"),\n",
        "]\n",
        "\n",
        "inventory = [asdict(d) for d in deps]\n",
        "print(json.dumps(inventory, indent=2))\n",
        "\n",
        "inventory_df = pd.DataFrame(inventory)\n",
        "inventory_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 2: Emit a governance review CSV\n",
        "\n",
        "Different dependency types imply different review checklists. This example exports a review-ready CSV that architecture and governance teams can inspect."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import csv\n",
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "\n",
        "agents = [\n",
        "    {\"agent\": \"SalesCopilot\", \"semantic_models\": [\"RevenueModel\"], \"kql_databases\": [], \"external_sources\": [\"SAP\"]},\n",
        "    {\"agent\": \"OpsAnalyst\", \"semantic_models\": [], \"kql_databases\": [\"PlantTelemetry\"], \"external_sources\": []},\n",
        "]\n",
        "\n",
        "rows = []\n",
        "for a in agents:\n",
        "    for model in a[\"semantic_models\"]:\n",
        "        rows.append([a[\"agent\"], \"semantic_model\", model, \"Validate RLS, sensitivity labels, endorsements\"])\n",
        "    for kql in a[\"kql_databases\"]:\n",
        "        rows.append([a[\"agent\"], \"kql_database\", kql, \"Validate retention, RBAC, query scope\"])\n",
        "    for src in a[\"external_sources\"]:\n",
        "        rows.append([a[\"agent\"], \"external_source\", src, \"Validate approved connector, secrets, egress\"])\n",
        "\n",
        "output_path = Path(\"fabric_agent_governance_inventory.csv\")\n",
        "with output_path.open(\"w\", newline=\"\", encoding=\"utf-8\") as f:\n",
        "    writer = csv.writer(f)\n",
        "    writer.writerow([\"agent\", \"dependency_type\", \"dependency_name\", \"review_focus\"])\n",
        "    writer.writerows(rows)\n",
        "\n",
        "print(f\"Wrote {output_path}\")\n",
        "pd.read_csv(output_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 3: Score risk by query context\n",
        "\n",
        "Not every route deserves the same review depth. This lightweight scoring model prioritizes review effort based on query language context and sensitivity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "contexts = [\n",
        "    {\"agent\": \"SalesCopilot\", \"query_language\": \"DAX\", \"asset\": \"RevenueModel\", \"sensitivity\": \"High\"},\n",
        "    {\"agent\": \"OpsAnalyst\", \"query_language\": \"KQL\", \"asset\": \"PlantTelemetry\", \"sensitivity\": \"Medium\"},\n",
        "    {\"agent\": \"VendorRisk\", \"query_language\": \"External\", \"asset\": \"ServiceNowAPI\", \"sensitivity\": \"High\"},\n",
        "]\n",
        "\n",
        "base = {\"DAX\": 2, \"KQL\": 3, \"External\": 4}\n",
        "boost = {\"Low\": 0, \"Medium\": 1, \"High\": 2}\n",
        "\n",
        "scored_rows = []\n",
        "for c in contexts:\n",
        "    score = base[c[\"query_language\"]] + boost[c[\"sensitivity\"]]\n",
        "    scored_rows.append({**c, \"risk_score\": score})\n",
        "    print(f\"{c['agent']}: {c['query_language']} on {c['asset']} => risk_score={score}\")\n",
        "\n",
        "risk_df = pd.DataFrame(scored_rows).sort_values(\"risk_score\", ascending=False)\n",
        "risk_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 4: Package audit evidence\n",
        "\n",
        "Auditability requires more than logs. You need a compact evidence package that ties control families to the agents under review."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "from pathlib import Path\n",
        "\n",
        "\n",
        "evidence = {\n",
        "    \"review_name\": \"Fabric Data Agent Query Context Review\",\n",
        "    \"controls\": {\n",
        "        \"semantic_model\": [\"RLS enforced\", \"Sensitivity labels present\", \"Certified or endorsed\"],\n",
        "        \"kql_database\": [\"RBAC validated\", \"Retention configured\", \"Cross-cluster access reviewed\"],\n",
        "        \"external_source\": [\"Approved connector\", \"Secret management\", \"Outbound egress approved\"],\n",
        "    },\n",
        "    \"agents_reviewed\": [\"SalesCopilot\", \"OpsAnalyst\", \"VendorRisk\"],\n",
        "}\n",
        "\n",
        "output_path = Path(\"fabric_agent_evidence.json\")\n",
        "with output_path.open(\"w\", encoding=\"utf-8\") as f:\n",
        "    json.dump(evidence, f, indent=2)\n",
        "\n",
        "print(f\"Wrote {output_path}\")\n",
        "print(output_path.read_text(encoding=\"utf-8\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 5: Check workspace governance posture\n",
        "\n",
        "The original post used PowerShell for workspace exports. Here we reproduce the same governance check in Python so the notebook remains executable end to end."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import pandas as pd\n",
        "from pathlib import Path\n",
        "\n",
        "workspaces = [\n",
        "    {\"Name\": \"Finance\", \"Capacity\": \"F64\", \"SensitivityLabels\": True, \"ManagedIdentity\": True},\n",
        "    {\"Name\": \"Operations\", \"Capacity\": \"F32\", \"SensitivityLabels\": True, \"ManagedIdentity\": False},\n",
        "]\n",
        "\n",
        "workspace_df = pd.DataFrame(workspaces)\n",
        "workspace_path = Path(\"workspace-governance-settings.csv\")\n",
        "workspace_df.to_csv(workspace_path, index=False)\n",
        "\n",
        "print(f\"Exported {workspace_path}\")\n",
        "workspace_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 6: Validate connection-rule posture\n",
        "\n",
        "External routing is where governance domains often shift. This example classifies connection posture so reviewers can quickly identify which external paths are ready and which need attention."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import pandas as pd\n",
        "from pathlib import Path\n",
        "\n",
        "connections = [\n",
        "    {\"Agent\": \"VendorRisk\", \"Source\": \"ServiceNowAPI\", \"ApprovedConnector\": True, \"PrivateLink\": False, \"SecretStore\": \"KeyVault\"},\n",
        "    {\"Agent\": \"SalesCopilot\", \"Source\": \"SAP\", \"ApprovedConnector\": True, \"PrivateLink\": True, \"SecretStore\": \"KeyVault\"},\n",
        "]\n",
        "\n",
        "connection_df = pd.DataFrame(connections)\n",
        "connection_df[\"Posture\"] = connection_df.apply(\n",
        "    lambda row: \"ReviewReady\" if row[\"ApprovedConnector\"] and row[\"SecretStore\"] == \"KeyVault\" else \"NeedsAttention\",\n",
        "    axis=1,\n",
        ")\n",
        "\n",
        "connection_path = Path(\"connection-rule-posture.csv\")\n",
        "connection_df.to_csv(connection_path, index=False)\n",
        "\n",
        "print(f\"Exported {connection_path}\")\n",
        "connection_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 7: Tie activity references back to governance review\n",
        "\n",
        "Operational evidence should connect back to governance review artifacts. This example exports activity references that can be linked to review packets and audit trails."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import pandas as pd\n",
        "from pathlib import Path\n",
        "\n",
        "activity = [\n",
        "    {\"Time\": \"2026-07-10T09:00:00Z\", \"Workspace\": \"Finance\", \"Operation\": \"DataAgentQuery\", \"ReferenceId\": \"evt-1001\"},\n",
        "    {\"Time\": \"2026-07-10T09:05:00Z\", \"Workspace\": \"Operations\", \"Operation\": \"ConnectionAccess\", \"ReferenceId\": \"evt-1002\"},\n",
        "]\n",
        "\n",
        "activity_df = pd.DataFrame(activity)\n",
        "activity_path = Path(\"activity-log-references.csv\")\n",
        "activity_df.to_csv(activity_path, index=False)\n",
        "\n",
        "print(f\"Exported {activity_path}\")\n",
        "activity_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Example 8: Build an admin summary across exports\n",
        "\n",
        "A compact summary helps review boards understand the current operating picture across workspaces, connections, and activity references."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import pandas as pd\n",
        "import json\n",
        "from pathlib import Path\n",
        "\n",
        "workspace_rows = pd.read_csv(Path(\"workspace-governance-settings.csv\"))\n",
        "connection_rows = pd.read_csv(Path(\"connection-rule-posture.csv\"))\n",
        "activity_rows = pd.read_csv(Path(\"activity-log-references.csv\"))\n",
        "\n",
        "summary = {\n",
        "    \"WorkspaceCount\": int(len(workspace_rows)),\n",
        "    \"ConnectionCount\": int(len(connection_rows)),\n",
        "    \"ReviewReadyConnections\": int((connection_rows[\"Posture\"] == \"ReviewReady\").sum()),\n",
        "    \"ActivityReferenceCount\": int(len(activity_rows)),\n",
        "}\n",
        "\n",
        "print(json.dumps(summary, indent=2))\n",
        "pd.DataFrame([summary])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Optional validation: Apply a routing policy hierarchy\n",
        "\n",
        "The blog recommends a clear hierarchy: semantic model first for business metrics, KQL for operational events, constrained direct access for approved file scenarios, and external retrieval as last-mile enrichment. This small simulation validates that policy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import pandas as pd\n",
        "\n",
        "questions = [\n",
        "    {\"question\": \"What was revenue yesterday?\", \"class\": \"business_metric\", \"needs_event_grain\": False, \"needs_external\": False},\n",
        "    {\"question\": \"How many machine alerts fired in the last hour?\", \"class\": \"operational_event\", \"needs_event_grain\": True, \"needs_external\": False},\n",
        "    {\"question\": \"Summarize the latest vendor incident from ServiceNow\", \"class\": \"external_enrichment\", \"needs_event_grain\": False, \"needs_external\": True},\n",
        "    {\"question\": \"List approved contract files uploaded this week\", \"class\": \"document_retrieval\", \"needs_event_grain\": False, \"needs_external\": False},\n",
        "]\n",
        "\n",
        "def choose_context(row):\n",
        "    if row[\"class\"] == \"business_metric\":\n",
        "        return \"DAX via semantic model\"\n",
        "    if row[\"class\"] == \"operational_event\" and row[\"needs_event_grain\"]:\n",
        "        return \"KQL via KQL database\"\n",
        "    if row[\"class\"] == \"document_retrieval\":\n",
        "        return \"OneLake direct access\"\n",
        "    if row[\"needs_external\"]:\n",
        "        return \"External retrieval\"\n",
        "    return \"Manual review\"\n",
        "\n",
        "routing_df = pd.DataFrame(questions)\n",
        "routing_df[\"selected_context\"] = routing_df.apply(choose_context, axis=1)\n",
        "routing_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Optional validation: Detect answer inconsistency risk\n",
        "\n",
        "A major failure mode in the post is inconsistent answers caused by different paths encoding business meaning differently. This example flags where the same business term appears across multiple contexts."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import pandas as pd\n",
        "\n",
        "semantic_catalog = pd.DataFrame([\n",
        "    {\"term\": \"sale\", \"context\": \"semantic_model\", \"definition\": \"recognized revenue transaction\"},\n",
        "    {\"term\": \"sale\", \"context\": \"kql_database\", \"definition\": \"order event emitted by checkout service\"},\n",
        "    {\"term\": \"sale\", \"context\": \"onelake_file\", \"definition\": \"daily CSV row from regional export\"},\n",
        "    {\"term\": \"incident\", \"context\": \"external_source\", \"definition\": \"ticket opened in ServiceNow\"},\n",
        "])\n",
        "\n",
        "term_counts = semantic_catalog.groupby(\"term\")[\"context\"].nunique().reset_index(name=\"distinct_contexts\")\n",
        "inconsistency_risk = term_counts[term_counts[\"distinct_contexts\"] > 1]\n",
        "\n",
        "print(\"Terms with potential semantic inconsistency across contexts:\")\n",
        "display(inconsistency_risk)\n",
        "\n",
        "semantic_catalog.merge(inconsistency_risk[[\"term\"]], on=\"term\", how=\"inner\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Optional validation: Self-assessment scorecard\n",
        "\n",
        "The post ends with a challenge: are you governing only access, or the routing decision too? Use this simple scorecard to rate current maturity from 1 to 5."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import pandas as pd\n",
        "\n",
        "scorecard = pd.DataFrame([\n",
        "    {\"dimension\": \"Access controls on sources\", \"score_1_to_5\": 4},\n",
        "    {\"dimension\": \"Documented routing hierarchy\", \"score_1_to_5\": 2},\n",
        "    {\"dimension\": \"Lineage from prompt to source\", \"score_1_to_5\": 2},\n",
        "    {\"dimension\": \"Audit evidence packaging\", \"score_1_to_5\": 3},\n",
        "    {\"dimension\": \"Semantic consistency across contexts\", \"score_1_to_5\": 2},\n",
        "])\n",
        "\n",
        "overall = round(scorecard[\"score_1_to_5\"].mean(), 2)\n",
        "print(f\"Overall query-path governance maturity: {overall}/5\")\n",
        "scorecard"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "This notebook validated the blog's main claim: governance for Fabric data agents must include the decision logic that selects DAX, KQL, OneLake, or external retrieval. We created dependency inventories, exported review artifacts, scored risk by context, packaged audit evidence, and simulated posture checks across workspaces, connections, and activity references.\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "1. Inventory every agent and every reachable dependency.\n",
        "2. Define a routing hierarchy with semantic model first for core business metrics.\n",
        "3. Add lineage expectations from prompt to selected context to returned answer.\n",
        "4. Review external retrieval as an egress policy decision, not just a connector setup task.\n",
        "5. Establish a recurring governance review that evaluates decision paths, not only prompt volume."
      ]
    }
  ]
}