{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.13.0"
    },
    "blog_metadata": {
      "topic": "How Azure Monitor in Fabric can become your AI-era observability layer",
      "slug": "how-azure-monitor-in-fabric-can-become-your-ai-era-observabi",
      "generated_by": "LinkedIn Post Generator + Azure OpenAI",
      "generated_at": "2026-07-28T03:21:04.277Z"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# How Azure Monitor in Fabric can become your AI-era observability layer\n",
        "\n",
        "This notebook turns the blog post into a hands-on validation flow. It keeps Azure Monitor and Application Insights in the operational loop while showing how selected telemetry can become governed analytical evidence for incident forensics, AI workload analysis, and cross-team collaboration.\n",
        "\n",
        "The examples focus on narrow scope, explicit time windows, and contextual joins so you can validate the pattern without copying every signal into a second observability stack."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install -q azure-identity azure-monitor-query pandas python-dotenv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from datetime import timedelta\n",
        "import os\n",
        "import csv\n",
        "import json\n",
        "\n",
        "import pandas as pd\n",
        "from azure.identity import DefaultAzureCredential\n",
        "from azure.monitor.query import LogsQueryClient, LogsQueryStatus"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Architecture at a glance\n",
        "\n",
        "The core pattern is simple: telemetry is collected and alerted on in Azure Monitor and Log Analytics, while governed queries and contextual joins extend that telemetry into a shared analytical evidence layer.\n",
        "\n",
        "This cell renders the architecture as structured data so you can inspect or reuse it programmatically."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "architecture = {\n",
        "    \"nodes\": [\n",
        "        \"Fabric workloads and notebooks\",\n",
        "        \"Azure Monitor ingestion\",\n",
        "        \"Log Analytics workspace\",\n",
        "        \"Governed KQL queries\",\n",
        "        \"Incident forensics notebook\",\n",
        "        \"Alerts and workbooks\",\n",
        "        \"Ops and platform owners\",\n",
        "        \"Deployment metadata join\"\n",
        "    ],\n",
        "    \"edges\": [\n",
        "        [\"Fabric workloads and notebooks\", \"Azure Monitor ingestion\"],\n",
        "        [\"Azure Monitor ingestion\", \"Log Analytics workspace\"],\n",
        "        [\"Log Analytics workspace\", \"Governed KQL queries\"],\n",
        "        [\"Governed KQL queries\", \"Incident forensics notebook\"],\n",
        "        [\"Log Analytics workspace\", \"Alerts and workbooks\"],\n",
        "        [\"Alerts and workbooks\", \"Ops and platform owners\"],\n",
        "        [\"Governed KQL queries\", \"Deployment metadata join\"],\n",
        "        [\"Deployment metadata join\", \"Incident forensics notebook\"]\n",
        "    ]\n",
        "}\n",
        "\n",
        "print(json.dumps(architecture, indent=2))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Required environment variables\n",
        "\n",
        "Set these before running the live Azure Monitor query examples:\n",
        "\n",
        "- `AZURE_TENANT_ID`\n",
        "- `AZURE_CLIENT_ID`\n",
        "- `AZURE_CLIENT_SECRET`\n",
        "- `LOG_ANALYTICS_WORKSPACE_ID`\n",
        "\n",
        "If you are using managed identity or Azure CLI authentication in your environment, `DefaultAzureCredential` may work without all of these variables, but `LOG_ANALYTICS_WORKSPACE_ID` is still required."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Governed Azure Monitor query with explicit scope and time window\n",
        "\n",
        "This example validates the blog's recommendation to keep queries narrow and governed. It uses a fixed workspace, a six-hour time window, and a small projected shape so the result is useful for investigation instead of becoming a fishing expedition."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "workspace_id = os.getenv(\"LOG_ANALYTICS_WORKSPACE_ID\", \"00000000-0000-0000-0000-000000000000\")\n",
        "credential = DefaultAzureCredential()\n",
        "client = LogsQueryClient(credential)\n",
        "\n",
        "timespan = timedelta(hours=6)\n",
        "kql = \"\"\"\n",
        "AppTraces\n",
        "| where TimeGenerated > ago(6h)\n",
        "| where SeverityLevel >= 2\n",
        "| project TimeGenerated, OperationId, Message, AppRoleName\n",
        "| take 20\n",
        "\"\"\"\n",
        "\n",
        "try:\n",
        "    result = client.query_workspace(workspace_id, kql, timespan=timespan)\n",
        "    if result.status == LogsQueryStatus.SUCCESS and result.tables:\n",
        "        columns = [c.name for c in result.tables[0].columns]\n",
        "        rows = result.tables[0].rows\n",
        "        df = pd.DataFrame(rows, columns=columns)\n",
        "        display(df)\n",
        "    else:\n",
        "        print(\"Query did not return a successful table result.\")\n",
        "except Exception as e:\n",
        "    print(f\"Live query could not be executed: {e}\")\n",
        "    print(\"Tip: set LOG_ANALYTICS_WORKSPACE_ID and authenticate with Azure before rerunning.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Incident forensics by joining failed requests with deployment metadata\n",
        "\n",
        "The operational value comes from the join, not the raw request stream. This example correlates failed requests with deployment ring, build version, and ownership metadata so you can test whether a release or ring change aligns with the incident."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "workspace_id = os.getenv(\"LOG_ANALYTICS_WORKSPACE_ID\", \"00000000-0000-0000-0000-000000000000\")\n",
        "client = LogsQueryClient(DefaultAzureCredential())\n",
        "\n",
        "kql = \"\"\"\n",
        "let deploymentMeta = datatable(OperationId:string, DeploymentRing:string, BuildVersion:string, Owner:string)\n",
        "[\n",
        "  \"op-1001\", \"prod\", \"2026.07.15.1\", \"fabric-ops\",\n",
        "  \"op-1002\", \"canary\", \"2026.07.16.3\", \"ml-platform\"\n",
        "];\n",
        "AppRequests\n",
        "| where TimeGenerated > ago(2h)\n",
        "| where Success == false\n",
        "| project TimeGenerated, OperationId, Name, DurationMs=DurationMs, ResultCode\n",
        "| join kind=leftouter deploymentMeta on OperationId\n",
        "| order by TimeGenerated desc\n",
        "\"\"\"\n",
        "\n",
        "try:\n",
        "    response = client.query_workspace(workspace_id, kql, timespan=timedelta(hours=2))\n",
        "    if response.tables:\n",
        "        columns = [c.name for c in response.tables[0].columns]\n",
        "        rows = response.tables[0].rows\n",
        "        df = pd.DataFrame(rows, columns=columns)\n",
        "        display(df)\n",
        "    else:\n",
        "        print(\"No rows returned.\")\n",
        "except Exception as e:\n",
        "    print(f\"Live query could not be executed: {e}\")\n",
        "    print(\"You can still inspect the KQL and adapt it to your workspace schema.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create a Log Analytics workspace for a narrow proof of concept\n",
        "\n",
        "The original post used PowerShell. This notebook converts that setup into Python using Azure CLI commands so you can execute it from a Python-first workflow.\n",
        "\n",
        "Review the values carefully before running. The command cell prints the Azure CLI commands by default; uncomment execution lines only when you are ready."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import subprocess\n",
        "\n",
        "subscription_id = os.getenv(\"AZURE_SUBSCRIPTION_ID\", \"00000000-0000-0000-0000-000000000000\")\n",
        "resource_group = \"rg-fabric-observability-poc\"\n",
        "location = \"eastus\"\n",
        "workspace_name = \"law-fabric-observability-poc\"\n",
        "\n",
        "commands = [\n",
        "    [\"az\", \"account\", \"set\", \"--subscription\", subscription_id],\n",
        "    [\"az\", \"group\", \"create\", \"--name\", resource_group, \"--location\", location],\n",
        "    [\n",
        "        \"az\", \"monitor\", \"log-analytics\", \"workspace\", \"create\",\n",
        "        \"--resource-group\", resource_group,\n",
        "        \"--workspace-name\", workspace_name,\n",
        "        \"--location\", location\n",
        "    ]\n",
        "]\n",
        "\n",
        "for cmd in commands:\n",
        "    print(\" \".join(cmd))\n",
        "\n",
        "# To execute, uncomment below:\n",
        "# for cmd in commands:\n",
        "#     subprocess.run(cmd, check=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Tag ownership and scope explicitly\n",
        "\n",
        "Tags are part of the operating model, not decoration. This example applies owner, scope, data classification, and cost center tags so telemetry routing has clear accountability from the start."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "subscription_id = os.getenv(\"AZURE_SUBSCRIPTION_ID\", \"00000000-0000-0000-0000-000000000000\")\n",
        "resource_group = \"rg-fabric-observability-poc\"\n",
        "workspace_name = \"law-fabric-observability-poc\"\n",
        "workspace_resource_id = f\"/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.OperationalInsights/workspaces/{workspace_name}\"\n",
        "\n",
        "tags = {\n",
        "    \"Owner\": \"platform-observability\",\n",
        "    \"Scope\": \"fabric-ai-poc\",\n",
        "    \"DataClassification\": \"OperationalTelemetry\",\n",
        "    \"CostCenter\": \"ENG-OBS\"\n",
        "}\n",
        "\n",
        "cmd = [\n",
        "    \"az\", \"tag\", \"update\",\n",
        "    \"--resource-id\", workspace_resource_id,\n",
        "    \"--operation\", \"Merge\",\n",
        "    \"--tags\"\n",
        "] + [f\"{k}={v}\" for k, v in tags.items()]\n",
        "\n",
        "print(\" \".join(cmd))\n",
        "\n",
        "# To execute, uncomment below:\n",
        "# subprocess.run(cmd, check=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Route a deliberately narrow slice of diagnostics\n",
        "\n",
        "This example keeps the proof of concept intentionally small by routing only selected storage diagnostic categories and a transaction metric into Log Analytics. Validate categories, volume, and query usefulness before adding more sources."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "subscription_id = os.getenv(\"AZURE_SUBSCRIPTION_ID\", \"00000000-0000-0000-0000-000000000000\")\n",
        "resource_group = \"rg-fabric-observability-poc\"\n",
        "workspace_name = \"law-fabric-observability-poc\"\n",
        "resource_id = f\"/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.Storage/storageAccounts/fabricdiagpoc\"\n",
        "workspace_resource_id = f\"/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.OperationalInsights/workspaces/{workspace_name}\"\n",
        "setting_name = \"route-selected-telemetry\"\n",
        "\n",
        "logs = '[{\"category\":\"StorageRead\",\"enabled\":true},{\"category\":\"StorageWrite\",\"enabled\":true}]'\n",
        "metrics = '[{\"category\":\"Transaction\",\"enabled\":true}]'\n",
        "\n",
        "cmd = [\n",
        "    \"az\", \"monitor\", \"diagnostic-settings\", \"create\",\n",
        "    \"--name\", setting_name,\n",
        "    \"--resource\", resource_id,\n",
        "    \"--workspace\", workspace_resource_id,\n",
        "    \"--logs\", logs,\n",
        "    \"--metrics\", metrics\n",
        "]\n",
        "\n",
        "print(\" \".join(cmd))\n",
        "\n",
        "# To execute, uncomment below:\n",
        "# subprocess.run(cmd, check=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Sequence of evidence flow\n",
        "\n",
        "This cell captures the operational sequence described in the post: workloads emit telemetry, Azure Monitor stores it in Log Analytics, a notebook runs governed KQL, and the resulting evidence is shared with platform owners."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "sequence = [\n",
        "    {\"from\": \"Fabric Workload\", \"to\": \"Azure Monitor\", \"action\": \"Emit diagnostics, traces, metrics\"},\n",
        "    {\"from\": \"Azure Monitor\", \"to\": \"Log Analytics\", \"action\": \"Store telemetry in workspace\"},\n",
        "    {\"from\": \"Forensics Notebook\", \"to\": \"Log Analytics\", \"action\": \"Run governed KQL query\"},\n",
        "    {\"from\": \"Log Analytics\", \"to\": \"Forensics Notebook\", \"action\": \"Return correlated incidents and metadata\"},\n",
        "    {\"from\": \"Forensics Notebook\", \"to\": \"Platform Owner\", \"action\": \"Share scoped findings and ownership context\"}\n",
        "]\n",
        "\n",
        "pd.DataFrame(sequence)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summarize AI-era workload hotspots by model and capacity\n",
        "\n",
        "AI incidents are cross-domain by default. This query groups dependency calls by model and capacity, then ranks by failure rate and p95 latency so teams can quickly separate platform pressure from application defects."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "client = LogsQueryClient(DefaultAzureCredential())\n",
        "workspace_id = os.getenv(\"LOG_ANALYTICS_WORKSPACE_ID\", \"00000000-0000-0000-0000-000000000000\")\n",
        "\n",
        "kql = \"\"\"\n",
        "AppDependencies\n",
        "| where TimeGenerated > ago(24h)\n",
        "| where Target has \"model\" or Name has \"inference\"\n",
        "| extend Capacity=tostring(Properties[\"capacityId\"]), Model=tostring(Properties[\"modelName\"])\n",
        "| summarize Calls=count(), Failures=countif(Success == false), P95=percentile(DurationMs, 95)\n",
        "    by Capacity, Model\n",
        "| extend FailureRate = todouble(Failures) / Calls\n",
        "| order by FailureRate desc, P95 desc\n",
        "\"\"\"\n",
        "\n",
        "try:\n",
        "    result = client.query_workspace(workspace_id, kql, timespan=timedelta(days=1))\n",
        "    if result.tables:\n",
        "        columns = [c.name for c in result.tables[0].columns]\n",
        "        rows = result.tables[0].rows\n",
        "        df = pd.DataFrame(rows, columns=columns)\n",
        "        display(df.head(10))\n",
        "    else:\n",
        "        print(\"No rows returned.\")\n",
        "except Exception as e:\n",
        "    print(f\"Live query could not be executed: {e}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Export a concise incident timeline for downstream sharing\n",
        "\n",
        "Instead of passing screenshots between teams, export a compact timeline with shared correlation identifiers. This makes incident evidence easier to hand off to release owners, platform teams, and application leads."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "client = LogsQueryClient(DefaultAzureCredential())\n",
        "workspace_id = os.getenv(\"LOG_ANALYTICS_WORKSPACE_ID\", \"00000000-0000-0000-0000-000000000000\")\n",
        "kql = \"\"\"\n",
        "AppEvents\n",
        "| where TimeGenerated > ago(1h)\n",
        "| where Name in (\"DeploymentStarted\", \"DeploymentCompleted\", \"InferenceFailure\")\n",
        "| project TimeGenerated, Name, OperationId, CorrelationId, AppRoleName\n",
        "| order by TimeGenerated asc\n",
        "\"\"\"\n",
        "\n",
        "try:\n",
        "    result = client.query_workspace(workspace_id, kql, timespan=timedelta(hours=1))\n",
        "    if result.tables:\n",
        "        columns = [c.name for c in result.tables[0].columns]\n",
        "        rows = result.tables[0].rows\n",
        "        with open(\"incident_timeline.csv\", \"w\", newline=\"\") as f:\n",
        "            writer = csv.writer(f)\n",
        "            writer.writerow(columns)\n",
        "            writer.writerows(rows)\n",
        "        print(\"Saved incident_timeline.csv\")\n",
        "        display(pd.DataFrame(rows, columns=columns).head())\n",
        "    else:\n",
        "        print(\"No rows returned.\")\n",
        "except Exception as e:\n",
        "    print(f\"Timeline export could not be executed: {e}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "This notebook validated the central architectural claim: Azure Monitor should remain the operational system for collection, alerting, and immediate response, while Fabric-style analytical workflows add value by turning selected telemetry into governed, shareable evidence.\n",
        "\n",
        "Key patterns demonstrated:\n",
        "- narrow telemetry scope instead of bulk duplication\n",
        "- explicit time windows and workspace boundaries\n",
        "- contextual joins with deployment and ownership metadata\n",
        "- summarized AI workload hotspot analysis\n",
        "- exportable incident timelines for cross-team reuse\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "1. Replace placeholder workspace and subscription IDs with your real values.\n",
        "2. Run the narrow Log Analytics proof of concept in a non-production subscription.\n",
        "3. Adapt the KQL to your actual tables and custom dimensions.\n",
        "4. Add deployment metadata, ownership tags, and correlation IDs consistently.\n",
        "5. Measure investigation time, query usefulness, access friction, and ingestion volume over 30 to 90 days."
      ]
    }
  ]
}