{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.13.0"
    },
    "blog_metadata": {
      "topic": "Why OneLake Catalog Matters More Than Another Fabric Feature Demo",
      "slug": "why-onelake-catalog-matters-more-than-another-fabric-feature",
      "generated_by": "LinkedIn Post Generator + Azure OpenAI",
      "generated_at": "2026-07-23T19:06:17.968Z"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Why OneLake Catalog Matters More Than Another Fabric Feature Demo\n",
        "\n",
        "This notebook turns the blog post into a hands-on validation flow using Python. The goal is to test the central claim: polished feature demos create attention, but catalog context creates trust, reuse, and operational durability across teams."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install pandas matplotlib networkx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from dataclasses import dataclass\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import networkx as nx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. Demos show features; catalogs make assets reusable\n",
        "\n",
        "This example summarizes a few assets with ownership, domain, and format. It validates the idea that a platform becomes useful when assets are understandable and reusable, not just technically present."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "assets = [\n",
        "    {\"name\": \"sales_orders\", \"owner\": \"data-eng\", \"domain\": \"sales\", \"format\": \"delta\"},\n",
        "    {\"name\": \"customer_dim\", \"owner\": \"analytics\", \"domain\": \"crm\", \"format\": \"parquet\"},\n",
        "    {\"name\": \"inventory_snapshot\", \"owner\": \"ops\", \"domain\": \"supply\", \"format\": \"delta\"},\n",
        "]\n",
        "\n",
        "for asset in assets:\n",
        "    summary = (\n",
        "        f\"{asset['name']} | domain={asset['domain']} | \"\n",
        "        f\"owner={asset['owner']} | format={asset['format']}\"\n",
        "    )\n",
        "    print(summary)\n",
        "\n",
        "print(\"Insight: demos show features; catalogs make assets reusable.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. Minimal metadata model for catalog value\n",
        "\n",
        "This example uses a small dataclass to show the difference between storage location and usable context. The path says where the asset lives, while owner and sensitivity help determine whether it can be trusted and reused."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from dataclasses import dataclass\n",
        "\n",
        "@dataclass\n",
        "class CatalogEntry:\n",
        "    name: str\n",
        "    path: str\n",
        "    owner: str\n",
        "    sensitivity: str\n",
        "\n",
        "entry = CatalogEntry(\n",
        "    name=\"sales_orders\",\n",
        "    path=\"onelake://contoso/lakehouse/Tables/sales_orders\",\n",
        "    owner=\"data-eng\",\n",
        "    sensitivity=\"internal\",\n",
        ")\n",
        "\n",
        "print(entry)\n",
        "print(f\"Discoverable asset: {entry.name} owned by {entry.owner}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. Search by business meaning instead of storage path\n",
        "\n",
        "A useful catalog supports business-oriented discovery. This example filters assets by tags such as finance, which is closer to how analysts think than raw file paths or table locations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "catalog = [\n",
        "    {\"name\": \"sales_orders\", \"tags\": {\"gold\", \"finance\", \"pii\"}},\n",
        "    {\"name\": \"sales_returns\", \"tags\": {\"silver\", \"finance\"}},\n",
        "    {\"name\": \"web_events\", \"tags\": {\"bronze\", \"marketing\"}},\n",
        "]\n",
        "\n",
        "query_tag = \"finance\"\n",
        "matches = [item[\"name\"] for item in catalog if query_tag in item[\"tags\"]]\n",
        "\n",
        "print(f\"Assets tagged '{query_tag}':\")\n",
        "for name in matches:\n",
        "    print(\"-\", name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. Governance checks: ownership matters\n",
        "\n",
        "This Python version of the governance check highlights a common failure mode: assets without accountable owners. In production, these are often the assets that create audit, migration, and AI-readiness problems."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "assets = [\n",
        "    {\"Name\": \"sales_orders\", \"Owner\": \"data-eng\", \"Sensitivity\": \"Internal\"},\n",
        "    {\"Name\": \"customer_dim\", \"Owner\": \"\", \"Sensitivity\": \"Confidential\"},\n",
        "    {\"Name\": \"web_events\", \"Owner\": \"marketing-bi\", \"Sensitivity\": \"Public\"},\n",
        "]\n",
        "\n",
        "for asset in assets:\n",
        "    if not str(asset[\"Owner\"]).strip():\n",
        "        print(f\"FAIL: {asset['Name']} has no owner\")\n",
        "    else:\n",
        "        print(f\"PASS: {asset['Name']} owned by {asset['Owner']}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. Score assets by catalog readiness\n",
        "\n",
        "This example scores assets on three simple dimensions: owner, tags, and lineage. It provides a lightweight way to assess whether important data products are actually ready for trusted reuse."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "assets = [\n",
        "    {\"Name\": \"sales_orders\", \"HasOwner\": True, \"HasTags\": True, \"HasLineage\": True},\n",
        "    {\"Name\": \"inventory_snapshot\", \"HasOwner\": True, \"HasTags\": False, \"HasLineage\": True},\n",
        "    {\"Name\": \"ad_hoc_extract\", \"HasOwner\": False, \"HasTags\": False, \"HasLineage\": False},\n",
        "]\n",
        "\n",
        "for asset in assets:\n",
        "    score = 0\n",
        "    if asset[\"HasOwner\"]:\n",
        "        score += 1\n",
        "    if asset[\"HasTags\"]:\n",
        "        score += 1\n",
        "    if asset[\"HasLineage\"]:\n",
        "        score += 1\n",
        "    print(f\"{asset['Name']}: catalog-readiness={score}/3\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 6. Lineage creates trust across teams and tools\n",
        "\n",
        "Lineage helps users understand where a dataset came from and what upstream dependencies exist. That context is essential when teams need to trust outputs or assess the impact of changes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "lineage = {\n",
        "    \"sales_mart\": [\"sales_orders_clean\", \"customer_dim\"],\n",
        "    \"sales_orders_clean\": [\"sales_orders_raw\"],\n",
        "    \"customer_dim\": [\"crm_export\"],\n",
        "}\n",
        "\n",
        "target = \"sales_mart\"\n",
        "print(f\"Lineage for {target}:\")\n",
        "for upstream in lineage[target]:\n",
        "    print(f\"{target} <- {upstream}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 7. Visualize the catalog-centered operating model\n",
        "\n",
        "The blog argues that the catalog sits between raw storage and enterprise reuse. This diagram is recreated in Python as a directed graph to validate that discovery, governance, and access all depend on shared metadata."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "edges = [\n",
        "    (\"Raw files in OneLake\", \"Catalog scans tables/files\"),\n",
        "    (\"Catalog scans tables/files\", \"Unified metadata: schema, owner, tags\"),\n",
        "    (\"Unified metadata: schema, owner, tags\", \"Discovery across Fabric experiences\"),\n",
        "    (\"Unified metadata: schema, owner, tags\", \"Governance and lineage\"),\n",
        "    (\"Unified metadata: schema, owner, tags\", \"Consistent SQL / Spark access\"),\n",
        "    (\"Discovery across Fabric experiences\", \"Faster reuse than isolated feature demos\"),\n",
        "    (\"Governance and lineage\", \"Faster reuse than isolated feature demos\"),\n",
        "    (\"Consistent SQL / Spark access\", \"Faster reuse than isolated feature demos\"),\n",
        "]\n",
        "\n",
        "G = nx.DiGraph()\n",
        "G.add_edges_from(edges)\n",
        "\n",
        "plt.figure(figsize=(14, 8))\n",
        "pos = nx.spring_layout(G, seed=42, k=1.2)\n",
        "nx.draw(\n",
        "    G,\n",
        "    pos,\n",
        "    with_labels=True,\n",
        "    node_size=3500,\n",
        "    node_color=\"#DCEBFA\",\n",
        "    font_size=9,\n",
        "    arrows=True,\n",
        "    arrowstyle=\"-|>\",\n",
        "    arrowsize=18,\n",
        ")\n",
        "plt.title(\"Catalog-Centered Reuse Model\")\n",
        "plt.axis(\"off\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 8. Producer-to-analyst flow through OneLake and the catalog\n",
        "\n",
        "This example converts the sequence diagram into a simple event trace. It shows how publishing, scanning, indexing, searching, and querying fit together in a governed discovery flow."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "events = [\n",
        "    (\"Producer\", \"OneLake\", \"Publish table/files\"),\n",
        "    (\"OneLake\", \"Catalog\", \"Expose metadata for scan/index\"),\n",
        "    (\"Catalog\", \"Catalog\", \"Store schema, tags, lineage\"),\n",
        "    (\"Analyst\", \"Catalog\", 'Search \"finance gold tables\"'),\n",
        "    (\"Catalog\", \"Analyst\", \"Return trusted assets\"),\n",
        "    (\"Analyst\", \"OneLake\", \"Query selected asset\"),\n",
        "]\n",
        "\n",
        "for src, dst, action in events:\n",
        "    print(f\"{src} -> {dst}: {action}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 9. Metadata compounds while demos expire\n",
        "\n",
        "The blog's core platform argument is that catalog value grows over time because it improves discovery, trust, and reuse across many experiences. This example prints a simple value progression and visualizes it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "feature_demo_value = 1\n",
        "catalog_value_over_time = [1, 2, 4, 7, 11]\n",
        "\n",
        "print(f\"One-time feature demo value: {feature_demo_value}\")\n",
        "for month, value in enumerate(catalog_value_over_time, start=1):\n",
        "    print(f\"Month {month}: catalog value index = {value}\")\n",
        "\n",
        "print(\"Takeaway: OneLake Catalog improves discovery, trust, and reuse over time.\")\n",
        "\n",
        "months = list(range(1, len(catalog_value_over_time) + 1))\n",
        "plt.figure(figsize=(8, 4))\n",
        "plt.plot(months, catalog_value_over_time, marker=\"o\", label=\"Catalog value over time\")\n",
        "plt.axhline(feature_demo_value, linestyle=\"--\", color=\"red\", label=\"One-time feature demo value\")\n",
        "plt.xticks(months)\n",
        "plt.xlabel(\"Month\")\n",
        "plt.ylabel(\"Value index\")\n",
        "plt.title(\"Why Catalog Value Compounds\")\n",
        "plt.legend()\n",
        "plt.grid(True, alpha=0.3)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 10. Turn the executive test into a reusable scorecard\n",
        "\n",
        "The post proposes a deliberately boring but practical test: can users find, understand, trust, and safely use the right asset? This code turns those questions into a small scorecard for validation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "assets = [\n",
        "    {\n",
        "        \"name\": \"sales_orders\",\n",
        "        \"approved\": True,\n",
        "        \"business_description\": True,\n",
        "        \"owner\": True,\n",
        "        \"allowed_to_use\": True,\n",
        "        \"lineage_context\": True,\n",
        "    },\n",
        "    {\n",
        "        \"name\": \"legacy_patient_gold\",\n",
        "        \"approved\": False,\n",
        "        \"business_description\": False,\n",
        "        \"owner\": False,\n",
        "        \"allowed_to_use\": False,\n",
        "        \"lineage_context\": False,\n",
        "    },\n",
        "    {\n",
        "        \"name\": \"inventory_snapshot\",\n",
        "        \"approved\": True,\n",
        "        \"business_description\": True,\n",
        "        \"owner\": True,\n",
        "        \"allowed_to_use\": True,\n",
        "        \"lineage_context\": False,\n",
        "    },\n",
        "]\n",
        "\n",
        "rows = []\n",
        "for asset in assets:\n",
        "    score = sum([\n",
        "        asset[\"approved\"],\n",
        "        asset[\"business_description\"],\n",
        "        asset[\"owner\"],\n",
        "        asset[\"allowed_to_use\"],\n",
        "        asset[\"lineage_context\"],\n",
        "    ])\n",
        "    rows.append({\"asset\": asset[\"name\"], \"trust_score\": score, \"max_score\": 5})\n",
        "\n",
        "scorecard = pd.DataFrame(rows)\n",
        "print(scorecard)\n",
        "\n",
        "plt.figure(figsize=(8, 4))\n",
        "plt.bar(scorecard[\"asset\"], scorecard[\"trust_score\"], color=\"#4C78A8\")\n",
        "plt.ylim(0, 5)\n",
        "plt.ylabel(\"Trust score\")\n",
        "plt.title(\"Executive Test for Trusted Reuse\")\n",
        "plt.grid(axis=\"y\", alpha=0.3)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 11. Simulate duplicate creation when trusted assets are hard to find\n",
        "\n",
        "One of the main failure modes in the post is duplication. This example shows how missing metadata can push teams to create new copies instead of reusing approved assets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "catalog_assets = [\n",
        "    {\"name\": \"sales_orders\", \"trusted\": True, \"easy_to_find\": True},\n",
        "    {\"name\": \"sales_orders_v2_copy\", \"trusted\": False, \"easy_to_find\": True},\n",
        "    {\"name\": \"sales_orders_final_final\", \"trusted\": False, \"easy_to_find\": True},\n",
        "    {\"name\": \"customer_dim\", \"trusted\": True, \"easy_to_find\": False},\n",
        "]\n",
        "\n",
        "for asset in catalog_assets:\n",
        "    status = \"reuse\" if asset[\"trusted\"] and asset[\"easy_to_find\"] else \"risk of duplication\"\n",
        "    print(f\"{asset['name']}: {status}\")\n",
        "\n",
        "trusted_and_findable = [a for a in catalog_assets if a[\"trusted\"] and a[\"easy_to_find\"]]\n",
        "print(f\"\\nTrusted and easy-to-find assets: {len(trusted_and_findable)}/{len(catalog_assets)}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "This notebook validated the blog's main argument with simple, testable examples: metadata, ownership, tags, lineage, and business context are what make Fabric assets reusable at enterprise scale. A feature demo may prove that something works once, but a strong catalog helps people find the right asset, trust it, and use it safely across domains.\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "- Replace the sample asset lists with your own top 20 Fabric assets.\n",
        "- Add real metadata fields such as steward, domain, certification status, and workspace.\n",
        "- Score your assets for owner, tags, lineage, and business description completeness.\n",
        "- Identify duplicate or orphaned assets and define remediation actions.\n",
        "- Extend the notebook to pull metadata from your actual governance or catalog sources."
      ]
    }
  ]
}