{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.13.0" }, "blog_metadata": { "topic": "How the Databricks and Microsoft partnership is reshaping enterprise AI architecture", "slug": "how-the-databricks-and-microsoft-partnership-is-reshaping-en", "generated_by": "LinkedIn Post Generator + Azure OpenAI", "generated_at": "2026-07-28T03:22:09.188Z" } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How the Databricks and Microsoft partnership is reshaping enterprise AI architecture\n", "\n", "This notebook turns the blog post into a hands-on validation workflow. It focuses on the core architectural claim: Azure, Fabric, Power BI, ADLS, and Databricks should be treated as complementary layers in a Microsoft operating model rather than as mutually exclusive platform bets. You will validate workload placement ideas with small Python examples that simulate governed data products, feature engineering, lightweight modeling, and hybrid Azure OpenAI integration patterns." ] }, { "cell_type": "code", "metadata": {}, "source": [ "%pip install -q pandas scikit-learn requests" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import os\n", "import json\n", "import textwrap\n", "import requests\n", "import pandas as pd\n", "from sklearn.linear_model import LogisticRegression\n", "\n", "try:\n", " from pyspark.sql import SparkSession\n", " from pyspark.sql.functions import col, month, to_date\n", " spark_available = True\n", "except Exception:\n", " SparkSession = None\n", " spark_available = False\n", "\n", "print({'spark_available': spark_available, 'pandas_version': pd.__version__})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Architecture framing\n", "\n", "The blog argues that the real decision is workload placement by responsibility: control plane, governed data products, semantic access, and AI operations. This cell captures the enterprise pattern as a Mermaid diagram so teams can review the intended role of Entra ID, Azure resources, Databricks workspace controls, and governed access together." ] }, { "cell_type": "code", "metadata": {}, "source": [ "control_plane_mermaid = '''\n", "flowchart TD\n", " A[Microsoft Entra ID] --> B[Azure Resources]\n", " A --> C[Databricks Workspace]\n", " C --> D[Unity Catalog Policies]\n", " D --> E[Governed Data Access]\n", " E --> F[Training and Inference]\n", " B --> G[Key Vault / Networking / Monitoring]\n", " G --> F\n", "'''\n", "\n", "print(control_plane_mermaid)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Ingest enterprise data into a governed table-first structure\n", "\n", "This example validates the blog's point that enterprise AI improves when raw data becomes a managed table rather than an unmanaged file pile. The code uses Spark and Delta when available, and falls back to pandas so the notebook remains runnable outside Databricks." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Ingest enterprise data into a Delta table in Databricks\n", "if spark_available:\n", " spark = SparkSession.builder.getOrCreate()\n", "\n", " raw_df = spark.createDataFrame(\n", " [\n", " (1, 'contoso', 1200.50, '2026-07-01'),\n", " (2, 'fabrikam', 845.10, '2026-07-01'),\n", " ],\n", " ['invoice_id', 'customer', 'amount', 'invoice_date'],\n", " )\n", "\n", " spark.sql('CREATE SCHEMA IF NOT EXISTS main.finance')\n", " (raw_df.write\n", " .format('delta')\n", " .mode('overwrite')\n", " .saveAsTable('main.finance.raw_invoices'))\n", "\n", " print('Created Delta table: main.finance.raw_invoices')\n", " spark.table('main.finance.raw_invoices').show()\n", "else:\n", " raw_pdf = pd.DataFrame(\n", " [\n", " (1, 'contoso', 1200.50, '2026-07-01'),\n", " (2, 'fabrikam', 845.10, '2026-07-01'),\n", " ],\n", " columns=['invoice_id', 'customer', 'amount', 'invoice_date'],\n", " )\n", " raw_pdf['invoice_date'] = pd.to_datetime(raw_pdf['invoice_date'])\n", " print('Spark not available; using pandas fallback for raw_invoices')\n", " print(raw_pdf)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Apply governance and create a curated table\n", "\n", "This step mirrors a Unity Catalog-style pattern: create a curated table with explicit ownership and access assumptions. In a real Databricks environment, the GRANT statement would enforce access for an analyst group; outside Spark, the notebook simulates the curated dataset and records the intended permission model." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Apply governance with Unity Catalog-style table access in Databricks SQL\n", "catalog = 'main'\n", "schema = 'finance'\n", "table = 'curated_invoices'\n", "\n", "if spark_available:\n", " spark = SparkSession.builder.getOrCreate()\n", " spark.sql(f'CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}')\n", "\n", " spark.sql(f'''\n", " CREATE OR REPLACE TABLE {catalog}.{schema}.{table} AS\n", " SELECT invoice_id, customer, amount, to_date(invoice_date) AS invoice_date\n", " FROM {catalog}.{schema}.raw_invoices\n", " ''')\n", "\n", " grant_sql = f\"GRANT SELECT ON TABLE {catalog}.{schema}.{table} TO `analysts`\"\n", " print('Governed table ready:', f'{catalog}.{schema}.{table}')\n", " print('Planned access statement:', grant_sql)\n", " spark.table(f'{catalog}.{schema}.{table}').show()\n", "else:\n", " curated_pdf = raw_pdf.copy()\n", " curated_pdf['invoice_date'] = pd.to_datetime(curated_pdf['invoice_date']).dt.date\n", " governance_metadata = {\n", " 'table': f'{catalog}.{schema}.{table}',\n", " 'granted_group': 'analysts',\n", " 'policy_intent': 'SELECT'\n", " }\n", " print('Spark not available; simulated governed table ready')\n", " print(governance_metadata)\n", " print(curated_pdf)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Trusted data products and semantic reuse\n", "\n", "The blog emphasizes that the value is not just model execution, but the integrated lifecycle across engineering, curation, governance, and AI. This diagram shows the broader enterprise path from operational sources through ADLS and the Databricks lakehouse into model training, serving, enterprise apps, and Power BI consumption." ] }, { "cell_type": "code", "metadata": {}, "source": [ "lakehouse_mermaid = '''\n", "flowchart TD\n", " A[Operational Data Sources] --> B[Azure Data Lake Storage]\n", " B --> C[Databricks Lakehouse]\n", " C --> D[Delta Tables + Unity Catalog]\n", " D --> E[Feature Engineering / ETL]\n", " E --> F[Model Training / Fine-tuning]\n", " F --> G[Model Serving]\n", " G --> H[Enterprise Apps on Azure]\n", " D --> I[Power BI / SQL Analytics]\n", " J[Microsoft Entra ID] --> C\n", " J --> H\n", " K[Azure OpenAI] --> F\n", " K --> G\n", "'''\n", "\n", "print(lakehouse_mermaid)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Build a feature table close to governed data\n", "\n", "This example validates the cost-discipline argument from the post: feature logic should stay near the governed lakehouse when possible to avoid duplicate semantics and drift. The code derives a simple month feature and a high-value invoice label from the curated invoice table." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Build a simple feature table for enterprise AI workloads\n", "if spark_available:\n", " spark = SparkSession.builder.getOrCreate()\n", " spark.sql('CREATE SCHEMA IF NOT EXISTS main.ml')\n", "\n", " features_df = (\n", " spark.table('main.finance.curated_invoices')\n", " .withColumn('invoice_month', month(col('invoice_date')))\n", " .withColumn('high_value_invoice', (col('amount') > 1000).cast('int'))\n", " )\n", "\n", " (features_df.write\n", " .format('delta')\n", " .mode('overwrite')\n", " .saveAsTable('main.ml.invoice_features'))\n", "\n", " print('Feature table created: main.ml.invoice_features')\n", " spark.table('main.ml.invoice_features').show()\n", "else:\n", " features_pdf = curated_pdf.copy()\n", " features_pdf['invoice_date'] = pd.to_datetime(features_pdf['invoice_date'])\n", " features_pdf['invoice_month'] = features_pdf['invoice_date'].dt.month\n", " features_pdf['high_value_invoice'] = (features_pdf['amount'] > 1000).astype(int)\n", " print('Spark not available; using pandas fallback for invoice_features')\n", " print(features_pdf)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train a lightweight model on lakehouse-derived features\n", "\n", "This cell demonstrates the blog's point that Databricks is strongest when engineering, feature creation, and model work are tightly connected. The example trains a minimal logistic regression model using the feature table; if the sample is too small for a stable fit, the code expands the dataset slightly for demonstration purposes." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Train a lightweight model in Databricks using lakehouse features\n", "if spark_available:\n", " spark = SparkSession.builder.getOrCreate()\n", " pdf = spark.table('main.ml.invoice_features').toPandas()\n", "else:\n", " pdf = features_pdf.copy()\n", "\n", "pdf['invoice_date'] = pd.to_datetime(pdf['invoice_date'])\n", "\n", "if pdf['high_value_invoice'].nunique() < 2:\n", " extra = pd.DataFrame([\n", " {'invoice_id': 3, 'customer': 'adatum', 'amount': 300.0, 'invoice_date': pd.Timestamp('2026-07-02'), 'invoice_month': 7, 'high_value_invoice': 0},\n", " {'invoice_id': 4, 'customer': 'northwind', 'amount': 2200.0, 'invoice_date': pd.Timestamp('2026-07-03'), 'invoice_month': 7, 'high_value_invoice': 1},\n", " {'invoice_id': 5, 'customer': 'tailspin', 'amount': 1500.0, 'invoice_date': pd.Timestamp('2026-08-01'), 'invoice_month': 8, 'high_value_invoice': 1},\n", " {'invoice_id': 6, 'customer': 'wingtip', 'amount': 450.0, 'invoice_date': pd.Timestamp('2026-08-02'), 'invoice_month': 8, 'high_value_invoice': 0},\n", " ])\n", " pdf = pd.concat([pdf, extra], ignore_index=True)\n", "\n", "X = pdf[['amount', 'invoice_month']]\n", "y = pdf['high_value_invoice']\n", "\n", "model = LogisticRegression()\n", "model.fit(X, y)\n", "\n", "print('Training rows:', len(pdf))\n", "print('Model coefficients:', model.coef_.tolist())\n", "print('Model intercept:', model.intercept_.tolist())\n", "print('Predictions:', model.predict(X).tolist())" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Required environment variables for hybrid Azure OpenAI validation\n", "\n", "Set these variables before running the next cell if you want to test a live endpoint:\n", "\n", "- `AZURE_OPENAI_ENDPOINT`\n", "- `AZURE_OPENAI_API_KEY`\n", "- `AZURE_OPENAI_DEPLOYMENT` (optional, defaults to `gpt-4o-mini`)\n", "- `AZURE_OPENAI_API_VERSION` (optional, defaults to `2024-02-15-preview`)\n", "\n", "If these are not set, the code will print the prepared request details without making a network call." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Validate a hybrid Azure OpenAI integration pattern\n", "\n", "The blog argues that model access alone is not the moat; business context and governed data are. This example shows how a Databricks-style workflow could call Azure OpenAI while keeping the prompt grounded in enterprise architecture language. The cell is safe to run without secrets because it falls back to a dry-run mode." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Call an Azure OpenAI endpoint from a Databricks notebook for hybrid AI patterns\n", "endpoint = os.getenv('AZURE_OPENAI_ENDPOINT', 'https://contoso-openai.openai.azure.com').rstrip('/')\n", "api_key = os.getenv('AZURE_OPENAI_API_KEY')\n", "deployment = os.getenv('AZURE_OPENAI_DEPLOYMENT', 'gpt-4o-mini')\n", "api_version = os.getenv('AZURE_OPENAI_API_VERSION', '2024-02-15-preview')\n", "url = f'{endpoint}/openai/deployments/{deployment}/chat/completions?api-version={api_version}'\n", "\n", "payload = {\n", " 'messages': [\n", " {'role': 'user', 'content': 'Summarize why governed lakehouse data improves enterprise AI.'}\n", " ],\n", " 'max_tokens': 80,\n", "}\n", "\n", "if api_key:\n", " try:\n", " response = requests.post(\n", " url,\n", " headers={'api-key': api_key, 'Content-Type': 'application/json'},\n", " json=payload,\n", " timeout=30,\n", " )\n", " print('status_code:', response.status_code)\n", " try:\n", " print(response.json())\n", " except Exception:\n", " print(response.text[:1000])\n", " except Exception as e:\n", " print('Request failed:', repr(e))\n", "else:\n", " print('Dry run only: AZURE_OPENAI_API_KEY not set')\n", " print('Prepared URL:', url)\n", " print('Prepared payload:', json.dumps(payload, indent=2))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Runtime path for AI application delivery\n", "\n", "This sequence diagram captures the end-to-end placement logic from the post: enterprise app, model serving, governed lakehouse retrieval, and Azure OpenAI are separate responsibilities. That separation is exactly why a Microsoft operating model can include multiple execution environments without collapsing governance." ] }, { "cell_type": "code", "metadata": {}, "source": [ "runtime_sequence_mermaid = '''\n", "sequenceDiagram\n", " participant U as Enterprise User\n", " participant A as Azure App Service\n", " participant D as Databricks Model Serving\n", " participant O as Azure OpenAI\n", " participant L as Lakehouse Data\n", "\n", " U->>A: Submit business question\n", " A->>D: Request prediction/context\n", " D->>L: Retrieve governed features\n", " D->>O: Enrich with LLM reasoning\n", " O-->>D: Generated response\n", " D-->>A: Prediction + explanation\n", " A-->>U: AI-powered result\n", "'''\n", "\n", "print(runtime_sequence_mermaid)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Workload placement checklist\n", "\n", "Use this quick validation checklist from the blog before approving an architecture:\n", "\n", "1. Is there a named owner for identity and access?\n", "2. Is there a named steward for the data product behind the workload?\n", "3. Is there a named operator for model or app runtime support?\n", "4. Is there one policy path for audit, retention, and lifecycle?\n", "5. Is the design duplicating semantics or pipelines without a business reason?\n", "\n", "A practical interpretation is:\n", "\n", "- Azure = enterprise control and cloud operating foundation\n", "- Fabric = Microsoft-native analytics and consumption layer\n", "- Databricks = data and AI execution environment where its model fits\n", "- Power BI = BI consumption where it fits\n", "- ADLS = durable data foundation\n", "\n", "## Next Steps\n", "\n", "- Replace the toy invoice data with a real governed dataset.\n", "- Map one current AI workload to control plane, data product owner, semantic owner, and runtime operator.\n", "- Decide whether the workload belongs in Databricks, Fabric, or another Azure-native path based on responsibility and operating constraints.\n", "- Document the boundary so teams do not rebuild the same logic in parallel." ] } ] }