{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Operating a Fabric + Databricks estate \u2014 a runnable runbook\n", "\n", "**Version 1.1 \u00b7 2026-09-19 \u00b7 Frank Garofalo**\n", "\n", "This is a runbook, not a checklist. Some cells explain a decision, some run a\n", "check that returns an answer, and the checks name an owner so an unowned one\n", "shows up as a gap. Edit the parameters cell below and run top to bottom;\n", "nothing else should need changing.\n", "\n", "It is written for the architecture argued in the companion post: Databricks as\n", "the engineering engine, Fabric as the BI and governance surface, one governed\n", "data layer underneath, and Entra ID as the single identity plane.\n", "\n", "---\n", "\n", "### Three things this runbook exists to stop you getting wrong\n", "\n", "**1. \"Mirroring\" is three different mechanisms.** *Metadata mirroring* (Azure\n", "Databricks Unity Catalog) mirrors catalog structure and reaches the data through\n", "OneLake shortcuts \u2014 nothing is copied. *Database mirroring* (Snowflake, Azure SQL,\n", "Cosmos DB, Oracle, SAP, BigQuery and the rest) continuously replicates into OneLake\n", "as Delta, publishing as fast as every 15 seconds \u2014 not a snapshot. *Open mirroring*\n", "has you land change data in a OneLake zone yourself. They have different failure\n", "modes, storage costs and staleness behavior. Reasoning about one using another's\n", "model is the most common architecture error in this pattern.\n", "\n", "**2. Access control at the seam is three outcomes, not one.** Table, schema and\n", "catalog GRANTs do not carry over, and every Fabric query against mirrored data runs\n", "as the *connection* credential rather than the end user. Tables carrying row filters\n", "or column masks **fail closed** \u2014 Databricks supports neither Unity REST API nor\n", "path-based access to them, and unsupported clients return no data. The one real\n", "bypass is a **direct ADLS read**, where Unity Catalog policies are not enforced at\n", "the storage layer. So re-implement in OneLake security \u2014 and in review, hunt for a\n", "filtered table someone unblocked by dropping the filter.\n", "\n", "**3. OneLake security covers row- and column-level controls.** Microsoft's\n", "documentation (rev. 2026-08-19) describes RLS and CLS as working controls with\n", "defined per-engine enforcement, not preview. Two caveats worth confirming for your\n", "own tenant: status for Mirrored Azure Databricks Catalog items specifically, and\n", "that GCC High preview still lacks Spark support for OneLake security.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parameters\n", "\n", "Everything environment-specific lives here. This cell is tagged `parameters`,\n", "so it also works with papermill-style injection if you schedule this notebook.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "parameters" ] }, "outputs": [], "source": [ "# --- Fabric -----------------------------------------------------------\n", "FABRIC_WORKSPACE_ID = \"\" # GUID of the Fabric workspace holding the mirror\n", "FABRIC_LAKEHOUSE_NAME = \"\" # lakehouse that consumes the mirrored data\n", "FABRIC_CAPACITY_ID = \"\" # capacity backing the workspace (for the FinOps cell)\n", "\n", "# --- Databricks -------------------------------------------------------\n", "DATABRICKS_WORKSPACE_URL = \"\" # https://adb-..azuredatabricks.net\n", "DATABRICKS_CATALOG = \"\" # Unity Catalog catalog being mirrored\n", "DATABRICKS_SCHEMA = \"\" # schema scoped for external use (Gold only)\n", "\n", "# --- Governance -------------------------------------------------------\n", "# The service principal that OWNS the mirroring connection. Every Fabric query\n", "# against mirrored data runs as THIS identity, so it must be least-privilege.\n", "# Use the APPLICATION (client) ID -- Unity Catalog does not resolve the Entra\n", "# object ID, and SHOW GRANTS with the wrong one returns empty rather than error.\n", "MIRROR_CONNECTION_SPN_APPLICATION_ID = \"\"\n", "\n", "# --- Cloud -------------------------------------------------------------\n", "# \"commercial\" or \"gcc-high\". GCC High uses a different Fabric REST endpoint\n", "# and a different portal, so getting this wrong fails with an auth error that\n", "# looks like a permissions problem.\n", "FABRIC_CLOUD = \"commercial\"\n", "\n", "# --- Thresholds the checks below alert on ------------------------------\n", "MIRROR_STALENESS_ALERT_MINUTES = 60 # replicated mirrors only; see cell 3\n", "CAPACITY_UTILISATION_ALERT_PCT = 80 # sustained, not instantaneous\n", "ACCESS_REVIEW_MAX_AGE_DAYS = 90\n", "\n", "# --- Owners: who gets paged. Fill these in; an unowned check is decoration.\n", "OWNER_PLATFORM = \"\" # e.g. \"data-platform@example.com\"\n", "OWNER_GOVERNANCE = \"\"\n", "OWNER_FINOPS = \"\"\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n", "\n", "Runs inside a Fabric notebook (uses `notebookutils` for the token when present)\n", "and degrades to an explicit message elsewhere, so you can read the notebook\n", "outside Fabric without it throwing.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json, datetime\n", "\n", "# The Fabric REST endpoint is DIFFERENT in GCC High, and a runbook whose\n", "# companion post is largely about GCC High had the commercial one hard-coded.\n", "# Sovereign endpoints per Microsoft Learn, 'Microsoft Fabric for US Government\n", "# GCC High customers' (rev. 2026-09-03).\n", "FABRIC_API_BY_CLOUD = {\n", " 'commercial': 'https://api.fabric.microsoft.com/v1',\n", " 'gcc-high': 'https://highapi.fabric.microsoft.us/v1',\n", "}\n", "if FABRIC_CLOUD not in FABRIC_API_BY_CLOUD:\n", " raise ValueError(\n", " f'FABRIC_CLOUD={FABRIC_CLOUD!r} is not one of '\n", " f'{sorted(FABRIC_API_BY_CLOUD)} - set it in the parameters cell'\n", " )\n", "FABRIC_API = FABRIC_API_BY_CLOUD[FABRIC_CLOUD]\n", "print(f'Fabric API for {FABRIC_CLOUD}: {FABRIC_API}')\n", "\n", "def _token(audience: str = 'pbi') -> str | None:\n", " \"\"\"Fabric-native token when available; None elsewhere.\"\"\"\n", " try:\n", " import notebookutils # available inside Fabric notebooks\n", " return notebookutils.credentials.getToken(audience)\n", " except Exception:\n", " return None\n", "\n", "def _get(url: str):\n", " import urllib.request\n", " tok = _token()\n", " if not tok:\n", " print('NOT RUNNING IN FABRIC - no token. Cells that call the API will skip.')\n", " return None\n", " req = urllib.request.Request(url, headers={'Authorization': f'Bearer {tok}'})\n", " with urllib.request.urlopen(req, timeout=60) as r:\n", " return json.loads(r.read().decode('utf-8'))\n", "\n", "def check(name: str, ok: bool, detail: str = '', owner: str = ''):\n", " \"\"\"One line per check. An unowned check is decoration, so owner prints.\"\"\"\n", " mark = 'PASS' if ok else 'FAIL'\n", " print(f'[{mark}] {name}' + (f' - {detail}' if detail else '')\n", " + (f' (owner: {owner})' if owner else ''))\n", " return ok\n", "\n", "print('runbook loaded', datetime.datetime.now(datetime.timezone.utc).isoformat())\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Identity \u2014 who is your mirror actually running as?\n", "\n", "**This is the item that matters most.** Every Fabric query against mirrored\n", "Databricks data executes under the\n", "credentials that configured the mirroring connection \u2014 not the end user's. If\n", "that identity is a human admin or an over-scoped service principal, then every\n", "Fabric consumer effectively inherits that access.\n", "\n", "What good looks like:\n", "\n", "- a dedicated **service principal**, used for nothing else\n", "- `EXTERNAL USE SCHEMA` on the specific Gold **schema**, not on the catalog\n", "- `USE CATALOG` on the catalog and `USE SCHEMA` on the schema \u2014 these are\n", " **required**, so seeing them is correct, not over-scoped\n", "- no workspace-admin, no `ALL PRIVILEGES`, no account-admin\n", "- rotated on the same cadence as your other platform credentials\n", "\n", "> **Use the application ID, not the object ID.** Unity Catalog identifies a service\n", "> principal by its **application (client) ID** \u2014 the same GUID Databricks shows in\n", "> its service-principal list. Entra's *object* ID is a different GUID for the same\n", "> identity, and `SHOW GRANTS` with it returns an empty result rather than an error \u2014\n", "> a clean-looking audit for an identity nothing ever looked at.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Run this in Databricks (SQL editor or a Databricks notebook), not in Fabric.\n", "# It shows exactly what the mirroring identity can reach.\n", "AUDIT_SQL = f'''\n", "SHOW GRANTS `{MIRROR_CONNECTION_SPN_APPLICATION_ID}` ON SCHEMA {DATABRICKS_CATALOG}.{DATABRICKS_SCHEMA};\n", "SHOW GRANTS `{MIRROR_CONNECTION_SPN_APPLICATION_ID}` ON CATALOG {DATABRICKS_CATALOG};\n", "'''\n", "print(AUDIT_SQL)\n", "print()\n", "print('Expected on the SCHEMA : EXTERNAL USE SCHEMA, USE SCHEMA, SELECT')\n", "print('Expected on the CATALOG: USE CATALOG -- and nothing else.')\n", "print()\n", "print('USE CATALOG at catalog scope is REQUIRED for external access, so it is not')\n", "print('a finding. What IS a finding: EXTERNAL USE SCHEMA granted at CATALOG scope')\n", "print('(it should be per-schema), ALL PRIVILEGES anywhere, or MANAGE/OWNER.')\n", "print()\n", "print('An EMPTY result is not a pass. Check you used the APPLICATION id.')\n", "check('mirror SPN application id is configured',\n", " bool(MIRROR_CONNECTION_SPN_APPLICATION_ID),\n", " 'set MIRROR_CONNECTION_SPN_APPLICATION_ID', OWNER_GOVERNANCE)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Which mirror do you have? \u2014 answer this before any other check\n", "\n", "Everything downstream depends on the answer, and they look similar in the portal.\n", "\n", "| | Metadata mirroring | Database mirroring | Open mirroring |\n", "|---|---|---|---|\n", "| Sources | Azure Databricks UC, Dremio | Snowflake, Azure SQL, Cosmos DB, Oracle, SAP, BigQuery, SQL Server | anything you write an adapter for |\n", "| What moves | catalog structure + shortcuts | the data itself, continuously | change data you land yourself |\n", "| Data copied into OneLake | **No** | **Yes**, as Delta | **Yes**, as Delta |\n", "| Storage cost in OneLake | none for the data | replicated size (free to 1 TB per CU) | replicated size |\n", "| Meaningful staleness clock | no \u2014 reads pass through | **yes** \u2014 as fast as 15s | **yes** \u2014 your feed's cadence |\n", "| \"Resync\" exists | **No** | Yes | you re-land it |\n", "\n", "If someone hands you a procedure with a resync step for a Databricks mirror,\n", "that procedure was written for one of the other mechanisms.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "items = _get(f'{FABRIC_API}/workspaces/{FABRIC_WORKSPACE_ID}/items') if FABRIC_WORKSPACE_ID else None\n", "if items:\n", " mirrors = [i for i in items.get('value', []) if 'Mirrored' in (i.get('type') or '')]\n", " for m in mirrors:\n", " kind = m.get('type')\n", " replicated = 'Databricks' not in kind\n", " print(f\"{m.get('displayName')}: {kind}\")\n", " print(' mechanism :', 'REPLICATION (data copied)' if replicated\n", " else 'METADATA + SHORTCUT (no copy, no resync)')\n", " print(' staleness :', f'alert at {MIRROR_STALENESS_ALERT_MINUTES} min'\n", " if replicated else 'not applicable - reads pass through to source')\n", " check('at least one mirrored item found', bool(mirrors), owner=OWNER_PLATFORM)\n", "else:\n", " print('Skipped - set FABRIC_WORKSPACE_ID and run inside Fabric.')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Security \u2014 re-implement, do not test propagation\n", "\n", "Unity Catalog policies are **not** enforced for Fabric consumers of mirrored\n", "data. There is no propagation to test. The action is to re-implement the\n", "equivalent controls in OneLake security. But the three outcomes differ, and the\n", "one people expect is the one that does not happen:\n", "\n", "| Object / path | What happens | What you do |\n", "|---|---|---|\n", "| Table, schema, catalog GRANTs | do not carry over; every reader is the connection identity | re-author in Fabric's model; scope that SPN |\n", "| Tables with row filters / column masks | **fail closed** \u2014 unreadable via Unity REST API or by path | inventory them *before* mirroring and agree the answer |\n", "| A direct ADLS read (trusted workspace access) | UC policy **not enforced at the storage layer** | this is the real bypass \u2014 take it to security |\n", "\n", "**The failure mode to hunt for in review:** a table that fails closed looks broken,\n", "not protected. Someone blocked on a deadline drops the row filter or lands an\n", "unfiltered copy for Fabric to read. No error, no change record, nothing alerts.\n", "\n", "Confirm for your own tenant before relying on any of this:\n", "\n", "- status for **Mirrored Azure Databricks Catalog** items specifically\n", "- in **GCC High**, preview still lacks **Spark** support for OneLake security\n", "- whether **cross-engine ABAC** (Beta as of 2026-09-11) is enabled \u2014 it enforces\n", " filters and masks for external engines and changes the second row above\n", "\n", "The cell below does not call an API. It prints the comparison to make by hand:\n", "every Unity Catalog row or column rule on the mirrored schema against the OneLake\n", "security roles on the consuming item.\n", "\n", "Compare the roles on the lakehouse against the Unity Catalog RLS/CLS rules on the\n", "source schema. **Any UC rule without an equivalent here is UNENFORCED for Fabric\n", "users.** OneLake security roles are managed per-item in the Fabric portal, under\n", "*Lakehouse > Manage OneLake data access*. Record the comparison in your access\n", "review: an empty role list next to a non-empty UC policy set is a finding, not a\n", "configuration choice.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f'Source to compare against: {DATABRICKS_CATALOG}.{DATABRICKS_SCHEMA}')\n", "check('governance owner assigned', bool(OWNER_GOVERNANCE),\n", " 'nobody is paged for a security gap', OWNER_GOVERNANCE)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Cost \u2014 measure the two curves separately, bill them together\n", "\n", "Fabric bills capacity units with smoothing and background contention.\n", "Databricks bills DBUs. They are different shapes and a single blended number\n", "hides which one is moving.\n", "\n", "**Mirroring compute is free**, and\n", "mirrored storage is free up to a capacity-based limit. \"A stale mirror triggers\n", "expensive re-sync\" is not a cost model to plan around \u2014 and on the Databricks\n", "path there is no resync at all.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if FABRIC_CAPACITY_ID:\n", " print('Use the Fabric Capacity Metrics app for the authoritative view.')\n", " print(f'Alert threshold configured here: {CAPACITY_UTILISATION_ALERT_PCT}% sustained.')\n", " print('Sustained matters - instantaneous spikes are normal under smoothing and')\n", " print('alerting on them trains people to ignore the alert.')\n", "else:\n", " print('Skipped - set FABRIC_CAPACITY_ID.')\n", "check('FinOps owner assigned', bool(OWNER_FINOPS), owner=OWNER_FINOPS)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Access review\n", "\n", "Run on a cadence, not after an incident. The review has to cover the mirroring\n", "identity from section 1 \u2014 it is the account most likely to accumulate scope and\n", "least likely to be noticed, because no human logs in as it.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f'Access review cadence: every {ACCESS_REVIEW_MAX_AGE_DAYS} days')\n", "print()\n", "print('Cover, at minimum:')\n", "for item in [\n", " 'the mirroring connection identity and its EXTERNAL USE SCHEMA grant',\n", " 'OneLake security roles on every lakehouse consuming mirrored data',\n", " 'Entra group membership behind Fabric workspace roles',\n", " 'any Fabric consumer that reads mirrored data without a OneLake role',\n", "]:\n", " print(f' - {item}')\n", "check('platform owner assigned', bool(OWNER_PLATFORM), owner=OWNER_PLATFORM)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Incident: a mirror looks wrong\n", "\n", "Branch on the mechanism from section 2. There are three, they fail in different\n", "ways, and almost nothing transfers between them.\n", "\n", "**Database mirroring \u2014 replicated (Snowflake, Azure SQL, Cosmos DB, Oracle, ...)**\n", "1. Check replication lag against `MIRROR_STALENESS_ALERT_MINUTES`.\n", "2. Check source availability \u2014 a paused or unreachable source stops replication.\n", " So does a paused or deleted Fabric capacity.\n", "3. A table missing entirely is usually a source-side DDL change or a dropped\n", " permission, not lag. Confirm the table still exists and is still in scope for\n", " the mirror before treating it as a staleness problem.\n", "4. After recovery, confirm replication has caught up to current before trusting\n", " downstream reports \u2014 this is a continuous change feed, not a snapshot, so\n", " there is no single point at which it is 'done'. Compare a known-recent row.\n", "5. Re-confirm OneLake security roles still exist on the target item.\n", "\n", "**Metadata mirroring \u2014 Databricks (catalog + shortcut)**\n", "1. There is no replication lag and no resync. Reads pass through to the source.\n", "2. A failure here is almost always *access*, not staleness \u2014 check the mirroring\n", " identity's grants first (section 1).\n", "3. Check the source table still exists and the connection identity still has\n", " `EXTERNAL USE SCHEMA` on its schema.\n", "4. A table that is simply absent is expected for materialized views, streaming\n", " tables and non-Delta external tables: those never appear. Confirm what kind\n", " of object it is before investigating further.\n", "5. A table carrying a row filter or column mask is also unreadable through this\n", " path, by design. It fails closed. Do not fix it by dropping the filter or\n", " landing an unfiltered copy. Escalate to the governance owner.\n", "6. Do **not** look for a resync job. There isn't one.\n", "\n", "**Open mirroring \u2014 you own the change feed**\n", "1. Nothing in Fabric is at fault until your writer is ruled out. Check that your\n", " application is still landing change data in the OneLake zone.\n", "2. A missed delete or a malformed batch is a bug in your writer, not a mirroring\n", " failure \u2014 there is no vendor connector to blame.\n", "3. Then, and only then, check the Fabric side for merge errors.\n", "\n", "**All three:** if the incident involved data that should have been filtered,\n", "treat it as a security incident, not a data incident \u2014 Unity Catalog was never\n", "enforcing on this path.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sources\n", "\n", "All read 2026-09-18. Microsoft's revision dates are given where published, because\n", "that is what tells you whether a page has moved since it was read.\n", "\n", "- OneLake table, column, and row-level security (rev. 2026-08-19) \u2014\n", " \n", "- OneLake data security overview (rev. 2026-09-01) \u2014\n", " \n", "- Mirroring in Fabric \u2014 the three mechanisms (rev. 2026-08-28) \u2014\n", " \n", "- Mirrored catalog from Azure Databricks \u2014\n", " \n", "- Mirrored databases from Azure Databricks: **security** (rev. 2026-05-01) \u2014\n", " the 'permissions do not carry over' and connection-credential statements \u2014\n", " \n", "- Azure Databricks \u2014 row filters and column masks (rev. 2026-09-11) \u2014 the\n", " limitations list, including Unity REST API and path-based access \u2014\n", " \n", "- Azure Databricks \u2014 cross-engine ABAC, Beta (rev. 2026-09-11) \u2014\n", " \n", "- Microsoft Fabric for US Government GCC High customers, public preview\n", " (rev. 2026-09-03) \u2014 the sovereign REST endpoint used in the setup cell, the\n", " two available regions, and the preview limits that matter here: all mirroring\n", " sources except Mirrored Azure SQL Database are unavailable, shortcuts are\n", " limited to Azure Blob Storage and ADLS Gen2, and OneLake security has no Spark\n", " support \u2014\n", " \n", "\n", "**OneLake security.** RLS and CLS are documented as working controls with defined\n", "per-engine enforcement, and carried no preview label as of 2026-09-18. For a GA date\n", "on a compliance artifact, ask your account team.\n", "\n", "Verify anything time-sensitive against the source before you plan around it.\n", "This notebook is dated at the top for exactly that reason.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "microsoft": { "language": "python" } }, "nbformat": 4, "nbformat_minor": 5 }