{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.13.0"
    },
    "blog_metadata": {
      "topic": "The AI-Native Data Engineer: Why Fabric, dbt, Devcontainers, and Copilot Are Converging",
      "slug": "the-ai-native-data-engineer-why-fabric-dbt-devcontainers-and",
      "generated_by": "LinkedIn Post Generator + Azure OpenAI",
      "generated_at": "2026-08-10T19:15:48.912Z"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# The AI-Native Data Engineer: Why Fabric, dbt, Devcontainers, and Copilot Are Converging\n",
        "\n",
        "This notebook turns the blog post into a hands-on validation workflow. It demonstrates the core idea that platform, transformation discipline, reproducible environments, and AI assistance are most valuable when they operate as one governed delivery system.\n",
        "\n",
        "The examples below simulate a lightweight local workflow using Python, DuckDB, Parquet, and generated project files so you can validate the operating model without needing a live Fabric environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install -q pandas pyarrow duckdb pyyaml"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "import textwrap\n",
        "from pathlib import Path\n",
        "\n",
        "import duckdb\n",
        "import pandas as pd\n",
        "import yaml"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Workflow convergence map\n",
        "\n",
        "The blog argues that the real shift is not a tool bake-off, but workflow convergence. This cell captures the architecture as structured data so it can be inspected and reused programmatically."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "workflow_graph = {\n",
        "    \"nodes\": [\n",
        "        \"Source Systems\",\n",
        "        \"Microsoft Fabric OneLake\",\n",
        "        \"Lakehouse / Warehouse\",\n",
        "        \"dbt Models\",\n",
        "        \"Semantic Layer / Gold Tables\",\n",
        "        \"BI, ML, Copilot Experiences\",\n",
        "        \"Devcontainer\",\n",
        "        \"Reproducible Tooling\",\n",
        "        \"dbt + Python + PowerShell\",\n",
        "        \"Copilot\",\n",
        "        \"Code Generation\",\n",
        "        \"Test Scaffolding\",\n",
        "        \"Docs + Refactors\"\n",
        "    ],\n",
        "    \"edges\": [\n",
        "        [\"Source Systems\", \"Microsoft Fabric OneLake\"],\n",
        "        [\"Microsoft Fabric OneLake\", \"Lakehouse / Warehouse\"],\n",
        "        [\"Lakehouse / Warehouse\", \"dbt Models\"],\n",
        "        [\"dbt Models\", \"Semantic Layer / Gold Tables\"],\n",
        "        [\"Semantic Layer / Gold Tables\", \"BI, ML, Copilot Experiences\"],\n",
        "        [\"Devcontainer\", \"Reproducible Tooling\"],\n",
        "        [\"Reproducible Tooling\", \"dbt + Python + PowerShell\"],\n",
        "        [\"dbt + Python + PowerShell\", \"dbt Models\"],\n",
        "        [\"Copilot\", \"Code Generation\"],\n",
        "        [\"Copilot\", \"Test Scaffolding\"],\n",
        "        [\"Copilot\", \"Docs + Refactors\"],\n",
        "        [\"Code Generation\", \"dbt + Python + PowerShell\"],\n",
        "        [\"Test Scaffolding\", \"dbt Models\"],\n",
        "        [\"Docs + Refactors\", \"Reproducible Tooling\"]\n",
        "    ]\n",
        "}\n",
        "\n",
        "print(json.dumps(workflow_graph, indent=2))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Devcontainer as a reproducibility contract\n",
        "\n",
        "A key claim in the post is that environment consistency is a control surface, not just a convenience. This cell writes an example `devcontainer.json` file that standardizes Python, PowerShell, Git, and Copilot-related editor setup."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "devcontainer = {\n",
        "    \"name\": \"fabric-dbt-ai-native\",\n",
        "    \"image\": \"mcr.microsoft.com/devcontainers/python:3.11\",\n",
        "    \"features\": {\n",
        "        \"ghcr.io/devcontainers/features/powershell:1\": {},\n",
        "        \"ghcr.io/devcontainers/features/git:1\": {}\n",
        "    },\n",
        "    \"postCreateCommand\": \"pip install dbt-core dbt-duckdb pandas pyarrow\",\n",
        "    \"customizations\": {\n",
        "        \"vscode\": {\n",
        "            \"extensions\": [\n",
        "                \"ms-python.python\",\n",
        "                \"ms-vscode.powershell\",\n",
        "                \"GitHub.copilot\"\n",
        "            ]\n",
        "        }\n",
        "    }\n",
        "}\n",
        "\n",
        "path = Path('.devcontainer')\n",
        "path.mkdir(exist_ok=True)\n",
        "file_path = path / 'devcontainer.json'\n",
        "file_path.write_text(json.dumps(devcontainer, indent=2))\n",
        "print(file_path.read_text())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Bootstrap script for local tooling\n",
        "\n",
        "The original post includes a PowerShell bootstrap script. Since this notebook uses Python, the next cell generates an equivalent script file so the setup can still be reviewed and versioned as part of the workflow."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "bootstrap_ps1 = textwrap.dedent('''\n",
        "# Bootstrap local tooling so every engineer starts from the same baseline\n",
        "$ErrorActionPreference = \"Stop\"\n",
        "\n",
        "python --version\n",
        "pwsh --version\n",
        "\n",
        "python -m venv .venv\n",
        "if ($IsWindows) {\n",
        "    .\\\\.venv\\\\Scripts\\\\Activate.ps1\n",
        "} else {\n",
        "    . ./.venv/bin/Activate.ps1\n",
        "}\n",
        "\n",
        "python -m pip install --upgrade pip\n",
        "pip install dbt-core dbt-duckdb pandas pyarrow\n",
        "dbt --version\n",
        "''').strip()\n",
        "\n",
        "Path('bootstrap-local-tooling.ps1').write_text(bootstrap_ps1)\n",
        "print(Path('bootstrap-local-tooling.ps1').read_text())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Minimal dbt project definition\n",
        "\n",
        "The post emphasizes dbt-style rigor: explicit, reviewable, testable, versioned logic. This cell creates a minimal `dbt_project.yml` to show how transformation conventions become part of the delivery contract."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "dbt_project = {\n",
        "    'name': 'fabric_ai_native_demo',\n",
        "    'version': '1.0.0',\n",
        "    'config-version': 2,\n",
        "    'profile': 'fabric_ai_native_demo',\n",
        "    'model-paths': ['models'],\n",
        "    'models': {\n",
        "        'fabric_ai_native_demo': {\n",
        "            'bronze': {'+materialized': 'view'},\n",
        "            'silver': {'+materialized': 'table'},\n",
        "            'gold': {'+materialized': 'table'}\n",
        "        }\n",
        "    }\n",
        "}\n",
        "\n",
        "Path('dbt_project.yml').write_text(yaml.safe_dump(dbt_project, sort_keys=False))\n",
        "print(Path('dbt_project.yml').read_text())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Generate bronze data locally\n",
        "\n",
        "This is the first executable data step in the workflow. It creates a small Parquet dataset to simulate landed raw events so the team can validate delivery mechanics before connecting to upstream systems."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from pathlib import Path\n",
        "import pandas as pd\n",
        "\n",
        "bronze_df = pd.DataFrame(\n",
        "    [\n",
        "        {\"user_id\": 1, \"event_type\": \"view\", \"event_ts\": \"2026-08-10T09:00:00\"},\n",
        "        {\"user_id\": 1, \"event_type\": \"click\", \"event_ts\": \"2026-08-10T09:05:00\"},\n",
        "        {\"user_id\": 2, \"event_type\": \"view\", \"event_ts\": \"2026-08-10T10:00:00\"},\n",
        "    ]\n",
        ")\n",
        "\n",
        "out = Path('data')\n",
        "out.mkdir(exist_ok=True)\n",
        "bronze_path = out / 'bronze_events.parquet'\n",
        "bronze_df.to_parquet(bronze_path, index=False)\n",
        "print(bronze_df)\n",
        "print(f'Wrote {bronze_path.resolve()}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Define the transformation as versioned SQL\n",
        "\n",
        "The blog's dbt model turns raw events into a reusable silver layer. This cell writes the SQL model to disk so the transformation logic is explicit and reviewable, even though execution here is simulated with DuckDB."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "silver_sql = textwrap.dedent('''\n",
        "-- A focused dbt model that turns raw events into a reusable silver layer\n",
        "with source_events as (\n",
        "    select\n",
        "        user_id,\n",
        "        event_type,\n",
        "        cast(event_ts as timestamp) as event_ts\n",
        "    from {{ ref('bronze_events') }}\n",
        "),\n",
        "\n",
        "sessionized as (\n",
        "    select\n",
        "        user_id,\n",
        "        event_type,\n",
        "        date_trunc('day', event_ts) as event_date\n",
        "    from source_events\n",
        ")\n",
        "\n",
        "select\n",
        "    user_id,\n",
        "    event_date,\n",
        "    count(*) as event_count\n",
        "from sessionized\n",
        "group by 1, 2\n",
        "''').strip()\n",
        "\n",
        "model_dir = Path('models/silver')\n",
        "model_dir.mkdir(parents=True, exist_ok=True)\n",
        "model_file = model_dir / 'silver_events.sql'\n",
        "model_file.write_text(silver_sql)\n",
        "print(model_file.read_text())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Simulate dbt build and test with DuckDB\n",
        "\n",
        "The original post uses `dbt parse`, `dbt debug`, `dbt build`, `dbt test`, and `dbt docs generate`. In this notebook, the next cell simulates that delivery cadence by materializing a silver table from the bronze Parquet file and writing the result to a target folder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from pathlib import Path\n",
        "import duckdb\n",
        "\n",
        "con = duckdb.connect()\n",
        "con.execute(\"\"\"\n",
        "create or replace table bronze_events as\n",
        "select *\n",
        "from read_parquet('data/bronze_events.parquet')\n",
        "\"\"\")\n",
        "\n",
        "con.execute(\"\"\"\n",
        "create or replace table silver_events as\n",
        "with source_events as (\n",
        "    select\n",
        "        user_id,\n",
        "        event_type,\n",
        "        cast(event_ts as timestamp) as event_ts\n",
        "    from bronze_events\n",
        "),\n",
        "sessionized as (\n",
        "    select\n",
        "        user_id,\n",
        "        event_type,\n",
        "        date_trunc('day', event_ts) as event_date\n",
        "    from source_events\n",
        ")\n",
        "select\n",
        "    user_id,\n",
        "    event_date,\n",
        "    count(*) as event_count\n",
        "from sessionized\n",
        "group by 1, 2\n",
        "order by 1, 2\n",
        "\"\"\")\n",
        "\n",
        "silver_df = con.execute('select * from silver_events').df()\n",
        "\n",
        "target_dir = Path('target/run_results')\n",
        "target_dir.mkdir(parents=True, exist_ok=True)\n",
        "target_path = target_dir / 'silver_events.parquet'\n",
        "con.execute(f\"copy silver_events to '{target_path.as_posix()}' (format parquet)\")\n",
        "\n",
        "print('Simulated dbt steps: parse -> debug -> build -> test -> docs generate')\n",
        "print(silver_df)\n",
        "print(f'Wrote {target_path.resolve()}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Post-build quality validation\n",
        "\n",
        "The post recommends validating outputs the same way a CI job would. This cell checks the generated target Parquet files and asserts that no negative aggregates exist."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import duckdb\n",
        "\n",
        "query = \"\"\"\n",
        "select user_id, event_date, event_count\n",
        "from read_parquet('target/**/*.parquet', union_by_name=true)\n",
        "where event_count < 0\n",
        "\"\"\"\n",
        "\n",
        "result = duckdb.connect().execute(query).fetchall()\n",
        "assert result == [], f'Found invalid aggregates: {result}'\n",
        "print('Quality check passed: no negative event counts.')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Sequence of the AI-native workflow\n",
        "\n",
        "The blog also describes the workflow as a sequence: engineer opens a standardized environment, uses AI assistance for scaffolding, runs dbt, and materializes curated outputs. This cell represents that sequence in structured form for inspection."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "sequence = [\n",
        "    {\"actor\": \"Engineer\", \"action\": \"Open repo in container\"},\n",
        "    {\"actor\": \"Devcontainer\", \"action\": \"Provide standardized Python + PowerShell toolchain\"},\n",
        "    {\"actor\": \"Engineer\", \"action\": \"Ask Copilot for model/test/doc scaffolding\"},\n",
        "    {\"actor\": \"Copilot\", \"action\": \"Suggest SQL, YAML, and scripts\"},\n",
        "    {\"actor\": \"Engineer\", \"action\": \"Run dbt build and test\"},\n",
        "    {\"actor\": \"dbt\", \"action\": \"Materialize curated tables\"},\n",
        "    {\"actor\": \"Fabric\", \"action\": \"Serve analytics-ready data\"}\n",
        "]\n",
        "\n",
        "for i, step in enumerate(sequence, start=1):\n",
        "    print(f\"{i}. {step['actor']}: {step['action']}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create a reusable Copilot prompt artifact\n",
        "\n",
        "The post recommends keeping practical prompts in the repo so AI assistance stays inside guardrails. This cell writes a prompt file that could be used to generate a gold-layer dbt model and its associated tests."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "copilot_prompt = textwrap.dedent('''\n",
        "Create a dbt model named gold_daily_active_users.\n",
        "Inputs:\n",
        "- silver_events(user_id, event_date, event_count)\n",
        "\n",
        "Requirements:\n",
        "- one row per event_date\n",
        "- count distinct active users where event_count > 0\n",
        "- add a schema.yml test for not_null and unique on event_date\n",
        "- include concise model documentation\n",
        "''').strip()\n",
        "\n",
        "prompt_path = Path('copilot-prompt.txt')\n",
        "prompt_path.write_text(copilot_prompt)\n",
        "print(prompt_path.read_text())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Generate a gold-layer artifact from the prompt requirements\n",
        "\n",
        "To make the prompt concrete, this cell creates a gold-layer SQL model and a matching schema test file. This demonstrates how AI assistance should target stable project conventions rather than bypass them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "gold_sql = textwrap.dedent('''\n",
        "with filtered as (\n",
        "    select *\n",
        "    from silver_events\n",
        "    where event_count > 0\n",
        ")\n",
        "select\n",
        "    event_date,\n",
        "    count(distinct user_id) as daily_active_users\n",
        "from filtered\n",
        "group by 1\n",
        "order by 1\n",
        "''').strip()\n",
        "\n",
        "gold_schema = {\n",
        "    'version': 2,\n",
        "    'models': [\n",
        "        {\n",
        "            'name': 'gold_daily_active_users',\n",
        "            'description': 'Daily active users derived from silver_events where event_count > 0.',\n",
        "            'columns': [\n",
        "                {\n",
        "                    'name': 'event_date',\n",
        "                    'description': 'Calendar date for the activity metric.',\n",
        "                    'tests': ['not_null', 'unique']\n",
        "                },\n",
        "                {\n",
        "                    'name': 'daily_active_users',\n",
        "                    'description': 'Distinct count of active users for the date.'\n",
        "                }\n",
        "            ]\n",
        "        }\n",
        "    ]\n",
        "}\n",
        "\n",
        "gold_dir = Path('models/gold')\n",
        "gold_dir.mkdir(parents=True, exist_ok=True)\n",
        "(gold_dir / 'gold_daily_active_users.sql').write_text(gold_sql)\n",
        "(gold_dir / 'schema.yml').write_text(yaml.safe_dump(gold_schema, sort_keys=False))\n",
        "\n",
        "print((gold_dir / 'gold_daily_active_users.sql').read_text())\n",
        "print('\\n---\\n')\n",
        "print((gold_dir / 'schema.yml').read_text())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Materialize and validate the gold layer\n",
        "\n",
        "This final executable step extends the tiny working loop from bronze to silver to gold. It shows how governed, testable outputs become AI-consumable organizational context."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "con = duckdb.connect()\n",
        "con.execute(\"\"\"\n",
        "create or replace table silver_events as\n",
        "select *\n",
        "from read_parquet('target/run_results/silver_events.parquet')\n",
        "\"\"\")\n",
        "\n",
        "con.execute(\"\"\"\n",
        "create or replace table gold_daily_active_users as\n",
        "with filtered as (\n",
        "    select *\n",
        "    from silver_events\n",
        "    where event_count > 0\n",
        ")\n",
        "select\n",
        "    event_date,\n",
        "    count(distinct user_id) as daily_active_users\n",
        "from filtered\n",
        "group by 1\n",
        "order by 1\n",
        "\"\"\")\n",
        "\n",
        "gold_df = con.execute('select * from gold_daily_active_users').df()\n",
        "print(gold_df)\n",
        "\n",
        "assert gold_df['event_date'].notna().all(), 'event_date contains nulls'\n",
        "assert gold_df['event_date'].is_unique, 'event_date is not unique'\n",
        "assert (gold_df['daily_active_users'] >= 0).all(), 'daily_active_users contains negative values'\n",
        "print('Gold-layer validation passed.')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Next Steps\n",
        "\n",
        "This notebook validated the blog's core thesis with a small local loop: declare the environment, generate raw data, define explicit transformations, materialize outputs, and validate them with tests. The important pattern is not any single tool, but the governed workflow connecting platform, transformation discipline, reproducibility, and AI assistance.\n",
        "\n",
        "Next steps:\n",
        "1. Replace the local Parquet bronze layer with a real landing zone.\n",
        "2. Swap the DuckDB simulation for an actual dbt project and adapter.\n",
        "3. Add CI checks for model tests, docs generation, and environment drift.\n",
        "4. Define semantic ownership for each gold-layer domain.\n",
        "5. Establish explicit AI guardrails for what can be generated, reviewed, and deployed."
      ]
    }
  ]
}