{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "bicep", "language": "bicep", "name": "bicep" }, "language_info": { "name": "bicep", "version": "1.0.0" }, "blog_metadata": { "topic": "Secure Native Access to AKS Private Clusters with Azure Bastion", "slug": "secure-native-access-to-aks-private-clusters-with-azure-bast", "generated_by": "LinkedIn Post Generator + Azure OpenAI", "generated_at": "2026-07-10T13:32:39.180Z" } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Secure Native Access to AKS Private Clusters with Azure Bastion\n", "\n", "This notebook turns the blog post into a hands-on validation flow for private AKS administration through Azure Bastion. The focus is the production-ready pattern of Bastion plus a hardened jump VM, with emphasis on validating network path, private DNS, identity, and Kubernetes authorization before blaming kubectl." ] }, { "cell_type": "code", "metadata": {}, "source": [ "%pip install -q pyyaml jmespath" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import os\n", "import json\n", "import textwrap\n", "import subprocess\n", "from pathlib import Path\n", "\n", "try:\n", " import yaml\n", "except ImportError:\n", " yaml = None" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Access pattern overview\n", "\n", "This diagram captures the intended operator path: workstation to Azure Bastion, then to a jump VM inside the AKS network path, with private DNS resolving the private API server name. The key validation point is that kubectl and Azure CLI run from the private side, not directly from the operator laptop." ] }, { "cell_type": "code", "metadata": {}, "source": [ "from IPython.display import Markdown, display\n", "\n", "diagram = r'''\n", "```mermaid\n", "flowchart TD\n", " U[Operator Laptop] --> B[Azure Bastion]\n", " B --> J[Jump VM in AKS VNet]\n", " J --> D[Private DNS Zone]\n", " J --> A[AKS Private API Server]\n", " J --> K[kubectl / Azure CLI]\n", " K --> A\n", " D -. resolves .-> A\n", " U -. no direct public API access .-> A\n", "```\n", "'''\n", "display(Markdown(diagram))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Required variables\n", "\n", "Set these values before deploying or validating resources:\n", "\n", "- `AZURE_SUBSCRIPTION_ID`\n", "- `AZURE_LOCATION`\n", "- `AZURE_RESOURCE_GROUP`\n", "- `AKS_NAME`\n", "- `VNET_NAME`\n", "- `AKS_SUBNET_ID` (for AKS deployment)\n", "- `JUMP_VM_NAME` (for Bastion session target)\n", "\n", "If you are running Azure CLI commands from this notebook, make sure you are already authenticated with `az login` and have selected the correct subscription." ] }, { "cell_type": "code", "metadata": {}, "source": [ "subscription_id = os.getenv('AZURE_SUBSCRIPTION_ID', '')\n", "location = os.getenv('AZURE_LOCATION', 'eastus')\n", "resource_group = os.getenv('AZURE_RESOURCE_GROUP', 'rg-aks-private')\n", "aks_name = os.getenv('AKS_NAME', 'aks-private-demo')\n", "vnet_name = os.getenv('VNET_NAME', 'aks-secure-vnet')\n", "aks_subnet_id = os.getenv('AKS_SUBNET_ID', '/subscriptions//resourceGroups//providers/Microsoft.Network/virtualNetworks/aks-secure-vnet/subnets/aks-subnet')\n", "jump_vm_name = os.getenv('JUMP_VM_NAME', 'jumpvm01')\n", "\n", "print({\n", " 'subscription_id': subscription_id,\n", " 'location': location,\n", " 'resource_group': resource_group,\n", " 'aks_name': aks_name,\n", " 'vnet_name': vnet_name,\n", " 'aks_subnet_id': aks_subnet_id,\n", " 'jump_vm_name': jump_vm_name,\n", "})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bicep: Bastion host and jump subnet\n", "\n", "This Bicep template creates the network skeleton for the admin path: a VNet, the required `AzureBastionSubnet`, a separate jumpbox subnet, a Standard public IP for Bastion, and the Bastion host itself. This is intentionally minimal so you can validate the pattern before layering on production hardening." ] }, { "cell_type": "code", "metadata": {}, "source": [ "bicep_bastion = r'''\n", "param location string = resourceGroup().location\n", "param vnetName string = 'aks-secure-vnet'\n", "param bastionPipName string = 'bastion-pip'\n", "param bastionName string = 'aks-bastion'\n", "\n", "resource vnet 'Microsoft.Network/virtualNetworks@2023-09-01' = {\n", " name: vnetName\n", " location: location\n", " properties: {\n", " addressSpace: { addressPrefixes: ['10.20.0.0/16'] }\n", " subnets: [\n", " { name: 'AzureBastionSubnet'; properties: { addressPrefix: '10.20.0.0/26' } }\n", " { name: 'jumpbox-subnet'; properties: { addressPrefix: '10.20.1.0/24' } }\n", " ]\n", " }\n", "}\n", "\n", "resource pip 'Microsoft.Network/publicIPAddresses@2023-09-01' = {\n", " name: bastionPipName\n", " location: location\n", " sku: { name: 'Standard' }\n", " properties: { publicIPAllocationMethod: 'Static' }\n", "}\n", "\n", "resource bastion 'Microsoft.Network/bastionHosts@2023-09-01' = {\n", " name: bastionName\n", " location: location\n", " properties: {\n", " ipConfigurations: [{\n", " name: 'bastion-ipcfg'\n", " properties: {\n", " subnet: { id: '${vnet.id}/subnets/AzureBastionSubnet' }\n", " publicIPAddress: { id: pip.id }\n", " }\n", " }]\n", " }\n", "}\n", "'''\n", "\n", "Path('01-bastion-network.bicep').write_text(bicep_bastion)\n", "print(Path('01-bastion-network.bicep').read_text())" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bicep: Private AKS cluster with Azure RBAC\n", "\n", "This Bicep template deploys a private AKS cluster into a subnet and enables Azure RBAC for Kubernetes authorization. The important validation point is `enablePrivateCluster: true`, because the rest of the admin path only matters if the API server is actually private." ] }, { "cell_type": "code", "metadata": {}, "source": [ "bicep_aks = r'''\n", "param location string = resourceGroup().location\n", "param aksName string = 'aks-private-demo'\n", "param dnsPrefix string = 'aksprivdemo'\n", "param subnetId string\n", "param kubernetesVersion string = '1.29.4'\n", "\n", "resource aks 'Microsoft.ContainerService/managedClusters@2024-02-01' = {\n", " name: aksName\n", " location: location\n", " identity: { type: 'SystemAssigned' }\n", " properties: {\n", " dnsPrefix: dnsPrefix\n", " kubernetesVersion: kubernetesVersion\n", " apiServerAccessProfile: { enablePrivateCluster: true }\n", " aadProfile: { managed: true, enableAzureRBAC: true }\n", " agentPoolProfiles: [{\n", " name: 'system'\n", " mode: 'System'\n", " count: 1\n", " vmSize: 'Standard_DS2_v2'\n", " osType: 'Linux'\n", " type: 'VirtualMachineScaleSets'\n", " vnetSubnetID: subnetId\n", " }]\n", " networkProfile: { networkPlugin: 'azure', networkPolicy: 'azure' }\n", " }\n", "}\n", "'''\n", "\n", "Path('02-private-aks.bicep').write_text(bicep_aks)\n", "print(Path('02-private-aks.bicep').read_text())" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optional deployment commands\n", "\n", "Use these Azure CLI commands if you want to deploy the Bicep templates from the notebook environment. They assume the resource group already exists and that the AKS subnet ID is valid." ] }, { "cell_type": "code", "metadata": {}, "source": [ "deploy_bastion_cmd = f\"\"\"\n", "az account set --subscription {subscription_id}\n", "az deployment group create \\\\\n", " --resource-group {resource_group} \\\\\n", " --template-file 01-bastion-network.bicep \\\\\n", " --parameters location={location} vnetName={vnet_name}\n", "\"\"\".strip()\n", "\n", "deploy_aks_cmd = f\"\"\"\n", "az account set --subscription {subscription_id}\n", "az deployment group create \\\\\n", " --resource-group {resource_group} \\\\\n", " --template-file 02-private-aks.bicep \\\\\n", " --parameters location={location} aksName={aks_name} subnetId='{aks_subnet_id}'\n", "\"\"\".strip()\n", "\n", "print(deploy_bastion_cmd)\n", "print()\n", "print(deploy_aks_cmd)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Session sequence overview\n", "\n", "This sequence diagram shows the validation order that matters operationally: open a Bastion session, land on the jump VM, resolve the private FQDN from that machine, then run `az aks get-credentials` and `kubectl get nodes`. DNS resolution from the jump VM is the critical checkpoint." ] }, { "cell_type": "code", "metadata": {}, "source": [ "sequence = r'''\n", "```mermaid\n", "sequenceDiagram\n", " participant O as Operator\n", " participant B as Azure Bastion\n", " participant J as Jump VM\n", " participant D as Private DNS\n", " participant A as AKS Private API\n", "\n", " O->>B: Open Bastion session\n", " B->>J: Connect over Azure backbone\n", " O->>J: Run az aks get-credentials\n", " J->>D: Resolve privateFqdn\n", " D-->>J: Private IP for API server\n", " J->>A: kubectl get nodes\n", " A-->>J: Authorized cluster response\n", "```\n", "'''\n", "display(Markdown(sequence))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Azure CLI validation: confirm the cluster is private\n", "\n", "Run this from a machine that has Azure CLI access to your subscription. It verifies the cluster name, whether private cluster mode is enabled, the private FQDN, and the node resource group, then attempts to inspect the private DNS records associated with the API server name." ] }, { "cell_type": "code", "metadata": {}, "source": [ "azure_cli_validation = r'''\n", "RG=\"rg-aks-private\"\n", "AKS=\"aks-private-demo\"\n", "\n", "az aks show -g \"$RG\" -n \"$AKS\" \\\n", " --query \"{name:name,private:apiServerAccessProfile.enablePrivateCluster,privateFqdn:privateFqdn,nodeRG:nodeResourceGroup}\" \\\n", " -o yaml\n", "\n", "API_FQDN=$(az aks show -g \"$RG\" -n \"$AKS\" --query privateFqdn -o tsv)\n", "echo \"Private API server FQDN: $API_FQDN\"\n", "\n", "az network private-dns record-set a list \\\n", " -g \"$RG\" \\\n", " -z \"$(echo \"$API_FQDN\" | cut -d. -f2-)\" \\\n", " -o table\n", "'''\n", "print(azure_cli_validation)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Design checks before touching kubectl\n", "\n", "Before opening a Bastion session, validate four things:\n", "\n", "1. Bastion lands in a network path that can reach the AKS private endpoint.\n", "2. The AKS private FQDN resolves from the jump VM side.\n", "3. Your Azure identity has permission to retrieve credentials.\n", "4. `kubectl` and Azure CLI are available on the jump VM.\n", "\n", "These checks help separate network, DNS, and authorization failures early." ] }, { "cell_type": "code", "metadata": {}, "source": [ "design_checks = {\n", " 'vnet_reachability': 'Bastion and jump VM must be in same VNet or a correctly peered path to the AKS private endpoint.',\n", " 'private_dns': 'AKS private FQDN must resolve from the jump VM.',\n", " 'identity_rbac': 'Azure permissions must allow az aks get-credentials and Kubernetes actions must be authorized.',\n", " 'tooling_location': 'Run Azure CLI and kubectl from the private side, not the operator laptop.'\n", "}\n", "print(json.dumps(design_checks, indent=2))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## PowerShell validation from the jump VM\n", "\n", "This sequence is meant to run after you connect to the jump VM through Bastion. It validates private cluster settings, retrieves kubeconfig, selects the context, checks control plane reachability, lists nodes, and tests authorization with `kubectl auth can-i`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "powershell_validation = r'''\n", "$rg = \"rg-aks-private\"\n", "$aks = \"aks-private-demo\"\n", "\n", "$cluster = az aks show -g $rg -n $aks | ConvertFrom-Json\n", "\"Private cluster: $($cluster.apiServerAccessProfile.enablePrivateCluster)\"\n", "\"Private FQDN: $($cluster.privateFqdn)\"\n", "\n", "az aks get-credentials -g $rg -n $aks --overwrite-existing | Out-Null\n", "kubectl config use-context $cluster.name | Out-Null\n", "\n", "kubectl cluster-info\n", "kubectl get nodes -o wide\n", "kubectl auth can-i get pods --all-namespaces\n", "'''\n", "print(powershell_validation)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Python diagnostic: separate DNS from RBAC problems\n", "\n", "This diagnostic is useful on the jump VM after kubeconfig is present. It reads the current context, extracts the API server hostname, attempts DNS resolution, and then probes the API with `kubectl get ns` to help distinguish DNS failures from authorization failures." ] }, { "cell_type": "code", "metadata": {}, "source": [ "import socket\n", "import subprocess\n", "import sys\n", "\n", "def sh(*args):\n", " return subprocess.run(args, capture_output=True, text=True)\n", "\n", "ctx = sh('kubectl', 'config', 'current-context')\n", "view = sh('kubectl', 'config', 'view', '--minify', '-o', 'jsonpath={.clusters[0].cluster.server}')\n", "server = view.stdout.strip().replace('https://', '').split(':')[0]\n", "\n", "print(f\"context={ctx.stdout.strip()}\")\n", "print(f\"api_server={server}\")\n", "\n", "try:\n", " print(f\"resolved_ip={socket.gethostbyname(server)}\")\n", "except socket.gaierror:\n", " print('diagnosis=DNS resolution failed; verify Bastion/jump VM VNet DNS and private zone linkage')\n", " raise SystemExit(2)\n", "\n", "probe = sh('kubectl', 'get', 'ns')\n", "if probe.returncode == 0:\n", " print('diagnosis=API reachable and authorized')\n", "elif 'Unauthorized' in probe.stderr or 'Forbidden' in probe.stderr:\n", " print('diagnosis=API reachable; likely Azure RBAC/Kubernetes RBAC issue')\n", "else:\n", " print('diagnosis=API name resolves, but connectivity or kubeconfig may be broken')\n", " if probe.stderr:\n", " print(probe.stderr)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Minimal workload manifest for private-path validation\n", "\n", "Once the control plane is reachable, apply a tiny workload to prove the session is actually usable. This manifest creates a namespace and a lightweight pod that can be used for simple DNS and service checks." ] }, { "cell_type": "code", "metadata": {}, "source": [ "connectivity_yaml = r'''\n", "apiVersion: v1\n", "kind: Namespace\n", "metadata:\n", " name: connectivity-check\n", "---\n", "apiVersion: v1\n", "kind: Pod\n", "metadata:\n", " name: dnsutils\n", " namespace: connectivity-check\n", "spec:\n", " containers:\n", " - name: dnsutils\n", " image: registry.k8s.io/e2e-test-images/agnhost:2.39\n", " args: [\"pause\"]\n", " restartPolicy: Always\n", "'''\n", "\n", "Path('connectivity-check.yaml').write_text(connectivity_yaml)\n", "print(Path('connectivity-check.yaml').read_text())" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## kubectl validation commands\n", "\n", "These commands apply the test pod and verify that the Bastion-backed admin path is functional enough for real operations. If the pod schedules and in-cluster DNS works, you have moved beyond basic API reachability into practical operability." ] }, { "cell_type": "code", "metadata": {}, "source": [ "kubectl_validation = r'''\n", "kubectl apply -f connectivity-check.yaml\n", "kubectl get pods -n connectivity-check\n", "kubectl exec -n connectivity-check dnsutils -- nslookup kubernetes.default.svc.cluster.local\n", "kubectl get svc kubernetes -n default -o wide\n", "kubectl logs -n connectivity-check dnsutils\n", "'''\n", "print(kubectl_validation)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Failure mode checklist\n", "\n", "Use this quick map when validation fails:\n", "\n", "- **Private DNS wrong**: private FQDN exists but does not resolve from the jump VM.\n", "- **NSG or routing blocked**: DNS resolves, but connections hang or time out.\n", "- **Credentials or authorization missing**: `az aks get-credentials` fails, or kubectl returns `Unauthorized` or `Forbidden`.\n", "- **Admin path confused with workload exposure**: Bastion solves operator access, not application ingress design.\n", "\n", "Treat these as separate problem classes so you do not waste time debugging the wrong layer." ] }, { "cell_type": "code", "metadata": {}, "source": [ "failure_modes = [\n", " {\n", " 'failure_mode': 'Private DNS is wrong',\n", " 'symptoms': ['privateFqdn exists', 'jump VM cannot resolve it', 'kubectl hangs or name resolution fails'],\n", " 'fix': ['verify private DNS zone exists', 'verify VNet links', 'test resolution from jump VM']\n", " },\n", " {\n", " 'failure_mode': 'NSG or routing blocks the path',\n", " 'symptoms': ['DNS works', 'hostname resolves', 'connection times out or hangs'],\n", " 'fix': ['inspect NSGs', 'inspect UDRs', 'validate peering and forwarding assumptions']\n", " },\n", " {\n", " 'failure_mode': 'Credentials or authorization are missing',\n", " 'symptoms': ['az aks get-credentials fails', 'kubectl returns Unauthorized or Forbidden'],\n", " 'fix': ['verify Azure role assignments', 'verify Kubernetes or Azure RBAC bindings', 'test with kubectl auth can-i']\n", " },\n", " {\n", " 'failure_mode': 'Teams confuse workload exposure with admin access',\n", " 'symptoms': ['Bastion path works but app ingress design is still unresolved'],\n", " 'fix': ['treat admin access and workload exposure as separate architectures']\n", " }\n", "]\n", "print(json.dumps(failure_modes, indent=2))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "The secure operating model for private AKS is not just `enablePrivateCluster: true`; it is a complete admin path that stays private end to end. Azure Bastion provides the managed ingress point, the jump VM provides a controlled tooling environment, and private DNS plus RBAC determine whether the path actually works in day-2 operations.\n", "\n", "## Next Steps\n", "\n", "- Add a hardened jump VM with no public IP to the jumpbox subnet.\n", "- Validate private DNS zone linkage from the jump VM before onboarding operators.\n", "- Standardize `az aks get-credentials`, `kubectl cluster-info`, and `kubectl auth can-i` as readiness checks.\n", "- Define a separate, audited break-glass process for node access.\n", "- Productionize the Bicep with NSGs, monitoring, policy, and role assignments." ] } ] }