{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.13.0"
    },
    "blog_metadata": {
      "topic": "What OpenAI’s Realtime Voice System Signals for Microsoft Copilot Experiences",
      "slug": "what-openai-s-realtime-voice-system-signals-for-microsoft-co",
      "generated_by": "LinkedIn Post Generator + Azure OpenAI",
      "generated_at": "2026-08-04T11:22:56.361Z"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# What OpenAI’s Realtime Voice System Signals for Microsoft Copilot Experiences\n",
        "\n",
        "This notebook turns the blog post into a hands-on validation workbook. It focuses on the operational claims behind realtime voice for Copilot experiences: latency, interruption handling, context grounding, and enterprise guardrails.\n",
        "\n",
        "Rather than treating voice as a demo feature, the exercises below help you test whether a voice-capable Copilot design is fast, governable, and useful for real work."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install -q pandas numpy matplotlib"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "import time\n",
        "from dataclasses import dataclass, asdict\n",
        "from typing import List, Dict, Any\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Interaction model overview\n",
        "\n",
        "The blog argues that voice changes the interaction standard because users expect immediate response, clean interruption handling, continuity across turns, and a reliable handoff into work. This cell captures the core architecture as structured data so you can inspect the flow programmatically."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "voice_flow = {\n",
        "    \"nodes\": [\n",
        "        \"User speaks in Copilot\",\n",
        "        \"Copilot client\",\n",
        "        \"Realtime session API\",\n",
        "        \"Streaming STT + turn detection\",\n",
        "        \"Reasoning + tool selection\",\n",
        "        \"Microsoft Graph / enterprise tools\",\n",
        "        \"TTS audio stream\"\n",
        "    ],\n",
        "    \"edges\": [\n",
        "        (\"User speaks in Copilot\", \"Copilot client\"),\n",
        "        (\"Copilot client\", \"Realtime session API\"),\n",
        "        (\"Realtime session API\", \"Streaming STT + turn detection\"),\n",
        "        (\"Streaming STT + turn detection\", \"Reasoning + tool selection\"),\n",
        "        (\"Reasoning + tool selection\", \"Microsoft Graph / enterprise tools\"),\n",
        "        (\"Microsoft Graph / enterprise tools\", \"Reasoning + tool selection\"),\n",
        "        (\"Reasoning + tool selection\", \"TTS audio stream\"),\n",
        "        (\"TTS audio stream\", \"Copilot client\"),\n",
        "        (\"Copilot client\", \"User speaks in Copilot\")\n",
        "    ]\n",
        "}\n",
        "\n",
        "print(\"Voice interaction flow nodes:\")\n",
        "for i, node in enumerate(voice_flow[\"nodes\"], start=1):\n",
        "    print(f\"{i}. {node}\")\n",
        "\n",
        "print(\"\\nFlow edges:\")\n",
        "for src, dst in voice_flow[\"edges\"]:\n",
        "    print(f\"{src} -> {dst}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Minimal realtime session payload\n",
        "\n",
        "This example converts the blog’s session design into Python. It creates a simple Copilot-style realtime session payload with model, voice, instructions, turn detection, and a tool definition."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "from dataclasses import dataclass, asdict\n",
        "\n",
        "@dataclass\n",
        "class RealtimeSession:\n",
        "    model: str\n",
        "    voice: str\n",
        "    instructions: str\n",
        "    turn_detection: dict\n",
        "    tools: list\n",
        "\n",
        "session = RealtimeSession(\n",
        "    model=\"gpt-realtime\",\n",
        "    voice=\"alloy\",\n",
        "    instructions=\"Act as Microsoft Copilot: concise, grounded, and enterprise-safe.\",\n",
        "    turn_detection={\"type\": \"server_vad\", \"silence_ms\": 500},\n",
        "    tools=[{\"type\": \"function\", \"name\": \"search_graph\"}],\n",
        ")\n",
        "\n",
        "print(json.dumps(asdict(session), indent=2))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Required environment variables for live API use\n",
        "\n",
        "The original blog also showed a request-building example that would normally use secrets. If you adapt this notebook to call a live realtime API, define variables such as:\n",
        "\n",
        "- `OPENAI_API_KEY`\n",
        "- `OPENAI_BASE_URL` (optional, if using a custom endpoint)\n",
        "- `OPENAI_REALTIME_MODEL` (optional override)\n",
        "\n",
        "This notebook keeps the request construction local and does not make external calls."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Build a realtime request payload in Python\n",
        "\n",
        "The blog included a PowerShell example for headers and JSON body construction. This Python version produces the same kind of request object so you can validate structure without sending it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import os\n",
        "import json\n",
        "\n",
        "api_key = os.getenv(\"OPENAI_API_KEY\", \"YOUR_API_KEY\")\n",
        "headers = {\n",
        "    \"Authorization\": f\"Bearer {api_key}\",\n",
        "    \"Content-Type\": \"application/json\"\n",
        "}\n",
        "\n",
        "body = {\n",
        "    \"model\": \"gpt-realtime\",\n",
        "    \"voice\": \"alloy\",\n",
        "    \"instructions\": \"You are Copilot for work. Be brief and cite enterprise sources when possible.\",\n",
        "    \"turn_detection\": {\n",
        "        \"type\": \"server_vad\",\n",
        "        \"silence_ms\": 500\n",
        "    }\n",
        "}\n",
        "\n",
        "print(\"Headers:\")\n",
        "print(json.dumps(headers, indent=2))\n",
        "print(\"\\nBody:\")\n",
        "print(json.dumps(body, indent=2))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Sequence of a grounded voice interaction\n",
        "\n",
        "This cell turns the sequence diagram into a simple event list. It helps validate the order of operations for a voice request that streams audio, retrieves enterprise context, and returns synthesized speech."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "sequence = [\n",
        "    (\"User\", \"Copilot\", \"Speak request\"),\n",
        "    (\"Copilot\", \"Realtime\", \"Stream audio chunks\"),\n",
        "    (\"Realtime\", \"Copilot\", \"Partial transcript\"),\n",
        "    (\"Realtime\", \"Graph\", \"Tool call for calendar/email/context\"),\n",
        "    (\"Graph\", \"Realtime\", \"Grounded enterprise data\"),\n",
        "    (\"Realtime\", \"Copilot\", \"Stream synthesized voice response\"),\n",
        "    (\"Copilot\", \"User\", \"Low-latency spoken answer\")\n",
        "]\n",
        "\n",
        "for actor_a, actor_b, action in sequence:\n",
        "    print(f\"{actor_a} -> {actor_b}: {action}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Simulate barge-in handling\n",
        "\n",
        "One of the blog’s strongest points is that interruption is the user’s control plane. This example simulates barge-in by stopping playback when the user starts speaking again."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import time\n",
        "\n",
        "audio_frames = [\"frame1\", \"frame2\", \"frame3\", \"frame4\"]\n",
        "user_started_speaking_at = 2\n",
        "\n",
        "for i, frame in enumerate(audio_frames, start=1):\n",
        "    if i == user_started_speaking_at:\n",
        "        print(\"Barge-in detected: stop TTS playback and switch to listening.\")\n",
        "        break\n",
        "    print(f\"Playing {frame}\")\n",
        "    time.sleep(0.1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Ground a response with bounded enterprise context\n",
        "\n",
        "The blog emphasizes that useful Copilot voice experiences depend on bounded context, not unlimited retrieval. This example creates a Graph-style context object and injects it into a prompt."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "graph_context = {\n",
        "    \"user\": \"alex@contoso.com\",\n",
        "    \"nextMeeting\": \"Quarterly planning at 2:00 PM\",\n",
        "    \"unreadEmails\": 14,\n",
        "    \"topDocument\": \"FY26-Strategy.docx\"\n",
        "}\n",
        "\n",
        "prompt = f\"\"\"You are Microsoft Copilot.\n",
        "Use this enterprise context to answer the user:\n",
        "{json.dumps(graph_context, separators=(',', ':'))}\n",
        "\"\"\"\n",
        "\n",
        "print(prompt)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Enforce voice-specific guardrails\n",
        "\n",
        "The blog notes that some information should never be spoken aloud, even if the user technically has access. This example implements a simple policy layer that blocks sensitive credential-like content from spoken output."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "def enforce_policy(text: str) -> str:\n",
        "    blocked = [\"password\", \"secret key\", \"token\"]\n",
        "    if any(term in text.lower() for term in blocked):\n",
        "        return \"I can’t read sensitive credentials aloud. I can help you rotate or store them securely.\"\n",
        "    return text\n",
        "\n",
        "samples = [\n",
        "    \"Your next meeting is at 2 PM.\",\n",
        "    \"The password for the admin account is ...\",\n",
        "]\n",
        "\n",
        "for s in samples:\n",
        "    print(enforce_policy(s))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Map the strategic value chain of realtime voice\n",
        "\n",
        "This cell captures the blog’s causal chain: lower latency, natural interruptions, and continuous context lead to more conversational and more useful enterprise copilots."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "value_chain = {\n",
        "    \"Realtime voice UX\": [\n",
        "        \"Lower latency\",\n",
        "        \"Natural interruptions\",\n",
        "        \"Continuous context\"\n",
        "    ],\n",
        "    \"Lower latency\": [\"Feels conversational in Copilot\"],\n",
        "    \"Natural interruptions\": [\"Supports multitasking and corrections\"],\n",
        "    \"Continuous context\": [\"Better grounding with Graph and plugins\"],\n",
        "    \"Better grounding with Graph and plugins\": [\"More useful enterprise copilots\"]\n",
        "}\n",
        "\n",
        "for source, targets in value_chain.items():\n",
        "    for target in targets:\n",
        "        print(f\"{source} -> {target}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Measure latency checkpoints\n",
        "\n",
        "The blog argues that end-to-first-audio latency is the user’s truth metric. This example logs the key checkpoints and computes total latency to first audio."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "checkpoints = {\n",
        "    \"CaptureStartMs\": 0,\n",
        "    \"FirstTranscriptMs\": 180,\n",
        "    \"ToolCallStartMs\": 320,\n",
        "    \"ToolCallEndMs\": 640,\n",
        "    \"FirstAudioOutMs\": 780,\n",
        "}\n",
        "\n",
        "for key, value in checkpoints.items():\n",
        "    print(f\"{key}: {value} ms\")\n",
        "\n",
        "total = checkpoints[\"FirstAudioOutMs\"] - checkpoints[\"CaptureStartMs\"]\n",
        "print(f\"End-to-first-audio latency: {total} ms\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Analyze latency budget visually\n",
        "\n",
        "To make the latency argument more concrete, this cell converts the checkpoints into a table and chart. It helps you see where time is being spent in the interaction path."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "latency_df = pd.DataFrame([\n",
        "    {\"checkpoint\": k, \"ms\": v} for k, v in checkpoints.items()\n",
        "])\n",
        "latency_df[\"delta_from_previous_ms\"] = latency_df[\"ms\"].diff().fillna(latency_df[\"ms\"])\n",
        "\n",
        "print(latency_df)\n",
        "\n",
        "plt.figure(figsize=(8, 4))\n",
        "plt.bar(latency_df[\"checkpoint\"], latency_df[\"ms\"], color=\"steelblue\")\n",
        "plt.xticks(rotation=30, ha=\"right\")\n",
        "plt.ylabel(\"Milliseconds\")\n",
        "plt.title(\"Realtime Voice Latency Checkpoints\")\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Score candidate Copilot voice scenarios\n",
        "\n",
        "The blog proposes evaluating scenarios across seven dimensions: latency sensitivity, interruption frequency, continuity requirements, data sensitivity, action authority, audit requirements, and fallback-to-human need. This cell creates a simple scoring model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "scenarios = [\n",
        "    {\n",
        "        \"scenario\": \"Workplace: summarize thread before meeting\",\n",
        "        \"latency_sensitivity\": 4,\n",
        "        \"interruption_frequency\": 3,\n",
        "        \"continuity_requirements\": 4,\n",
        "        \"data_sensitivity\": 3,\n",
        "        \"action_authority\": 1,\n",
        "        \"audit_requirements\": 2,\n",
        "        \"fallback_to_human\": 1,\n",
        "    },\n",
        "    {\n",
        "        \"scenario\": \"Contact center: retrieve account and recommend next step\",\n",
        "        \"latency_sensitivity\": 5,\n",
        "        \"interruption_frequency\": 4,\n",
        "        \"continuity_requirements\": 5,\n",
        "        \"data_sensitivity\": 5,\n",
        "        \"action_authority\": 3,\n",
        "        \"audit_requirements\": 5,\n",
        "        \"fallback_to_human\": 4,\n",
        "    },\n",
        "    {\n",
        "        \"scenario\": \"Field work: log issue against asset 4427\",\n",
        "        \"latency_sensitivity\": 5,\n",
        "        \"interruption_frequency\": 4,\n",
        "        \"continuity_requirements\": 4,\n",
        "        \"data_sensitivity\": 3,\n",
        "        \"action_authority\": 3,\n",
        "        \"audit_requirements\": 4,\n",
        "        \"fallback_to_human\": 3,\n",
        "    },\n",
        "]\n",
        "\n",
        "score_df = pd.DataFrame(scenarios)\n",
        "score_df[\"risk_total\"] = score_df[[\n",
        "    \"data_sensitivity\", \"action_authority\", \"audit_requirements\", \"fallback_to_human\"\n",
        "]].sum(axis=1)\n",
        "score_df[\"interaction_demand_total\"] = score_df[[\n",
        "    \"latency_sensitivity\", \"interruption_frequency\", \"continuity_requirements\"\n",
        "]].sum(axis=1)\n",
        "\n",
        "score_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Apply a simple ship / do-not-ship rule\n",
        "\n",
        "The blog recommends not shipping scenarios that are high in sensitivity and action authority before traceability is proven, or high in interruption frequency when barge-in is weak. This cell encodes a simple decision rule."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "def recommend(row):\n",
        "    if row[\"data_sensitivity\"] >= 5 and row[\"action_authority\"] >= 3:\n",
        "        return \"Do not ship yet\"\n",
        "    if row[\"interruption_frequency\"] >= 4 and row[\"latency_sensitivity\"] >= 5:\n",
        "        return \"Ship only if barge-in and latency are proven\"\n",
        "    return \"Good pilot candidate\"\n",
        "\n",
        "score_df[\"recommendation\"] = score_df.apply(recommend, axis=1)\n",
        "score_df[[\"scenario\", \"risk_total\", \"interaction_demand_total\", \"recommendation\"]]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Classify actions by confirmation boundary\n",
        "\n",
        "The blog suggests four action buckets: information retrieval, draft generation, reversible changes, and irreversible or regulated actions. This cell turns that taxonomy into a reusable policy table."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "action_taxonomy = pd.DataFrame([\n",
        "    {\n",
        "        \"bucket\": \"Information retrieval\",\n",
        "        \"risk_level\": \"Low\",\n",
        "        \"confirmation\": \"Usually none\",\n",
        "        \"rollback\": \"Not needed\",\n",
        "        \"example\": \"What changed in the deck since yesterday?\"\n",
        "    },\n",
        "    {\n",
        "        \"bucket\": \"Draft generation\",\n",
        "        \"risk_level\": \"Medium\",\n",
        "        \"confirmation\": \"User review expected\",\n",
        "        \"rollback\": \"Discard or edit draft\",\n",
        "        \"example\": \"Draft the follow-up email\"\n",
        "    },\n",
        "    {\n",
        "        \"bucket\": \"Reversible changes\",\n",
        "        \"risk_level\": \"Medium-High\",\n",
        "        \"confirmation\": \"Explicit confirmation required\",\n",
        "        \"rollback\": \"Clean rollback required\",\n",
        "        \"example\": \"Reschedule the meeting\"\n",
        "    },\n",
        "    {\n",
        "        \"bucket\": \"Irreversible or regulated actions\",\n",
        "        \"risk_level\": \"High\",\n",
        "        \"confirmation\": \"Strong confirmation and often human approval\",\n",
        "        \"rollback\": \"May be impossible or restricted\",\n",
        "        \"example\": \"Submit a regulated customer change\"\n",
        "    },\n",
        "])\n",
        "\n",
        "action_taxonomy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Separate conversational record from business-action record\n",
        "\n",
        "A key governance point in the blog is that enterprises need two records, not one. This example models both records separately so you can validate traceability design."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "conversational_record = {\n",
        "    \"session_id\": \"sess-001\",\n",
        "    \"user\": \"alex@contoso.com\",\n",
        "    \"utterance\": \"Pull the QBR deck, check the latest sales notes, and draft the follow-up.\",\n",
        "    \"context_used\": [\"calendar\", \"email\", \"files\"],\n",
        "    \"assistant_proposal\": \"I found the latest QBR deck and sales notes. I can draft a follow-up email for your review.\"\n",
        "}\n",
        "\n",
        "business_action_record = {\n",
        "    \"session_id\": \"sess-001\",\n",
        "    \"approved_action\": \"Draft follow-up email\",\n",
        "    \"system_changed\": \"Exchange draft mailbox\",\n",
        "    \"authorized_by\": \"alex@contoso.com\",\n",
        "    \"tool_executed\": \"draft_email\",\n",
        "    \"status\": \"completed\"\n",
        "}\n",
        "\n",
        "print(\"Conversational record:\")\n",
        "print(json.dumps(conversational_record, indent=2))\n",
        "print(\"\\nBusiness-action record:\")\n",
        "print(json.dumps(business_action_record, indent=2))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Team readiness self-assessment\n",
        "\n",
        "The blog ends with a practical question: if Copilot became voice-first tomorrow, how ready are your identity, audit, and confirmation boundaries for real work? This cell provides a simple scoring worksheet."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "readiness = {\n",
        "    \"identity_binding\": 4,\n",
        "    \"least_privilege\": 3,\n",
        "    \"confirmation_boundaries\": 2,\n",
        "    \"audit_trail\": 3,\n",
        "    \"retention_rules\": 4,\n",
        "    \"escalation_paths\": 3,\n",
        "}\n",
        "\n",
        "readiness_df = pd.DataFrame(list(readiness.items()), columns=[\"capability\", \"score_1_to_5\"])\n",
        "avg_score = readiness_df[\"score_1_to_5\"].mean()\n",
        "\n",
        "print(readiness_df)\n",
        "print(f\"\\nAverage readiness score: {avg_score:.2f} / 5\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Summary\n",
        "\n",
        "This notebook validated the blog’s main thesis: realtime voice is not just another interface layer, but a pressure test on the full Copilot interaction model. The practical checks centered on latency, barge-in, bounded context, action taxonomy, and governance records.\n",
        "\n",
        "## Next Steps\n",
        "\n",
        "1. Replace the mock context objects with your own Microsoft 365 or line-of-business metadata.\n",
        "2. Instrument real latency checkpoints from capture to first audio in your prototype.\n",
        "3. Test interruption handling under realistic multitasking conditions.\n",
        "4. Define confirmation rules separately for retrieval, drafting, reversible changes, and irreversible actions.\n",
        "5. Review whether your audit, retention, and identity controls are strong enough for voice-first work."
      ]
    }
  ]
}