One developer has custom GitHub Copilot instructions. Another has Claude agents. Someone else has Cursor rules. The platform team wants approved Azure patterns, Microsoft Learn grounding, cost guidance, security checks, and repeatable onboarding.

Six months later, the audit question is simple and uncomfortable:

What agent context was installed, where did it come from, and who allowed it?

Agent Package Manager, or APM, addresses that gap.

APM is a dependency manager for AI agents. The official project describes the core model as declaring the skills, prompts, instructions, plugins, and MCP servers a project needs in one apm.yml, then running apm install to deploy the supported primitives to each agent harness. It also positions the lockfile as the reproducibility artifact, with exact versions and content hashes, and apm-policy.yml as the install-time governance control for dependencies, MCP servers, and targets.

Agent configuration needs this kind of mundane, reviewable plumbing.

Why this matters for Azure teams​

Azure engineering work rarely happens in one tool.

A realistic cloud engineering repo might use:

• GitHub Copilot for day-to-day implementation.
• Claude Code for deep repo work.
• Cursor or OpenCode for local workflows.
• Microsoft Learn MCP for current Azure documentation.
• Azure MCP for cloud resource inspection and operations.
• Foundry MCP for Microsoft Foundry work.
• Context7 for current SDK and framework documentation.
• Team-specific skills for Azure Kubernetes Service, App Service, Azure Functions, Azure API Management, identity, monitoring, reliability, and cost.

In my own Coding repo, the APM manifest now declares the active agent targets explicitly:

targets:
  - copilot
  - claude
  - cursor
  - opencode

It also declares local primitives and reusable agent-skill packages:

dependencies:
  apm:
    - DietrichGebert/ponytail
    - microsoft/skills/.github/plugins/azure-sdk-python
    - microsoft/skills/.github/plugins/azure-sdk-dotnet
    - microsoft/skills/.github/plugins/azure-sdk-typescript
    - microsoft/skills/.github/plugins/azure-sdk-rust
    - microsoft/skills/.github/plugins/microsoft-foundry
    - "microsoft/hve-core#c7ee5b9642ab1877a6b5ce5336ebc448b89b2708"
    - dotnet/skills

The workflow is familiar: declare dependencies, lock the result, and validate it.

MCP servers are dependencies too​

The part I like most is that MCP servers sit in the same dependency story.

That matters because MCP is a tool boundary. If an agent gets access to a server, it gets access to whatever that server exposes. For Azure work, that is powerful and useful, but it should never be invisible.

The repo manifest declares the MCP surface. These are representative entries; the complete manifest is linked in the references.

dependencies:
  mcp:
    - name: microsoft.learn.mcp
      registry: false
      transport: http
      url: https://learn.microsoft.com/api/mcp
    - name: context7
      registry: false
      transport: http
      url: https://mcp.context7.com/mcp
    - name: azure
      registry: false
      transport: stdio
      command: npx
      args:
        - -y
        - '@azure/mcp@latest'
        - server
        - start

This moves MCP from “whatever is configured on my machine” into the repo contract.

The manifest declares the active targets and approved MCP servers:

Governance without ceremony​

I would not start with a giant enterprise policy.

The smallest useful version is an allowlist:

name: Hypervelocity APM baseline
version: 1.0.0
enforcement: block

dependencies:
  allow:
    - DietrichGebert/ponytail
    - dotnet/skills
    - microsoft/hve-core
    - microsoft/skills/**

mcp:
  allow:
    - azure
    - context7
    - foundry-mcp
    - iseplaybook
    - microsoft.learn.mcp
    - microsoft-docs
    - microsoft.mrc.mcp
  transport:
    allow:
      - http
      - sse
      - stdio
  self_defined: allow

compilation:
  target:
    allow:
      - claude
      - copilot
      - cursor
      - opencode

The policy allowlists dependency sources, MCP servers and transports, agent targets, and required manifest metadata. It is install-time governance, not a runtime sandbox: it enforces declarations before files are written, while runtime behaviour remains with the agent harness and exposed tools.

From one repository to an organisation​

The interesting part of APM is not just what happens in one repository.

It is what happens when ten, fifty, or five hundred repositories need the same agent engineering baseline.

An organisation might have separate platform, data, security, application, and AI engineering teams. They should not all have to independently discover the same skills, MCP servers, policies, and agent configuration.

The model can instead be layered:

                         GitHub Organisation
                                │
                       Organisation policy
                                │
                ┌───────────────┼───────────────┐
                │               │               │
             Platform          Data          Security
               team            team             team
                │               │               │
          shared skills    shared skills    shared skills
                │               │               │
                └───────────────┼───────────────┘
                                │
                         Team repositories
                                │
                    ┌───────────┼───────────┐
                    │           │           │
                  Repo A      Repo B      Repo C
                    │           │           │
                  apm.yml     apm.yml     apm.yml
                    │           │           │
                    └───────────┼───────────┘
                                │
                               APM
                                │
                       Agent-specific output

The important distinction is that teams do not need identical agent configurations.

They need a common starting point.

A platform team might provide Azure architecture, AKS, networking, monitoring, reliability, and cost guidance. A security team might provide security and identity guidance. An AI engineering team might provide Foundry and Agent Framework skills.

Individual application teams can then add their own domain-specific context.

That gives the organisation a useful division of responsibility:

Organisation
    │
    └── What is allowed and required?

Team
    │
    └── What capabilities does this team need?

Repository
    │
    └── What does this workload specifically require?

APM
    │
    └── Resolve, lock, install, and audit the result.

This is where APM starts to look less like a developer convenience and more like an organisational distribution mechanism for agent context.

GitHub makes this practical​

For organisations using GitHub, APM can work alongside the existing repository and organisation model rather than creating another management hierarchy.

The organisation can maintain an APM policy in its GitHub organisation configuration. APM can discover that organisation-level policy from the repository's Git remote, allowing the same baseline to apply across participating repositories.

For example, the organisation could establish rules around:

• approved package sources
• approved MCP servers
• approved transports
• allowed agent targets
• required packages
• required manifest metadata

Teams still maintain their own apm.yml.

That means the platform team does not need to edit every application repository whenever the organisational policy changes.

The relationship is closer to:

GitHub Organisation
        │
        ├── Organisation APM policy
        │
        ├── Platform packages
        │
        └── Team repositories
                 │
                 ├── apm.yml
                 ├── apm.lock.yaml
                 └── application code

GitHub then provides the repository governance around that model.

Organisation rulesets can target multiple repositories and can require workflows or status checks before changes are merged. This provides a natural enforcement point for an APM audit workflow.

The result could look like:

Pull Request
     │
     ▼
GitHub Actions
     │
     ├── apm audit --ci
     │
     ├── dependency checks
     │
     └── other engineering validation
     │
     ▼
GitHub Ruleset
     │
     ▼
Required checks pass
     │
     ▼
Merge

This is important because APM policy is an install-time and CI governance mechanism, not a replacement for runtime security or agent sandboxing. Runtime permissions still belong to the agent harness and the underlying platform. APM policy documentation

The platform team becomes a paved road​

This changes the role of a central platform team.

It does not need to become the team that configures every developer's agent.

Instead, it can provide a supported baseline.

For example:

HVE Azure Engineering Baseline

├── Azure architecture
├── Azure Well-Architected
├── AKS
├── App Service
├── Functions
├── API Management
├── Container Apps
├── Identity
├── Monitoring
├── Reliability
├── Cost
└── Approved MCP servers

An application team might then consume that baseline and add:

Application Team

├── HVE Azure Engineering Baseline
├── Payments domain skills
├── Application-specific instructions
└── Workload-specific MCP dependencies

Another team might consume the same baseline but add:

Data Team

├── HVE Azure Engineering Baseline
├── Fabric skills
├── Data engineering guidance
└── Data-specific MCP dependencies

The organisation gets consistency without requiring every team to become identical.

Scaling the rollout​

I would also avoid trying to deploy this everywhere on day one.

The rollout can be incremental:

Pilot
  │
  ▼
One platform team
  │
  ▼
Several application teams
  │
  ▼
Organisation baseline
  │
  ▼
Required CI checks
  │
  ▼
Fleet-wide adoption

Start with a small set of capabilities that the platform team can actually support.

Pin reviewed versions.

Run APM audits in CI.

Use GitHub rulesets to make the checks required where appropriate.

Then expand the baseline as the organisation gains confidence.

That is a much more realistic approach than attempting to define every possible agent policy before anyone has used it.

The goal is not to centrally control every prompt an engineer writes.

The goal is to make the supported path easier than building an agent environment from scratch.

Lockfiles give you the forensic answer​

The lockfile is where APM starts to feel familiar.

In this repo, apm.lock.yaml records the CLI version, generation time, resolved commits, deployed files, and content hashes. For example:

lockfile_version: '1'
generated_at: '2026-08-30T20:20:30.928873+00:00'
apm_version: 0.20.0
dependencies:
- repo_url: DietrichGebert/ponytail
  host: github.com
  resolved_commit: 2ed6c52c9d7e5e56942508591085fd45dea277d3
  package_type: marketplace_plugin
  content_hash: sha256:595855eee726cb344087c38d7a3c29586ce8a0dd9a2e7ad8e8eedc91c8d8c575
- repo_url: dotnet/skills
  host: github.com
  resolved_commit: d68dd70857076a17d4b418649bbcd20a315d59c3
  package_type: marketplace_plugin
  content_hash: sha256:34013ed32b6592ccd762f59215480fa5df94b82d0443b7f5288291ac786978e8

That answers “what was active?” more reliably than a screenshot of local settings.

The lockfile records resolved commits and content hashes for review:

Evidence from the repo​

I validated the current setup with APM CLI 0.20.0.

The useful result:

[*] All primitives validated successfully!
[i] Validated 86 primitives:
[i]   * 11 chatmodes
[i]   * 75 instructions
[i]   * 0 contexts
[i]   * 7 MCP dependencies

[*] All 27 check(s) passed

The command used:

apm --version
apm compile --validate
apm audit --ci --policy .\apm-policy.yml --no-drift --no-fail-fast

The --no-drift flag is worth calling out. I used it because this working tree has generated-output drift unrelated to the policy example. That means the evidence proves manifest, lockfile, MCP config, and policy compliance. It does not claim the generated deployed files are fully drift-clean.

That distinction is important. APM can be used in stages:

1. Get the manifest right.
2. Generate and commit the lockfile.
3. Add a small policy.
4. Pass policy checks in CI.
5. Tighten drift enforcement once generated files are clean and reproducible.

This staged rollout avoids making a giant governance programme a prerequisite.

What HVE Core is​

Hypervelocity Engineering (HVE) Core is Microsoft's open-source library of agentic SDLC workflow building blocks for GitHub Copilot. It brings together specialised agents for research, planning, implementation, and review; reusable prompts; coding instructions; and skills.

The value is not a magic workflow. It is a consistent set of starting points that teams can review, adapt, and use repeatedly. HVE Core is explicitly opinionated and rapidly evolving, so treat it as a source of patterns to evaluate rather than an unexamined platform dependency.

It also makes HVE Core a useful APM example. Instead of copying external skills, agents, prompts, and instructions into a repo, APM declares the upstream package, applies the same policy controls, and locks the exact content alongside the team's local context.

The following is a complete example for someone new to HVE Core: declare the exact source, allow it in policy, install it, and commit the resulting lockfile. The installed content lands in the shared .agents/skills directory, which supported agent harnesses can use.

The quoted reference is deliberate: YAML would otherwise treat # and the commit as a comment.

dependencies:
  apm:
    - "microsoft/hve-core#c7ee5b9642ab1877a6b5ce5336ebc448b89b2708"

dependencies:
  allow:
    - microsoft/hve-core

apm install --target agent-skills
apm audit --ci --policy .\apm-policy.yml

This recorded run uses that setup in an isolated agent-skills target. APM resolves the exact commit, integrates 74 HVE Core skills, and records the commit and content hash in its lockfile:

For a team rollout, start with the subset you can support. Pin a reviewed revision, assess the agents and skills you select, and update deliberately.

For an Azure platform team, I would keep the first implementation small:

apm init
apm install
apm lock
apm audit --ci --policy .\apm-policy.yml

Then add a required CI check:

- uses: actions/checkout@v4
- uses: microsoft/apm-action@v1
- run: apm audit --ci --no-cache

The central platform package would carry approved guidance:

• Azure Well-Architected review prompts.
• AKS architecture and production-readiness skills.
• Azure Functions, App Service, APIM, Container Apps, and IaC skills.
• Azure AI Foundry and Agent Framework skills.
• Security, identity, monitoring, reliability, and cost instructions.

Application repos would keep local context for their own architecture and domain.

The policy should stay focused on the supply-chain boundary:

• allowed package sources
• allowed MCP servers
• allowed transports
• allowed targets
• required manifest fields

Do not encode every engineering preference in policy. That belongs in reviewed skills, instructions, and architecture guidance.

APM does not replace Azure Policy, Defender for Cloud, Entra ID, workload identity, network controls, human review, or runtime permission models. It is package management and CI governance for agent context.

The practical caveat​

The ecosystem is still young.

In my setup, apm lock --parallel-downloads 0 was the reliable path for generating the lockfile. A full install also surfaced useful rough edges: generated hook path drift, target-specific hook naming warnings, and checkout friction in one package on Windows.

I consider that useful signal, not a reason to ignore the model.

The lesson is the same as every other package ecosystem:

• pin what you can
• audit what you install
• keep generated outputs boring
• fix drift before making drift checks mandatory

The practical takeaway​

Skills that guide AKS deployment, APIM configuration, Application Insights instrumentation, or Microsoft Foundry work should be versioned and reviewable. MCP servers exposing Azure operations should be declared and explicitly allowed. APM provides the manifest, lockfile, and policy checks for that work.

References​

• APM homepage
• APM targets matrix
• APM policy overview
• Hypervelocity Engineering Core

The symptom​

Point vectorStoreType at AzureAISearch and the reaction pod crash-loops before it processes a single event:

Vector store connectivity test failed for collection _drasi_test_<hash>
... Index name must only contain lowercase letters, digits or dashes

My first read of that was the obvious one: Azure AI Search must forbid underscores in index names, and the reaction's self-test happens to generate one. Case closed, use a different backend. Except the fix has to survive a search service that's already working correctly for every other collection in the same account, so "underscores are banned" was worth checking rather than assuming.

Reading the source instead of the error text​

The reaction is open source, and the crash trace pointed straight at a specific method. QueryInitializationService.cs runs a connectivity self-test before it touches any of your configuration:

private async Task TestVectorStoreConnectivity(CancellationToken cancellationToken)
{
    var testCollectionName = $"_drasi_test_{Guid.NewGuid():N}";
    // ... creates a collection with this exact name, then throws if it fails
}

That call happens as the literal first line of InitializeQueriesAsync(), before per-query config, before createCollection: false on your own collection gets read, before anything you control has a chance to run. No secret reference, no alternate collection name, no client-side setting avoids it. The self-test builds its own configuration object from scratch every time.

So the question stopped being "how do I configure around this" and became "what is the actual naming rule, and is it really as broad as the error text implies." I tested it directly against the live search service rather than trust the message:

curl -X PUT "$ENDPOINT/indexes('drasi_test_abc123')?api-version=2024-07-01" -H "api-key: $KEY" -d @schema.json
# HTTP 201 Created

curl -X PUT "$ENDPOINT/indexes('_drasi_test_abc123')?api-version=2024-07-01" -H "api-key: $KEY" -d @schema.json
# HTTP 400 InvalidName

Same schema, same account, one character different. Microsoft's own naming-rules documentation confirms the actual constraint: an index name must start with a letter or digit. The error text only calls out the dash rule explicitly, which is why "no underscores" is such an easy wrong conclusion to reach from the message alone. The real rule is narrower: no leading underscore or dash, full stop. Underscores anywhere else in the name are completely fine.

That distinction matters because it changes what the fix has to be. "No underscores" is unfixable without owning the reaction's source. "No leading underscore" is a single character.

The bug shows up twice, not once​

Once I'd worked around the first occurrence (more on how below), the reaction crash-looped again on a second, more consequential prefix. SyncPointManager.cs hardcodes the name of the permanent collection every reaction instance uses to track what it's already processed:

private const string MetadataCollectionPrefix = "_drasi_metadata_";

This is the sync-point bookkeeping the reaction depends on for its entire lifetime, created fresh on every bootstrap. An upstream fix limited to the self-test's name would still leave AzureAISearch unusable because this collection hits the same constraint. Both prefixes need the same one-character correction.

Why a proxy alone wasn't enough​

The obvious workaround, once you know the exact string being generated, is a reverse proxy that rewrites the name in flight: catch any request path or body containing _drasi_ and strip the leading underscore before it reaches the real service. I built this with mitmproxy in reverse mode and a short addon script, and it worked exactly as expected against the naming check.

It still crashed, on a different exception:

System.ArgumentException: endpoint only supports https. (Parameter 'endpoint')
   at Azure.Search.Documents.SearchExtensions.AssertHttpsScheme(Uri endpoint, String paramName)

The Azure Search SDK refuses a plain HTTP endpoint outright, and there's no configuration surface on the reaction to disable certificate validation either. A self-signed certificate on the proxy wouldn't have helped: the .NET client validates the certificate chain the same way it would for any other HTTPS call, and nothing in the reaction's config lets you skip that. Getting past both problems meant the proxy needed a certificate a real public certificate authority actually issued.

Building the rest of it​

Standing up a trusted certificate for a Kubernetes-hosted test service, without owning a domain, took three pieces:

• ingress-nginx and cert-manager, both official Helm charts, installed with no changes beyond the defaults.
• A ClusterIssuer for Let's Encrypt production, using the HTTP-01 challenge against the ingress.
• A hostname to issue the certificate for. I looked at dnsbox.io first, which turned out to be self-hosted DNS-authority software you install and run yourself as an internet-facing nameserver, a bigger trust decision than the problem warranted. nip.io is simpler: it's a wildcard DNS resolver that maps an IP address straight into a hostname (20-40-178-12.nip.io resolves to 20.40.178.12), no signup, nothing to install, nothing running on your own infrastructure that you'd need to trust.

With the ingress's public IP in hand, an Ingress resource pointed at that nip.io hostname, and the ClusterIssuer annotation in place, cert-manager requested a certificate. It failed the first time, and not for a reason that had anything to do with certificates.

The gotcha that looked like a firewall problem​

External traffic to the ingress timed out completely, both from a plain curl and from Let's Encrypt's own HTTP-01 validator. Every layer I checked looked correct: the pod was Ready, the Kubernetes Service had an external IP, and the network security group had an explicit allow rule for ports 80 and 443 from the internet. None of that mattered.

The actual cause was Azure's Standard Load Balancer, which by default health-checks the same node port your real traffic uses, on path /. ingress-nginx's default backend returns a 404 for any request whose Host header doesn't match a configured Ingress, which is exactly what an untargeted Azure health probe sends. Azure marked the backend unhealthy and dropped every packet to it, which looks identical to a firewall block from the outside while every dashboard claims the service is healthy.

kubectl annotate svc ingress-nginx-controller -n ingress-nginx \
  service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path=/healthz

That points the probe at ingress-nginx's own always-200 health endpoint instead of the data path. My first attempt at this exact command silently failed for a completely different reason: Git Bash on Windows rewrites a bare leading slash before kubectl ever sees it, so /healthz became C:/Program Files/Git/healthz with no error from anything. MSYS_NO_PATHCONV=1 in front of the command, or doubling the leading slash, avoids it. Worth knowing if you're running kubectl or az from Git Bash on Windows and an argument starts with /.

Confirming the fix​

With the health probe fixed, the certificate issued on the first real attempt. I pointed the reaction's connectionString at the proxy's new HTTPS hostname, triggered a genuine write against the underlying Postgres table, and watched the reaction log a real embedding call and a real upsert. Rather than trust that log line the way I'd learned not to trust the original error text, I checked the actual index directly, with no proxy in the request path this time:

curl "$ENDPOINT/indexes('products-test')/docs('prod-1')?api-version=2024-07-01" -H "api-key: $KEY"

The response came back with the real product row and its full 3072-dimension embedding vector, stored in the real Azure AI Search service. AzureAISearch isn't a backend to avoid at this platform version. It's a backend that needs one string fixed in two places, and until that ships upstream, a rewrite proxy in front of a properly certificated endpoint gets you the rest of the way there.

Operational notes​

None of this is something to leave running. The proxy holds a real Azure AI Search admin key, and standing up a public ingress means exactly that: public. Within an hour of the certificate issuing, the access log already showed automated scanning traffic hitting the IP on unrelated paths, fake browser user agents probing for open admin panels and API endpoints that don't exist here. Ordinary background noise for anything with a public IP, not an attack on this setup specifically, but a reminder that "it's just a test proxy" doesn't make it invisible.

Nothing here (ingress-nginx, cert-manager, the proxy deployment, the ClusterIssuer) was provisioned through the project's own infrastructure-as-code, so azd down or an equivalent teardown of the managed resources won't remove any of it. Tear it down explicitly, and don't leave a naming-rewrite proxy with a live admin key sitting on the internet longer than the test needs.

What this is actually worth​

The specific bug will get fixed or it won't. What's reusable is the checking habit: an error message describes what a service noticed, not necessarily the actual rule. "Index names forbid underscores" was a plausible reading of that error text and it was wrong, in a way that would have sent me down a much longer path (switching vector store backends entirely) if I hadn't tested the boundary case directly. When a config surface won't budge and the fix looks like it needs to live outside the application, ask whether the actual constraint is narrower than the failure message suggests before reaching for infrastructure.

References​

• Azure AI Search - index naming rules
• drasi-project/drasi-platform
• cert-manager documentation
• ingress-nginx documentation
• nip.io

Drasi has an MCP Reaction that closes that gap. It exposes each continuous query as an MCP resource that a client can subscribe to, and the server pushes a notification the moment the result set changes, no polling, no re-fetching on a timer. I wanted to see whether that actually held up with a real client rather than trusting the resource model on paper, so I connected the official MCP Inspector to a live reaction and watched what happened.

Resources and subscriptions in ninety seconds​

Every Drasi continuous query becomes a resource at drasi://query/{id}. A client that subscribes to it starts receiving notifications/resources/updated whenever the result set changes, tagged with the operation type (added, updated, or deleted) and a Handlebars-formatted payload. Your agent doesn't poll, the data arrives when it changes, and until a client subscribes the server does nothing but track state quietly in the background.

The dotted lines show the model: subscribe once, then the server pushes changes to the client.

The server YAML, corrected​

The nested YAML shape some inventory examples show does not parse against the pinned platform version. spec.queries.<query-id> has to be a JSON string, the same convention SyncVectorStore uses for its own per-query config, not a nested map with description: and added: as literal YAML keys. Get the shape wrong and it fails fast, before the reaction pod even starts, with 400 Bad Request: Json deserialize error: invalid type: map, expected a string.

kind: Reaction
apiVersion: v1
name: inventory-mcp
spec:
  kind: MCP
  queries:
    low-stock-alerts: |
      {
        "description": "Products below reorder threshold",
        "added": {
          "template": "{ \"productId\": \"{{after.productId}}\", \"alert\": \"{{after.productName}} is below reorder level\" }"
        },
        "updated": {
          "template": "{ \"productId\": \"{{after.productId}}\", \"stockChanged\": \"{{before.stock}} -> {{after.stock}}\" }"
        },
        "deleted": {
          "template": "{ \"productId\": \"{{before.productId}}\", \"alert\": \"back in stock\" }"
        }
      }

The template value is a JSON string nested inside the outer JSON string, so its quotes need escaping. If one level is wrong, the error message does not identify the broken layer. Build the queries one at a time.

The before/after template context here is the same model as EventGrid reaction templates, learn it once and it carries across every Drasi reaction that uses templates.

Tested with MCP Inspector​

I ran the official MCP Inspector against a port-forwarded reaction, connected as streamable-http (the endpoint is the bare root /, not /mcp, worth probing before you assume a path), and subscribed to a query watching a Kubernetes deployment that was sitting under-replicated in the cluster.

I then fixed the deployment with a normal kubectl update. A notifications/resources/updated message arrived at the client unprompted, with no preceding request, and re-reading the resource confirmed the payload had gone from the under-replicated row to empty. The live client received the pushed update without polling.

One small thing to check before you build against this yourself: resources/templates/list returns Method not found. That's an unimplemented optional capability, not a bug, the reaction only implements resources/list, resources/read, and resources/subscribe.

The pinned MCP spec and negotiated client version​

Drasi's documentation states it pins MCP spec 2025-03-26, while upstream MCP has since shipped 2025-06-18, 2025-11-25, and 2026-07-28 (the last of which introduced an extensions architecture: Tasks, MCP Apps, Skills over MCP). In practice, put auth enforcement at the gateway, don't expect OIDC discovery metadata, and record the exact spec revision pair in your own ADR before you build against it.

One loose end I haven't resolved: the live Inspector session negotiated protocol version 2025-06-18 during initialize, not the 2025-03-26 the documentation states. I don't yet know whether that means the documented pin is stale, or whether the reaction simply accepts whatever version a client requests without enforcing its own. Confirm which before you rely on an exact spec-lag number for your own build, and don't take either side's claim as gospel until you've checked it against your own client.

Two client paths that don't do this today​

I checked both the Microsoft Agent Framework SDK and the current Copilot Studio MCP integration. Neither supports resource subscriptions.

Microsoft Agent Framework's built-in MCP support (MCPStreamableHTTPTool, MCPStdioTool) connects to a server and hands the agent its functions. Tools, callable on request. Every code path in the framework that touches an MCP server does the same thing: enumerate the tools, append them to the agent's toolset, call them when the model decides to. There's no resources/subscribe API anywhere in the SDK. To wire up the pattern in this post you'd drop past Agent Framework's MCP wrapper entirely, use a lower-level MCP client library to hold the subscription yourself, and feed whatever arrives into the agent's context on your own terms. That's a real, buildable pattern. I just haven't built and tested it yet, and I'm not going to write it up as five minutes of framework support when it isn't.

Copilot Studio's MCP integration is the same story from a different angle. The current documentation describes exactly one connection shape: add an MCP server as a tool, configure a transport and auth, and the agent orchestrator calls it. The sample schema even names the operation InvokeMCP. Nothing in the current docs mentions resource subscriptions or push notifications. If you want your Copilot Studio agent reacting to a Drasi query changing rather than calling it as a tool, today that means bridging it yourself, an HTTP reaction posting to a Power Automate flow or a custom connector, not something Copilot Studio does for you out of the box.

Neither of those is a knock on Drasi. The MCP Reaction does exactly what it claims, subscribe and get pushed updates, and I proved that above with a real client. The lag is on the client ecosystem side, which still treats MCP as tools first, resources later. Plan around that, because both of these frameworks are built on the tools-only assumption today.

Hopefully this gives your Copilot something better to work with than a snapshot from five minutes ago.

References​

• Drasi documentation
• Model Context Protocol specification
• MCP Inspector
• Microsoft Agent Framework
• Connect your agent to an existing MCP server - Microsoft Copilot Studio
• drasi-project/drasi-platform

The mental model shift​

The continuous query result set is the live corpus. Sync reconciles against that result set, and embeddings happen per change. The embedding bill scales with churn rather than corpus size. A nightly reindex spends N times the corpus in embeddings every day, no matter how little moved.

The working sample​

apiVersion: v1
kind: Source
name: products-db
spec:
  kind: PostgreSQL
  properties:
    host: <your-server>.postgres.database.azure.com
    port: 5432
    database: products
    ssl: true
---
apiVersion: v1
kind: ContinuousQuery
name: searchable-products
spec:
  mode: query
  sources:
    subscriptions:
      - id: products-db
  query: >
    MATCH (p:products)
    WHERE p.instock = true AND p.description IS NOT NULL
    RETURN p.id AS product_id, p.name AS name,
           p.category AS category, p.description AS description
---
apiVersion: v1
kind: Reaction
name: product-vectors
spec:
  kind: SyncVectorStore
  properties:
    vectorStoreType: InMemory # see "upstream bug" below, this is currently the only backend that works end to end
    embeddingServiceType: AzureOpenAI
    embeddingEndpoint: https://<your-foundry>.openai.azure.com/
    embeddingModel: text-embedding-3-large
    embeddingApiKey:
      kind: Secret
      name: vectorstore-keys
      key: openai-key
    embeddingDimensions: 3072
    distanceFunction: CosineSimilarity
    indexKind: Hnsw
  queries:
    searchable-products: |
      {
        "collectionName": "products",
        "keyField": "product_id",
        "documentTemplate": "Product: {{name}}\nCategory: {{category}}\n{{description}}",
        "titleTemplate": "{{name}}",
        "createCollection": true
      }

keyField drives upsert idempotency; duplicates must dedupe on this key or copies accumulate silently. documentTemplate is Handlebars, so shape it for embedding quality as you would any RAG chunk, rather than as a data dump. embeddingServiceType is AzureOpenAI only at the pinned platform version; a guide mentioning "and OpenAI" predates this release.

One config gotcha cost me a round of debugging: Secret references work for embeddingApiKey, but not for connectionString, the object gets passed through verbatim and the connectivity test fails. Apply-time substitution is the workaround for the connection string specifically.

Deletion semantics, the headline result​

The central claim of this whole approach is that the continuous query result set is the corpus, continuously reconciled, including the deletion half. I tested this directly rather than trusting the docs to be right about it.

With the pipeline running and one product synced, I ran UPDATE products SET instock = false WHERE id = 'prod-1'; directly against Postgres. The product left the continuous query's result set within seconds, and the reaction logs showed the remaining changes.

That's a real Azure OpenAI embedding call and a real vector store deletion, not a description of expected behaviour. When a row leaves the result set the reaction deletes the corresponding document (Deleted: 1 in the log), and when the row re-enters, it generates a fresh embedding and adds it back (Added: 1). Deletion included, the claim holds up, which is the bit most of these write-ups skip.

The cost math follows from the same log line. The reaction issues exactly one embedding batch per result-set transition, not per corpus:

Processing change event for query searchable-products with sequence 141.
Added: 0, Updated: 0, Deleted: 1

Run the numbers on a 100,000-document catalogue with 1% daily churn and the difference is stark: a nightly reindex costs 100,000 embeddings a day regardless of what changed, Drasi costs roughly 1,000. That's the per-event batch shape scaled up, measured, not projected.

Why this post uses the in-memory backend​

I tested the AzureAISearch backend first, and it is unusable at this platform version. The reaction crash-loops on startup because it hardcodes a leading-underscore index name (_drasi_test_<hash>), which Azure AI Search rejects. It is a one-character bug in the reaction's own code, not a configuration gap on your side.

The InMemory vector store proves the sync, embedding, and reconciliation logic works end to end while that upstream bug gets fixed. The mechanism, including deletion, is backend-agnostic. The separate post below covers the root cause, workaround, and live verification for AzureAISearch:

• Fixing Drasi's 'unusable' Azure AI Search backend

Operational notes​

• Failed embeddings are retried then logged, poison documents are not auto-dropped, alert on the log signature rather than assuming silent success.
• Sync is asynchronous, vector writes don't block the change pipeline.
• drasi apply is not an upsert, re-applying an existing resource returns 500 Internal Server Error rather than updating it. Delete then apply is the safe pattern for any config change.

Beyond freshness​

The same pattern extends past a product catalogue. A continuous query can pre-filter by access control list for permission-aware RAG, so the vector store only ever contains documents a given caller is allowed to see in the first place, rather than filtering after retrieval. A query scoped per tenant gives you a multi-tenant collection the same way, without building that isolation into the reaction itself. Neither of those is something I've tested end to end yet, but the mechanism is identical to what's already proven above, the filter just lives in the WHERE clause instead of the instock flag.

And if you want an agent to see these changes live rather than querying the index at all, Drasi has a different reaction built for exactly that, the MCP Reaction, which exposes a continuous query as a subscribable resource instead of a search index.

References​

• Drasi documentation
• Drasi SyncVectorStore reaction reference
• Azure AI Search - index naming rules
• drasi-project/drasi-platform

Drasi answers relational questions about live cluster state after admission: is this Deployment still under-replicated after five minutes, and is that Pod running an unapproved image? I tested standing queries over a live AKS resource graph. Only two of the six rules I set out to build survived unchanged; the gap between what looked right on paper and what ran was the useful result.

What each tool actually owns​

Drasi's niche is standing Cypher over the live resource graph, with temporal functions (trueFor, trueLater) and a result set that stays current on its own. Reactions fire on transitions, not on poll ticks.

Deploy Drasi into the same cluster, or point an external instance at it via a kubeconfig Secret. RBAC matters here: the source credential only needs list/watch on the resource types your rules actually read, scope it the way you'd scope a read-only dashboard service account.

The rulebook, corrected​

I wrote six rules against the documentation first, then ran every one of them against a live cluster with deliberately induced violations. Only two survived unchanged. Two needed rewriting, and two don't work at all, and that last pair isn't because of a query bug.

The Kubernetes source watches a fixed list of twelve resource types, no matter what your queries reference: Pod, Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, Service, ServiceAccount, Node, Ingress, PersistentVolume, and PersistentVolumeClaim. Namespace, NetworkPolicy, PodDisruptionBudget, and Secret aren't on that list, and no RBAC grant changes it. So those three cut rules are a platform limitation, not a query bug. You work around it by picking a different resource to watch, not by Cypher-ing your way out of it.

The cut rules all trace back to the watch list, not bad query logic. Three replacement rules earned a spot. Two are proven against real violations; I only proved the third on syntax.

Under-replicated deployments, sustained​

MATCH (d:Deployment)
WHERE d.spec.replicas > coalesce(d.status.readyReplicas, 0)
  AND drasi.trueFor(d.spec.replicas > coalesce(d.status.readyReplicas, 0),
        duration({ minutes: 5 }))
RETURN d.metadata.name AS name, d.metadata.namespace AS namespace,
       d.spec.replicas AS desired, coalesce(d.status.readyReplicas, 0) AS ready

The trueFor debounce is the whole point here, flapping during a normal rollout doesn't fire, genuine degradation does. readyReplicas is null during rollout, so coalesce isn't optional, the original query without it errors on any deployment mid-rollout.

I induced a real violation (an impossible nodeSelector that could never schedule) and tested with a shortened five-minute window rather than production timing, purely to see the transition without a long wait. It fired as expected once the debounce elapsed:

{ "desired": 2, "name": "broken-app", "namespace": "drift-test", "ready": 0 }

Images outside the approved registry​

MATCH (p:Pod)-[:HAS]->(c:Container)
WHERE left(c.image, 22) <> 'ghcr.io/drasi-project/'
  AND left(c.image, 18) <> 'mcr.microsoft.com/'
RETURN p.metadata.namespace AS namespace, p.metadata.name AS pod, c.image AS image

STARTS WITH, and CONTAINS and ENDS WITH alongside it, don't exist in the pinned Cypher parser. left(image, N) <> 'prefix' is the working shape. Admission policy already blocks these at create, this rule catches anything that mutates state afterward, a debug override, a manual kubectl, or compromised tooling.

The live result set correctly flagged a docker.io/library/nginx deployment I'd deliberately mislabelled. It also surfaced two allowlist cases that a toy cluster would miss:

• Dapr sidecars report as docker.io/daprio/daprd:1.14.5. Allowlist docker.io/daprio/ explicitly, or every Drasi pod with a sidecar flags.
• Calico components report a bare sha256:<digest>, with no registry prefix at all. A prefix-based rule is blind to digest-pinned images, and that's a genuine blind spot worth calling out rather than a footnote.

Cut: namespaces without a NetworkPolicy​

-- doesn't run: NetworkPolicy and Namespace aren't in the Kubernetes
-- Source's fixed watch list
MATCH (n:Namespace)
WHERE NOT EXISTS { MATCH (n)-[:HAS]->(:NetworkPolicy) }
RETURN n.metadata.name

Two independent problems stack up here. EXISTS { MATCH ... } existential subqueries don't parse in this Cypher subset regardless, and even a rewritten version has nothing to match against, because Namespace and NetworkPolicy nodes never materialise in the graph. The graph shape genuinely does make absence first-class, a standing query can express "this relationship never showed up" in a way a metrics system struggles with, just not for resource types the source doesn't watch yet.

Cut: certificate expiry and PodDisruptionBudget exhaustion​

Both hit the same wall. Certificate expiry needs to watch Secrets, PDB exhaustion needs to watch PodDisruptionBudgets, and neither resource type is on the fixed watch list. The trueLater self-scheduling idea behind the certificate rule is still the right teaching point, no cron job scanning secrets on a timer, it just isn't provable against Kubernetes Secrets today. If cert-manager's Certificate CRD is on your cluster, that's the more promising angle, though I haven't tested it.

New: sustained node pressure, syntax proven, not yet fired on a real violation​

MATCH (n:Node)-[:HAS]->(c:NodeCondition)
WHERE c.type IN ['MemoryPressure', 'DiskPressure']
  AND c.status = 'True'
  AND drasi.trueFor(c.type IN ['MemoryPressure', 'DiskPressure'] AND c.status = 'True',
        duration({ seconds: 60 }))
RETURN n.metadata.name AS node, c.type AS condition, c.status AS status

Node is on the watch list, but its conditions live under status.conditions, the same array shape as a container's status, so it needs the same unwind treatment as the crashlooping rule below. It applies cleanly and reaches Running, correctly empty against a healthy cluster, both nodes reporting MemoryPressure: false and DiskPressure: false throughout. What I didn't do is force an actual violation to prove the trueFor transition, this node is shared with Drasi's own control plane, and deliberately exhausting its memory to trigger a real pressure condition risks taking down the whole environment for the sake of one test. The syntax and the unwind mechanics are proven, the live transition isn't, and I'd rather say that plainly than claim a clean fire I didn't actually see.

New: crashlooping containers​

MATCH (c:Container)
WHERE c.state.waiting.reason = 'CrashLoopBackOff'
RETURN c.metadata.namespace, c.metadata.name

This needs an unwind middleware to extract containerStatuses off each Pod into its own Container nodes first. The middleware block nests under sources: in the manifest, not spec:, get that wrong and the apply silently fails with no obviously helpful error. Correctly empty against a healthy cluster, then fired the moment I pushed a container into CrashLoopBackOff.

New: replicaless deployments​

MATCH (d:Deployment)
OPTIONAL MATCH (d)-[:owns]->(rs:ReplicaSet)
WITH d, count(rs) AS replicaSetCount
WHERE replicaSetCount = 0
RETURN d.metadata.namespace, d.metadata.name

My first instinct was an EXISTS {} subquery again, same parser problem. OPTIONAL MATCH with count() is the working equivalent, and it correctly stayed empty, every deployment on the test cluster had a ReplicaSet, as expected.

Parser surface, learned the hard way​

None of the following came from documentation, I read it off the query-host's own error diagnostics:

• Supported comparisons: =, <>, !=, <, <=, >, >=, IN, IS, plus arithmetic
• Not supported: STARTS WITH, CONTAINS, ENDS WITH, =~ regex, and EXISTS { MATCH } existential subqueries
• Supported: left(), coalesce(), OPTIONAL MATCH with aggregation, and the unwind middleware
• Middleware nests under sources: in the manifest, not spec:

One small bonus for anyone verifying this themselves: listing the actual pod images on the test cluster exposed the platform's real component names (query-container-query-host, query-container-view-svc, query-container-publish-api, source-query-api, source-change-dispatcher, source-change-router), which explains why bare query-host and view-svc registry probes always come back 404.

Alert philosophy mapping​

The rulebook is version-controlled YAML, reviewed like code, for the resources the source can actually see today.

Operational notes from a live build​

• Kubeconfig is stored as a Kubernetes Secret. Avoid exec-based kubeconfigs in runtime containers.
• Watch for ResourceVersionTooOld events on the source namespace, a desynced watch cache silently misses changes without raising anything obvious.
• Modifying the Kubernetes Source means delete and recreate, plus recreating any dependent queries, unlike SQL Server sources which re-apply cleanly by name.
• Default provider registration is automatic after drasi init. Older guidance saying you need to apply the default provider manifests yourself is stale for the current CLI generation.

When not to do this​

Admission enforcement still belongs to Gatekeeper or Kyverno, Drasi observes, it doesn't block. Git-sync drift still belongs to Flux or Argo. And keep one Drasi instance per cluster, don't point a single instance across clusters, that's a lesson I only half learned by reading about it.

Hopefully this saves you from rediscovering the watch-list gap the hard way. Five working rules against a real cluster, with the exact syntax that actually parses, is a better starting rulebook than six that look right on paper.

References​

• Drasi documentation
• Drasi Kubernetes Source configuration
• Azure Kubernetes Service RBAC
• drasi-project/drasi-platform

Drasi, a CNCF Sandbox project for continuous queries over changing data, handles a question SQL cannot express well: "alert me when something stops happening." I have been testing it against a real AKS cluster and Azure Database for PostgreSQL.

The polling tax​

"Alert me if a device hasn't checked in for 20 minutes" sounds like a simple requirement until you have to build it. The honest version needs a poller running on a schedule, a query for max(last_seen) per device, a state table so you don't re-fire the same alert every poll cycle, and careful handling of the boundary between polls (a device that goes quiet at minute 19 of a 20-minute check shouldn't wait an extra cycle to be noticed). None of that logic is about the business problem. It compensates for a normal query's inability to identify an event that failed to occur during a time window.

Drasi's standing queries flip this. You write the query once, and the result set is always current, maintained continuously as the underlying data changes, rather than recomputed on a schedule.

The left side is every poller I have ever written, and that's the set of problems that sold me on this. The right side is the same requirement with the scaffolding gone, no timer, no state table, no dedup logic. The result set is the state now, it maintains itself, and for absence specifically a pair of temporal functions carry the whole thing.

The function toolbox​

While trueLater waits for its scheduled timestamp, it returns the special value drasi.awaiting, so the result set holds rows that have not resolved yet. I tested this against a live cluster, and the behaviour is specific to trueLater: a trueFor row stays hidden until the debounce window elapses and never surfaces in an awaiting state. Conflating the two makes reaction logic treat a normal pending row as a bug.

Worked example: the silent sensor​

The pattern that sold me on the whole model is the "device gone quiet" query, because it has to do two things at once: catch devices that are already overdue when the query is created, and schedule a future check for devices that will go quiet later.

MATCH (d:devices)
WHERE drasi.trueLater(
      datetime() >= datetime(d.lastseen + 'Z') + duration({ minutes: 20 }),
      datetime(d.lastseen + 'Z') + duration({ minutes: 20 }))
RETURN d.id AS id, d.lastseen AS lastseen

The + 'Z' appends the timezone marker. PostgreSQL's timestamptz columns arrive over the wire as offset-less ISO strings (2026-08-26T05:35:41.699091, no timezone marker), and feeding one directly into datetime() throws FunctionError: InvalidFormat because it requires an RFC 3339 string. The first bad row terminal-errors the whole query. Appending the offset before parsing fixes it.

Worked example: unacknowledged alert escalation​

The silent-sensor pattern schedules one future check per device. The second half of absence detection is the sustained condition: not "has it gone quiet," but "has this been true for too long." That is a different function, trueFor, and it is the difference between escalation-as-a-query and escalation-as-a-workflow-engine:

MATCH (a:alerts)
WHERE a.severity = 'P1'
  AND a.acknowledgedby IS NULL
  AND drasi.trueFor(
        a.severity = 'P1' AND a.acknowledgedby IS NULL,
        duration({ seconds: 60 }))
RETURN a.id AS id, a.severity AS severity, a.raisedat AS raisedat

trueLater puts a row in the result set at a future moment; trueFor keeps a row out until a condition has held continuously. One schedules and the other debounces. Together they cover absence detection.

I tested this with a shortened 60-second window rather than a real SLA, purely to see the transition without waiting around. ALR-100, a P1 alert with nobody assigned, entered the result set only once the full 60 seconds of continuous truth had elapsed, never before. Use whatever duration your actual SLA needs, the mechanism doesn't care.

Both results above came straight off the Debug Reaction gateway (GET /api/queries/{id}) against a live cluster, not a docs example. The three devices sitting in the silent-devices result set are the trueLater awaiting state in practice, present in the result well before their 20-minute deadlines, each carrying its lastseen value so a reaction can show useful context rather than a bare row.

Gotchas from a live build​

Each of these cost me a debugging round against a real Azure PostgreSQL Flexible Server and AKS, so here they are up front rather than buried in a support thread somewhere:

1. Grant ordering. Running GRANT SELECT ... ON ALL TABLES before the tables exist grants nothing to anything created afterwards. Bootstrap then fails with permission denied for table.
2. timestamptz arrives as an offset-less ISO string. Covered above, datetime(col) alone throws and terminal-errors the whole query.
3. The 0.10.x PostgreSQL source manages its own filtered publication (rg_<source-name>), in addition to any you create yourself. It needs, in order, CREATE ON DATABASE, table ownership (for publication membership, which itself needs GRANT drasi_user TO admin), and CREATE ON SCHEMA public for the new owner. Bootstrap works fine without any of this, streaming just dies silently, which is the trap.
4. wal_level=logical is a static Postgres parameter. ARM reports value=logical the instant you set it, but the running server stays on replica until a full restart. Symptom: bootstrap succeeds, zero changes ever stream, and slot creation fails with "logical decoding requires wal_level >= logical."
5. The server admin login doesn't have the REPLICATION attribute by default, even as a member of azure_pg_admin. Point a source straight at the admin login (rather than a dedicated replication role) and it fails with a different error again, FATAL: permission denied to start WAL sender. Fix: ALTER ROLE <login> WITH REPLICATION;.
6. Deleting a row can crash the entire reactivator, not just the affected query. Postgres's default REPLICA IDENTITY only sends primary-key columns in a DELETE's old-row image, everything else arrives as null. Drasi generates a message schema mirroring your NOT NULL constraints, so a legitimately-null column in that DELETE event fails the generated schema's own validation and takes the whole connector down (Invalid value: null used for required field, then CrashLoopBackOff). Run ALTER TABLE <table> REPLICA IDENTITY FULL; on any watched table with NOT NULL columns outside the primary key, before you ever delete a row from it.
7. drasi apply is not an upsert. Re-applying an existing ContinuousQuery returns 500 Internal Server Error, and it blocks every other document in the same multi-doc file from applying. Delete, then apply.
8. Default providers are auto-registered. Older guidance saying source and reaction providers need a separate apply step after drasi init is stale for the current CLI generation, I watched them register automatically during init.

When not to do this​

Standing queries are the wrong tool for high-frequency flapping conditions on their own, a raw absence check on something that oscillates rapidly will fire constantly. Combine it with trueFor to debounce rather than alerting on every raw transition, the escalation example above is really that same pattern applied to alert acknowledgement.

That's enough to go and swap a poller for a standing query. The gotchas above are the ones that cost me actual debugging rounds, so hopefully they read as a head start rather than a repeat of the same wall.

References​

• Drasi documentation
• Drasi custom functions reference
• Drasi for Kubernetes installation guides
• Azure Database for PostgreSQL Flexible Server - logical replication
• drasi-project/drasi-platform

You've been in the room. The energy is good, the sticky notes pile up faster than you can read them, and half the team thinks the key insight was something someone mentioned in passing at 2:15 PM. When you're trying to hand off to delivery two weeks later, the connection between "what a customer said" and "what the team decided to build" usually lives in someone's memory, not a record.

That gap is the point of Evidence to Decision. I set one constraint and let it drive everything else: evidence stays attributable, and decisions stay accountable. An AI agent can draft and recommend, but it can never approve or mutate canonical state. That rule shapes the domain model, the agent prompts, and the UI.

The pipeline works like this: facilitators capture evidence (what was said, by whom, about what), the room runs through ideation rounds with live voting and clustering on a shared mural, an AI agent drafts a candidate problem frame citing the evidence it sees, a reviewer approves or blocks the draft, and only then does the frame become canonical. Every step is a separate surface, every transition requires human judgment, and the AI is advisory-only.

Architecture​

I went for clean architecture: Domain / Application / Infrastructure / API layers, with a single canonical OpportunityGraph per engagement as the only authoritative state. Everything else (recommendations, cards, review views, handoff artefacts) is derived and cannot approve or mutate it.

Cosmos DB holds an event-style record of the graph, with optimistic concurrency via ETags on every mutation. It is the only system of record.

Azure SignalR Service backs the live surfaces. Vote tallies update as people vote, live cursors show where other facilitators are pointing, and the shared mural broadcasts placements and drags in real time.

Microsoft Foundry and Microsoft Agent Framework power the AI capabilities: recommendation drafting, Discovery Card suggestions, evidence-quality checks, Frame drafting, and board clustering. Each uses the same audited, policy-gated call pattern. There are seven agents, including a Frame-drafting agent that triggers an automated critique workflow through Agent Framework.

Azure Container Apps hosts the API and Azure Static Web Apps hosts the frontend. Bicep deploys both from one azd environment, using managed identity throughout instead of connection strings or account keys.

Entra ID authenticates facilitators and reviewers. Anonymous workshop participants use a separate HMAC-signed, short-lived JWT scheme. It lets them vote, view, and comment, but it cannot mutate the canonical graph.

The live mural, without a whiteboard SDK​

I built the shared canvas (zoom and pan, drag-to-place cards, live cursors, undo, private and reveal rounds, fullscreen presenter mode) directly on Azure SignalR groups rather than reaching for a commercial whiteboard SDK.

SignalR groups are opaque. You can broadcast to a group, but you cannot enumerate its members. That constraint shaped how "start a new private round" works. In a private round, only facilitators see the cards until someone reveals them. The server tracks round state, and clients ask whether reveal mode is active on every render.

The zoom and pan math, the mini-map, and the zone-count overlay are all built on normalised (0..1) card coordinates that stay meaningful regardless of canvas size. That way a card placed on the big screen stays in the same relative position when someone's viewing it on a laptop or a tablet.

AI drafts, never decides​

This is the bit that took the most tuning. The Frame-drafting agent only sees evidence explicitly captured in the workshop. It cites literal evidence IDs (never invents or paraphrases one). A deterministic C# validator rejects any output that cites something outside its authorised context before a human ever sees it.

Every agent shares the same advisory-only contract: never approve a gate, never claim to mutate canonical state, and always name what a human still needs to check. The Frame-draft agent can search ideation notes beyond the default recency window. It then passes its draft to a citation-groundedness critique workflow in Agent Framework. frame_draft.generate feeds into frame_critique.generate under one operation, with gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and critique.concern_count: 0 showing that the critique stage ran without raising a concern.

I planted weak evidence twice, an invented "80% faster at another company, never verified" claim, to see if the agent would cite it. It did not. The draft agent cited the stronger evidence before the critique agent ran.

Also live: "Ask AI for card suggestions" on the Discovery Cards page. A second, independent agent (FoundryDiscoveryCardSuggestionAgent) reads the current persona and journey step and proposes AI-capability cards with a one-line rationale tied to that persona's actual pain point. Each is addable to the shortlist with one click.

Governance is part of the product​

Every model call and consequential action is checked against an allow-listed set of evaluation points and tools, in evaluation-only or enforce mode. Every decision is written to an append-only audit sink.

Decision review is where state actually becomes approved. A human evaluates trust and readiness, names blockers (if any), records a decision. It's the only place that flows into canonical state.

The audit trail lives in Application Insights alongside the regular telemetry: policy.gate_name, policy.mode, policy.result, and the full context of what the model saw and what it returned. You can trace from a delivery document back through the approval chain back to the original evidence.

Shipping on Azure​

The Container Apps API and Static Web Apps frontend deploy together from one azd environment. They use managed identity, with no connection strings in configuration. Bicep configures CORS, SignalR groups, and the Foundry connections.

I started on .NET 8 and moved to the .NET 11 preview partway through. It brought newer LINQ features and better performance on hot paths. It also meant waiting for a few NuGet packages tied to the release calendar before production.

I tested every feature in a browser against the deployed app as well as running unit tests. That found a client-side "Start an Ideation Round" error. Backend session creation succeeds, and Application Insights records 200 responses, but the client does not receive the message. I documented it as a separate follow-up because it does not block this workflow.

The output is evidence-first delivery​

The final artefact is a generated delivery document (a technical delivery brief) built from the approved opportunity record. It includes an AI-written summary explicitly labelled "NOT VERIFIED" with a note that a human should check it against the structured fields below.

It's the same drafts-never-decides contract right down to the last page. The AI suggests, the human approves.

Why this shape matters​

Most enterprise AI features follow a simple pattern: the AI acts and the human watches. Evidence to Decision makes the human create the record and decision, while the AI suggests patterns in the evidence. This takes more effort, but it leaves an auditable decision chain six months later.

For an AI-envisioning workshop tool, that record is the product. The customers funding this need a defensible way to turn workshops into delivery, not a faster sticky note app.

I built it on Azure because Foundry, Agent Framework, SignalR, Cosmos DB, and Entra ID provide those capabilities without custom infrastructure. The platform supports the policy gates, audit trail, and identity model this workflow needs.

For AI features that need to stay accountable, use the same sequence: evidence → recommendation → review → decision. Let the AI suggest. Keep approval and the record with people.

Links​

• Microsoft Foundry
• Microsoft Agent Framework
• Azure SignalR Service
• Azure Container Apps
• Azure Static Web Apps

I was wrong, and the way I was wrong is the useful part.

The trap​

You create an SLI on a service group, and you want it to alert. So you read the API. Microsoft.Monitor/slis at 2025-03-01-preview gives you exactly one alerting property:

"enableAlert": true

That is the whole surface. No threshold, no lookback window, no severity, and no action group. I checked the TypeSpec in azure-rest-api-specs, all seven files. I checked every API version the resource provider reports. There is only one. I checked whether a newer version existed. It does not.

Then I set enableAlert: true on a badly breaching SLI, with attainment at 79.9% against a 99% target and the error budget long since exhausted, and waited. No alert appeared in Alerts Management after fifteen minutes.

I was starting to think that the feature was portal-only and could not be expressed as code, which was weird as the portal is essentially an ARM (Azure Resource Manager) client. That conclusion was wrong.

What the portal actually does​

The portal's Baseline + Alert tab offers fast burn rate, slow burn rate, and action group selection. Since the SLI API has none of those fields, I captured what the portal sends when you select Save.

It updates the SLI, at the same API version I was already using, but only to flip enableAlert. Then it creates three ordinary Microsoft.Insights/metricAlerts resources in the workload resource group:

<sliName> baseline alert
<sliName> fast burn alert
<sliName> slow burn alert

Those resources use api-version=2024-03-01-preview, with odata.type: Microsoft.Azure.Monitor.PromQLCriteria. They scope the query to the Azure Monitor Workspace (targetResourceType: microsoft.monitor/accounts), not to the SLI.

enableAlert is a display flag. The alerts are the mechanism, and they are entirely ordinary ARM resources. Nothing private is involved. The undocumented part is the relationship between the two resource types.

The queries​

The alerts read the SLI's own emitted metrics. The SLI lists them under destinationMetrics as <sli>:Good, <sli>:Total, and <sli>:Value.

For a request-based SLI, the baseline rule checks attainment against the target:

(
  sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") ({"availability:good"})
  /
  sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") ({"availability:total"})
) * 100 < 99

The burn-rate rule divides the observed error ratio by the error budget:

(
  (
    sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") (increase({"availability:total"}[15m]))
    -
    sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") (increase({"availability:good"}[15m]))
  )
  /
  (
    sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") (increase({"availability:total"}[15m])) * (1 - 0.99)
  )
) > 14

Two details are easy to get wrong.

First, the correct function is increase(), not sum_over_time(). This corrects what I originally put in the post. The SLI publishes :good and :total as cumulative totals over the compliance window. They are not counters in the reset-on-restart sense, but they are monotonically increasing.

Early on, rate() against a freshly created SLI returned an empty result. I concluded from that single data point that the series was incompatible with the rate family of PromQL functions. It was not. The likely cause was too few samples. Rate-family functions need at least two samples spanning enough of the window to extrapolate from, and a new SLI cannot provide that.

After the series had been running for hours, rate() and increase() returned normal, consistent values. sum_over_time(x[15m]) sums fifteen cumulative snapshots of an ever-growing total. That gives you a lookback-averaged cumulative ratio, not the error rate during those fifteen minutes.

Measured live, a total outage held burn rate at about 3.5 against a 14x threshold for more than five minutes. A fault confined to one SLI partition took 49 minutes to cross the same threshold. With increase(), the replay crossed at T+5 minutes, and a fresh live fault fired the alert at 10m06s end to end. That includes the mandatory for: PT5M sustained-condition window after the metric crossed the threshold.

If a PromQL function returns an empty result, that means the query has insufficient data at that point. It does not mean the data shape is incompatible. Retest after the series has real volume.

Second, sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") is deliberate. The label does not exist, so the aggregation removes no real dimensions. A partitioned SLI still alerts per partition. It is the portal's own idiom, and copying it avoids changing the alert's scope accidentally.

The metric names also need quoted-selector syntax: {"availability:good"} rather than a bare identifier, because the names contain : and -.

Burnrate exists, but is not published​

Partway through the investigation I found Burnrate declared as a first-class sampling type in kqlmQueryResult.tsp, alongside Good, Total, Uptime, Downtime, and Value. That looked like the answer. The service computes burn rate itself, so surely there must be a metric to alert on.

There is not.

kqlmQueryResult.tsp describes the SLI query result model, not the metric contract. The live workspace publishes :good, :total, and :value for request-based SLIs, plus :uptime, :downtime, and :value for window-based SLIs. It does not publish :burnrate. Even with enableAlert: true, destinationMetrics lists only Good, Total, and Value.

The service computes burn rate internally and exposes it through its query surface, but does not publish it as a metric that an alert can read. The alert rules therefore calculate burn rate from :good and :total.

Four errors between here and a working deployment​

Each of these took a deployment cycle to find.

The template language function 'mul' expects its first parameter to be of type 'Integer'.

I was calculating the baseline threshold as json(sloTarget) * 100 to turn 0.99 into 99. ARM's mul is integer-only. Carry the percentage as its own parameter.

Query-based alert rule doesn't support 'global' location.

Classic metric alerts are global resources, so every example you have copied probably sets location: 'global'. A PromQL-based alert is regional. Set location to the workload's region.

Query-based metric alert rules need a managed identity. Make sure it's configured.

Add identity: { type: 'SystemAssigned' }.

Then, silently, nothing evaluates. That identity needs Monitoring Data Reader (b0d8363b-8ddd-447d-831f-62ca05bff136) on the Azure Monitor Workspace it queries. The rule deploys cleanly without the role assignment.

The Bicep​

This is the important part of the resource. The full implementation is in infra/modules/sli-native-alerts.bicep.

resource fastBurnAlert 'Microsoft.Insights/metricAlerts@2024-03-01-preview' = {
  name: '${sliName} fast burn alert'
  location: location          // NOT 'global'
  identity: {
    type: 'SystemAssigned'    // required, and needs Monitoring Data Reader on the AMW
  }
  properties: {
    enabled: true
    severity: 1
    targetResourceType: 'microsoft.monitor/accounts'
    scopes: [ azureMonitorWorkspaceResourceId ]
    evaluationFrequency: 'PT1M'
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.PromQLCriteria'
      allOf: [
        {
          name: 'SliAlertCriterion'
          query: burnRateQuery
          criterionType: 'StaticThresholdCriterion'
        }
      ]
      failingPeriods: { for: 'PT5M' }
    }
    actions: [ { actionGroupId: pageActionGroupResourceId } ]
    customProperties: {
      serviceGroupId: '/providers/Microsoft.Management/serviceGroups/<sg>'
      sliId: '/providers/Microsoft.Management/serviceGroups/<sg>/providers/Microsoft.Monitor/slis/${sliName}'
      alertKind: 'fast-burn-rate'
      burnRate: '14'
      lookback: '15m'
    }
  }
}

customProperties is not decoration. The portal uses that shape to reconstruct its Baseline + Alert UI from the alert resources. If you want alerts deployed from Bicep to render correctly in the SLI blade, reproduce the shape.

The complete relationship looks like this:

Why the native path matters​

Because I believed SLI alerting was not IaC-able, I built multi-window burn-rate alerting myself in Microsoft.AlertsManagement/prometheusRuleGroups, recomputing the error ratio from raw application counters.

That parallel implementation drifted in the dangerous direction.

My PromQL summed across all three service tiers, so one user request counted once per hop and inflated the denominator roughly threefold. It also left /healthz and /readyz in the denominator, even though those endpoints never fail. Both errors pushed the ratio down, making the alert less sensitive than the SLO it was meant to enforce.

Neither object looked wrong in isolation. The SLI was correct. The alert rule was syntactically valid and fired in testing, because the test fault was a total outage and a total outage trips almost anything. I found the problem by measuring detection latency before and after correcting the selectors. The same outage went from 9m47s to 5m20s.

Alerts derived from the SLI's own published output cannot drift from the SLO they enforce. There is no second implementation to disagree with. That is a stronger property than saving a few lines of Bicep.

What I'd tell Microsoft​

The capability is good. The burn-rate formula is right, the multi-window design is right, and the portal experience is clean.

The gap is that nothing points from the SLI resource to the alert resources. The enableAlert boolean actively misleads: it looks like the alerting switch, and setting it does nothing observable by itself.

A line in the SLI documentation saying that alert configuration is stored as Microsoft.Insights/metricAlerts resources with PromQLCriteria would have saved me a day and a wrong-headed workaround. The four deployment requirements would also make a useful quickstart:

• ARM mul expects an integer
• PromQL metric alerts need the workload's regional location
• PromQL metric alerts need a managed identity
• That identity needs Monitoring Data Reader on the workspace

The lesson I'd keep​

I read the spec, tested the one field it exposed, and concluded that a feature did not exist. The spec was accurate. My inference from it was not.

When a portal can do something an API apparently cannot, the portal is still an ARM client. Twenty seconds in the network tab would have told me what a day of reading TypeSpec did not.

Check what the product does before concluding what it cannot.

The full Bicep for this alerting layer, and the rest of the showcase it belongs to, is public in lukemurraynz/AzureSLI-AzureSREAgent.

Hopefully this article saves you the day I spent looking in the wrong resource type.

References​

• AzureSLI-AzureSREAgent repository
• Native SLI alert implementation
• AzureSLI-AzureSREAgent architecture and reliability chain
• AzureSLI-AzureSREAgent deployment and demo runbook
• Why SLOs are not just alerts

Getting the infrastructure up was the easy part. Getting a single SLI created against Managed Prometheus metrics took considerably longer, and almost none of the reasons were in the documentation. This post is the list of things I wish I had known before I started, in the order they bit me.

If you only read one line: the metric namespace for Managed Prometheus metrics is customdefault, not prometheus. That one value cost me the most time by a wide margin.

The setup​

A three-tier app on AKS, instrumented with the Prometheus client, scraped by Managed Prometheus into an Azure Monitor Workspace (AMW). The SLI reads http_requests_total and compares good requests (status code below 500) against total requests. Standard availability SLI, nothing exotic.

Worth knowing up front, because it shapes everything else:

• An SLI is an extension resource on a tenant-scoped service group, at /providers/Microsoft.Management/serviceGroups/{sg}/providers/Microsoft.Monitor/slis/{name}. That is not the scope an azd Bicep deployment targets, so in practice you apply SLIs with a direct ARM call rather than from your main template.
• The resource type is Microsoft.Monitor/slis, and at the time of writing the only published API version is 2025-03-01-preview, even though the SLI/SLO feature itself is GA.

Gotcha 1: the metric namespace is customdefault​

This is the big one. Every Microsoft example I could find uses an MDM style metric namespace, and the obvious guess for Managed Prometheus metrics is prometheus, especially since the generated query text shows metricNamespace("prometheus") quite happily when you supply it.

Here is the trap. With prometheus, the metric name resolves without complaint, so you get no "metric not found" error. What you get instead is one of these two, depending on whether you declared any dimensions:

QueryErrorCE2001: Name 'service' does not exist in current context

NoPartitioningDimension: Query metadata returned no partitioning dimension,
for account: mac_..., metricName: http_requests_total, metricNamespace: prometheus

Both of those point you at dimensions, which is where I spent hours. I checked the Prometheus endpoint and confirmed all 17 labels were present and queryable. I checked the portal, and the SLI blade happily listed service, namespace, status_code, path, and the rest in its dimension picker. Everything said the dimensions existed, and the API insisted they did not.

The dimensions were fine. The namespace was wrong.

{
  "signalSourceId": "A",
  "metricNamespace": "customdefault",
  "metricName": "http_requests_total",
  "sourceAmwAccountResourceId": "/subscriptions/.../providers/microsoft.monitor/accounts/<amw>",
  "sourceAmwAccountManagedIdentity": "/subscriptions/.../userAssignedIdentities/<identity>",
  "filters": [
    {
      "dimensionName": "status_code",
      "operator": "notstartswith",
      "value": "5"
    }
  ],
  "spatialAggregation": { "type": "Sum", "dimensions": ["service"] },
  "temporalAggregation": { "type": "Average" }
}

Change prometheus to customdefault and the same request that had been failing for hours returns provisioningState: Succeeded.

Note also that an SLI needs at least one partitioning dimension. An empty dimensions array fails with NoPartitioningDimension, so pick something meaningful (I use service).

Gotcha 2: the identity roles go on the DCR, not the workspace​

The documented prerequisites say the SLI identity needs Monitoring Reader on the source workspace, and Monitoring Reader plus Monitoring Metrics Publisher on the destination workspace's default data collection rule (DCR).

I read that, and assigned both roles on the Azure Monitor Workspace resource. That looks right, it passes review, and it fails at SLI creation with:

[DestinationAmwAccountAccessValidator] Access denied to target resource
/subscriptions/.../resourceGroups/MA_<amw>_<region>_managed/providers/Microsoft.Insights/dataCollectionRules/<amw>
for identity /subscriptions/.../userAssignedIdentities/<identity>

The default DCR is not in your resource group. It lives in the workspace's managed resource group, named MA_<amw-name>_<region>_managed, and it shares the workspace's name.

In Bicep you need a separate module scoped to that resource group:

module sliDcrRbac 'modules/sli-dcr-rbac.bicep' = {
  scope: az.resourceGroup('MA_${amwName}_${location}_managed')
  name: 'sli-dcr-rbac'
  params: {
    azureMonitorWorkspaceName: amwName
    sliIdentityPrincipalId: sliIdentity.outputs.principalId
  }
}

with the DCR referenced as existing inside that module, and role assignments scoped to it.

Gotcha 3: the preview call is useless for debugging, the create call is excellent​

There is a sliSignalPreview action that the portal's Validate button uses. It is tempting to gate your deployment on it, which is exactly what I did, and it was a mistake.

For essentially every failure, sliSignalPreview returns this:

{
  "code": "MalformedStructureError",
  "message": "The request properties payload is not in the correct format."
}

No detail, no field name, nothing. Meanwhile the actual create call (PUT .../slis/{name}) returns the generated query, the validator that rejected it, and the offending identifier by name. Same inputs, completely different diagnostic value.

So do not gate creation on a successful preview. Attempt the create and read what comes back. My script now runs the preview as advisory output only.

Gotcha 4: the real error is often in the portal notification pane​

SLI creation runs several validators in sequence, and each one masks the next. When it fails in the portal, the blade shows a generic validation banner and resets the wizard back to Basics, which tells you nothing.

The full error, including the validator name, shows up in the notification pane (the bell icon). The DestinationAmwAccountAccessValidator failure above was visible nowhere else: not in the blade, and not in the response to the preview call.

If both your API calls and the portal appear to fail identically, check the notifications before you conclude anything about the platform. I twice decided this was a platform limitation and twice I was wrong.

Gotcha 5: prerequisites that fail as generic errors​

Two more that produce contentless errors rather than saying what is missing.

The service group needs a default Azure Monitor Workspace. Until Monitor settings are configured on the service group, every SLI call fails. The portal path is service group, then Monitoring, then Monitor settings. Microsoft documents the subscription scoped association:

PUT https://management.azure.com/subscriptions/<sub>/providers/microsoft.monitor/settings/default?api-version=2025-06-03-preview
{ "properties": { "defaultAzureMonitorWorkspace": "<amw resource id>" } }

What is not documented, but works, is the same settings resource at service group scope:

PUT https://management.azure.com/providers/Microsoft.Management/serviceGroups/<sg>/providers/microsoft.monitor/settings/default?api-version=2025-06-03-preview

That one returns a proper resource with an etag, and it means you can keep the whole thing in code instead of clicking through the portal. Treat it as undocumented and re-check it on each new API version.

The metrics have to exist first. An SLI requires its input metric and its partitioning dimensions to already be present in Managed Prometheus at creation time. Create the SLI before the series materialises and validation fails. In a fresh environment that is a real wait: allow for the scrape interval plus two to three minutes of ingestion lag before you try. My deployment script polls the Prometheus endpoint for the metric name and only then creates the SLI.

Gotcha 6: enableAlert looks like the switch, and it lies​

At 2025-03-01-preview, the SLI resource exposes a single boolean, enableAlert. The fast burn and slow burn thresholds and the action group selection that the portal offers on the Baseline + Alert tab have no representation anywhere in the API surface, and there is no SLI specific alert type under Microsoft.AlertsManagement or Microsoft.Insights either. Read the API cold and the reasonable conclusion is that SLI alerting is portal-only, a boolean with nowhere for the real configuration to live.

I believed that for the better part of a day, and started building a parallel implementation in Microsoft.AlertsManagement/prometheusRuleGroups to work around it. Wrong. enableAlert is a display flag. The real configuration lives in ordinary Microsoft.Insights/metricAlerts resources, using Microsoft.Azure.Monitor.PromQLCriteria, that read the SLI's own published :good/:total metrics, and every bit of it deploys from Bicep. I cover the discovery, the queries, and the working resource definition in a companion post: Azure Monitor SLI alerting really is deployable as code, just not where you'd look.

The parallel prometheusRuleGroups implementation I built first is worth knowing about as a cautionary tale, not a recommendation. My selectors summed across all three service tiers and left health-check paths in the denominator, both of which quietly made the alert less sensitive than the SLO it was meant to enforce. Deriving alerts from the SLI's own output instead of recomputing them is the actual fix, and it's the pattern I'd ship.

Gotcha 7: the burn rate windows everybody quotes assume a 30 day budget​

This one is not an Azure quirk at all, which is exactly why it is worth writing down. It cost me more time than any of the API problems above, and it will be sitting in a lot of SLO implementations right now.

Every reference for multi window burn rate alerting gives you the same table: fast burn at 14.4x over 1h and 5m, slow burn at 6x over 6h and 30m. What almost nobody carries along with it is the precondition. Those windows assume a 30 day error budget.

My demo uses a 1-day compliance window, because on 30 days a fresh error budget barely moves and there is nothing to show. I copied the canonical windows in anyway.

The failure mode is not what you would expect. It is not that the alert is noisy, or that it fires late. It is that the long window is the rate limiter, and it makes partial faults invisible. A rolling 1h average starting from zero has to be physically dragged up to the threshold, and how long that takes depends on how big the fault is. Against a 7.2% threshold:

Read that table again, because the shape of it is the point. The alert fires promptly for a total outage and effectively never fires for a realistic partial degradation.

And here is why it survives review: the fault everyone tests with is a total outage. Kill the pod, watch the page arrive, tick the box. The configuration passes every test you throw at it while being unable to detect the degradations you are most likely to actually experience.

I only noticed because I wanted a better demo. A crashing pod is a weak SLO story, since Kubernetes already tells you about a crashing pod. What I wanted was the fault where every pod is Running with zero restarts and users are still failing, because that is the entire argument for SLIs on one screen. When I injected a 20% error rate, nothing fired. My first instinct was that the fault was too small. It was not, the window was too long.

Shortening the fast burn long window from 1h to 15m fixed it. Measured on the live environment: a 20% error rate now pages at Sev1 in 8 minutes 36 seconds, which is faster than the total outage managed under the old windows.

If you take one thing from this post, take this: derive your long window from your compliance window, and test with a partial fault. Compute long_window x threshold / expected_error_rate and ask whether that many minutes is acceptable for the smallest degradation you care about. Copying 1h/5m onto a 1-day budget is as wrong as copying it onto a 1-hour budget, and it fails silently in the direction of missing real incidents.

A related trap while you are testing: a long window keeps carrying the previous incident's errors well after recovery, so a slow burn rule can still be Fired from your last test run. I nearly recorded a stale Sev2 as a successful detection. Filter alerts by startDateTime against your injection time rather than trusting what is on the screen.

Gotcha 8: the signal model is much more capable than the examples suggest​

Every worked example I found uses a single signal source and a signalFormula of "A". That makes the model look thin. It is not, and I built three noticeably better SLIs once I read the TypeSpec instead of the examples.

signalFormula combines multiple sources. Give each source a distinct signalSourceId and reference them in the formula. Dividing a histogram's _sum by its _count gives you mean latency computed inside the SLI, with no recording rule and no change to your app:

"signals": {
  "signalFormula": "A / B",
  "signalSources": [
    { "signalSourceId": "A", "metricName": "http_request_duration_seconds_sum" },
    { "signalSourceId": "B", "metricName": "http_request_duration_seconds_count" }
  ]
}

in and notin take a ^^ delimited string, not a JSON array. This is documented only in a doc comment on the ConditionOperator union, and it is useful rather than trivia:

{
  "dimensionName": "service",
  "operator": "in",
  "value": "frontend^^api^^upstream-auth-service"
}

That is one SLI spanning a whole call chain instead of three separate ones. It matters for design, not just syntax. An edge only availability SLI stays green when something fails deep in the chain and the edge masks it with a cache or a fallback. A journey scoped one does not.

windowUptimeCriteria.target is a float, so sub-second thresholds work fine, and window-based SLIs are worth having alongside request-based ones rather than instead of them. The request-based one answers "what share of requests were good". The window-based one answers "how much of the time was the service bad", and it is the one that stays meaningful in a quiet period where a handful of slow requests can swing a ratio.

One caveat to close that loop, and it is the mistake I made. Because burn rate alerting lives in a parallel system rather than on the SLI resource, nothing keeps the two in step. I ended up with four SLIs, two availability and two latency, and burn rate rules for availability only. The latency SLIs charted perfectly and paged nobody. The list looks symmetrical, which is precisely why nobody catches it. Enumerate your SLIs and your alert rules side by side.

Gotcha 9: enabling the monitoring add-on does not enable monitoring​

This is the one I am most annoyed about, because I hit the same failure mode twice in the same build and still did not recognise it the second time.

Managed Prometheus first. Setting azureMonitorProfile.metrics.enabled = true in Bicep gets you the metrics agent and no metrics. You also need a data collection endpoint, a data collection rule, and an association to the cluster. I knew this, I built it, it worked.

Then Container Insights. omsAgentEnabled: true with the right workspace id. Addon reports enabled. The ama-logs pods run happily. And every single Container Insights table was empty for the entire life of the cluster:

union withsource=T *
| where TimeGenerated > ago(24h)
| summarize n=count() by T

No rows. Not "a bit thin", not "one table missing". Nothing, anywhere, ever. Same root cause as Prometheus: the enable flag provisions the agent, and a data collection rule associated to the cluster is what gives it a job. az aks enable-addons creates that rule for you. Bicep does not.

az monitor data-collection rule association list --resource <cluster-id> --query "[].name" -o tsv

Expect one association per collection path. I had exactly one, the Prometheus one, and no error anywhere in the portal, the CLI, the addon status, or the pod logs to suggest anything was missing.

Take the DCR shape and the stream list from the canonical onboarding template in microsoft/Docker-Provider rather than writing it from memory. The stream names are exact strings and there are twelve of them.

The knock-on effect is worth stating: my workbook had never had data, and my write-up claimed the SRE Agent "cites the AKS pod event as root-cause evidence". It cannot cite a pod event from an empty table. I had asserted that from the design rather than from an observation.

Gotcha 10: an empty panel and an impossible query look identical​

Once Container Insights was actually collecting, one workbook panel stayed empty. The query was fine, in the sense that it parsed and ran and returned zero rows:

InsightsMetrics
| where Namespace == 'container.azm.ms/memory'
| where Name == 'memoryWorkingSetBytes'

There is no container.azm.ms/memory namespace. InsightsMetrics carries node, disk and kube-state metrics. Container CPU and memory live in the Perf table under ObjectName == 'K8SContainer', and to get from there to a namespace you have to join back to KubePodInventory, because Perf.InstanceName is <clusterResourceId>/<podUID>/<containerName>:

let podMap = KubePodInventory
| where Namespace == 'payment'
| distinct ContainerName, Name;
Perf
| where ObjectName == 'K8SContainer' and CounterName == 'memoryWorkingSetBytes'
| extend ContainerName = strcat(tostring(split(InstanceName, '/')[-2]), '/', tostring(split(InstanceName, '/')[-1]))
| join kind=inner podMap on ContainerName
| summarize WorkingSetMiB = avg(CounterValue) / 1024 / 1024 by bin(TimeGenerated, 1m), Name

The lesson generalises well beyond workbooks: a query against the wrong table returns zero rows forever, and zero rows is indistinguishable from "no data yet" or "nothing is broken right now". Before you believe an empty panel, strip the filters and run a positive control:

InsightsMetrics | summarize count() by Namespace, Name | order by count_ desc

That one command would have shown me the real namespaces immediately, and it is the first thing I will run next time.

Gotcha 11: Service Group membership looks tenant-blocked, and is actually an api-version trap​

This one I got wrong, publicly and for weeks, so it is worth walking through properly.

Service groups are pitched as spanning resources across subscriptions, which is exactly what makes them attractive for a service-level SLO. I could not add a single member. The path, each step producing a different error:

1. PUT .../providers/Microsoft.Relationships/serviceGroupMember/<name> returns SubscriptionNotRegistered.
2. az provider register --namespace Microsoft.Relationships succeeds.
3. Retry: HttpRequestPayloadAPISpecValidationFailed. Two things buried in the details array: the name must match ^[a-zA-Z0-9]{3,64}$, so hyphens are rejected, and sourceId is required alongside targetId.
4. Retry with a valid payload: RelLifecycleNotEnabledForTenant, "Relationship lifecycle callbacks are not enabled for tenant."

Undocumented flag, no self-serve way to enable it. I recorded it as a hard limitation and designed around it.

It is not a limitation. Step 4 only happens on 2026-03-01-preview.

# fails: RelLifecycleNotEnabledForTenant
PUT .../serviceGroupMember/slisreShowcase?api-version=2026-03-01-preview
{"properties":{"sourceId":"/subscriptions/<sub>","targetId":"<sg>"}}

# succeeds
PUT .../serviceGroupMember/slisreShowcase?api-version=2023-09-01-preview
{"properties":{"targetId":"<sg>"}}

The older version infers sourceId from the parent scope and provisions in seconds. The newer one demands sourceId explicitly and then refuses on a tenant flag. So the newest preview is the more restrictive one, which is the opposite of the assumption that sent me down this path. I had reached for the newest api-version precisely because the older one seemed to be failing, and in doing so swapped a solvable error for an unsolvable-looking one.

What actually misled me is subtler than the version number. Errors 1, 3 and 4 are all real and all different, so each retry felt like progress toward a wall. A single error repeated would have made me question the request; a sequence of distinct, plausible errors reads like a narrowing path toward a wall.

And once it works, it is ordinary Bicep, because the relationship's parent is the member, not the service group:

targetScope = 'subscription'

resource serviceGroupMembership 'Microsoft.Relationships/serviceGroupMember@2023-09-01-preview' = {
  name: 'slisreShowcaseBicep'          // alphanumeric only
  properties: {
    targetId: '/providers/Microsoft.Management/serviceGroups/${serviceGroupId}'
  }
}

That is a subscription-scoped resource, so it deploys from the same template as everything else. Which produces an odd asymmetry worth knowing:

Same feature area, opposite answers, purely because of which end of the relationship owns the resource.

Does membership matter? Less than the effort suggests, and the reason is worth understanding. An SLI does not reference Azure resources at all: its scope is sourceAmwAccountResourceId plus metricNamespace, metricName, and dimension filters. My journey-availability SLI spans three tiers purely because of service in "frontend^^api^^upstream-auth-service". Membership gives you topology and rollup; it changes no SLI's computation.

What it does buy is letting something else tell you what composes the service, rather than you asserting it. That matters for the SRE Agent, which scopes its knowledge graph to a resource group: if a service actually spans resource groups, the agent investigates one and silently misses the rest.

The lesson, and it is the same one as the alerting post: when an Azure preview API refuses, check whether an older api-version accepts. Newer previews tighten validation and add gates, and the resource provider will happily tell you which versions exist: the error at the wrong version named all of them, and I did not read it.

Gotcha 12: GET returns shapes that PUT rejects​

Round-tripping a resource (GET it, change one field, PUT it back) is the most natural thing to try, and it fails in several places across this stack:

• SLI: the identity.userAssignedIdentities values from GET are rejected on PUT with InvalidIdentityValues. They must be empty objects ({}).
• SRE Agent response plans: GET returns isDeleted, documentType, partitionKey, createdAt, updatedAt; POST rejects all five with Unknown incident filter properties.
• SRE Agent skills and subagents: GET returns array fields as stringified lists ("tools": "['RunAzCliReadCommands']"), while PUT requires real JSON arrays. Sending back what you received yields 400 ... could not be converted to System.Collections.Generic.List.

Strip server-generated fields and rebuild write payloads from the documented shape rather than from a GET response.

Two SLI correctness traps while you are here​

Neither of these is an Azure quirk, they are just easy to get wrong, and both quietly produce an SLI that measures the wrong thing.

Health probe traffic. Your scraped series includes /healthz and /readyz. Those never fail, so leaving them in the denominator dilutes the error ratio and your SLO will not move the way you expect. Filter to the user journey path.

Multi-tier double counting. If every tier emits the same metric name, one user request is counted once per hop. Filter to the edge service, otherwise a three-tier app inflates the denominator threefold.

What actually found the answers​

Worth saying, because the method matters more than the specific values.

Searching the docs and Learn got me nowhere on the namespace. What found it was searching GitHub code for the literal error string, DestinationAmwAccountAccessValidator. That returned exactly one result, a workarounds.md in Toru Makabe's aks-chaos-lab, which documents both the DCR RBAC requirement and the metrics-must-exist-first constraint, and whose SLI Bicep module has metricNamespace defaulting to customdefault. Full credit there, that repo saved me a support case.

If you are stuck on an Azure error that reads like an internal validator name, put the exact string into GitHub code search before you do anything else. Somebody has usually hit it and written it down.

The other thing that paid off was reading the TypeSpec rather than the examples. sliProperties.tsp in Azure/azure-rest-api-specs is about 260 lines and took five minutes, and it is where the ^^ delimiter, the float target, and the multi source formula all are. The examples had led me to believe none of those existed.

And one anti method, since it wasted an hour. When I could not find an API to list the SRE Agent's incidents, I started guessing endpoint paths. Three of them returned HTTP 200, which felt like progress, right up until I looked at the body and found the portal's single page app HTML rather than JSON. A 200 from a SPA host means "this host serves a web app", not "this endpoint exists". If you are guessing paths, you have already left the part of the problem that is solvable by guessing.

What actually worked was asking what the service emits rather than what it exposes. The agent resource has a logConfiguration.applicationInsightsConfiguration.appId, and the agent writes an IncidentActivitySnapshot custom event to it on every incident transition, carrying the response plan id, the autonomy level, the handled timestamp, and the agent's written investigation summary. That answered in one KQL query what an hour of path guessing could not. When an Azure service will not give you an API, check its telemetry before you reach for its UI.

Wrap up​

None of this makes Azure Monitor SLIs a bad feature. The error budget and burn rate experience is useful, and having it native beats maintaining your own burn rate rules. But the preview API surface is sharp in places, and the failure modes are almost all generic errors that point away from the real cause.

The short version for anyone starting out:

• Use customdefault as the metric namespace for Managed Prometheus metrics.
• Put the identity roles on the default DCR in MA_<amw>_<region>_managed, not on the workspace.
• Debug with the create call, not sliSignalPreview.
• Check the portal notification pane for the real error.
• Configure the service group default AMW first, and wait for your metrics to land.
• Don't recompute burn rate alerting yourself. Derive it from the SLI's own :good/:total metrics via Microsoft.Insights/metricAlerts and PromQLCriteria, it's IaC-able, just not where the SLI resource itself suggests.

Hopefully this saves you the afternoon it cost me.

The full Bicep, scripts, and agent configuration for the showcase this was all found while building are public: lukemurraynz/AzureSLI-AzureSREAgent.

References​

• Create service level indicators in Azure Monitor
• Microsoft.Monitor/slis template reference
• Use Azure Monitor issues (default AMW association)
• Service groups overview
• torumakabe/aks-chaos-lab
• lukemurraynz/AzureSLI-AzureSREAgent

That gap is a product and service-design problem as much as a monitoring problem. The people responsible for a payment journey need to know whether checkout works, how quickly it works, and which customer segment is paying the cost of an incident. The on-call engineer needs enough evidence to choose the right first action. The business needs a trustworthy way to decide when an error budget has been spent.

I wanted to see whether an Azure Monitor SLI could give the Azure SRE Agent enough context to investigate an Azure Kubernetes Service failure, rather than just react to a pod alert. I built a small payment service, chose signals around the customer journey, and deliberately broke it.

The result is an infrastructure-as-code deployment that runs with azd up. It measures availability, latency, the full request journey, the public path, and customer tiers. Azure Monitor alerts on those service-level signals, and the Azure SRE Agent investigates the incident and, when allowed, fixes it.

The important result is not the number of Azure resources. It is the chain from customer need to operational decision:

I got several things wrong. The corrections stay in the post because they expose the difference between a design that looks plausible and one that has been tested with the people, decisions, and failure modes it is meant to serve.

Who this is for​

This walkthrough is aimed at teams that share responsibility for a customer-facing service:

• Platform engineers who own Azure Monitor workspaces, identities, collection, and RBAC
• SRE and operations teams who define SLOs, error-budget policy, and incident response
• Product and engineering leaders who need reliability data expressed in terms of customer impact
• Teams evaluating how much operational work an AI agent should investigate, propose, or perform

Start with the service promise​

The arrow from slis to alerts is the design decision I cared about. The alerts read the SLI's own published metrics, not a second implementation built from raw application counters. That keeps the alert and the SLO measuring the same customer promise.

This is the first design-thinking checkpoint: define the outcome before choosing the tool. “A pod is ready” is an implementation fact. “A customer can complete checkout within 500 milliseconds” is an outcome. Both are useful, but they answer different questions and should not be treated as interchangeable.

Give the service a durable identity​

An SLI in Azure Monitor is an extension resource on a tenant-scoped Service Group:

/providers/Microsoft.Management/serviceGroups/<sg>/providers/Microsoft.Monitor/slis/<name>

That scope is the point. A service is rarely one resource group: frontend here, API there, and a database in a shared subscription. A tenant-scoped Service Group gives the service an identity that outlives any particular resource layout, so the SLO follows the customer experience rather than a cluster.

For a platform team, this reduces the cost of change. Teams can move workloads, split deployments, or replace a dependency without making the reliability conversation start again. For a product team, it creates a stable place to discuss whether the service is meeting its promise.

There is an important boundary here: a Service Group is an operational view, not a security boundary. It is currently in public preview, and membership does not grant access to the resources inside it. A resource can belong to multiple Service Groups, which is useful when platform, product, and customer-support teams need different views of the same service. Keep resource RBAC and Service Group RBAC as separate design decisions.

The Service Group does not define what the SLI measures. There is no membership involved. An SLI's scope is sourceAmwAccountResourceId (Azure Monitor Workspace) plus a metric name and dimension filters:

"filters": [
  { "dimensionName": "namespace",   "operator": "eq",            "value": "payment" },
  { "dimensionName": "service",     "operator": "eq",            "value": "frontend" },
  { "dimensionName": "path",        "operator": "eq",            "value": "/checkout" },
  { "dimensionName": "status_code", "operator": "notstartswith", "value": "5" }
]

The Service Group is the SLO's home, not its definition. Membership gives you topology and rollup, but changes nothing about what any SLI computes.

Plan the SLI identity path​

The SLI also has an identity and data-storage path that is easy to miss when the portal hides the plumbing. Azure Monitor reads the source Azure Monitor Workspace and writes the evaluated SLI results to a destination workspace. A user-assigned managed identity needs Monitoring Reader on the source workspace, Monitoring Reader plus Monitoring Metrics Publisher on the destination workspace, and Monitoring Reader on the destination workspace's default data collection rule. The source and destination can be the same workspace, but separating raw telemetry from evaluated reliability data can make ownership and access easier to explain.

This is a useful handoff between platform and product teams. The product team defines what “good” means. The platform team owns the identity, workspace boundaries, collection rules, and retention that make that definition executable.

Membership is ordinary Bicep, with one sharp edge. 2026-03-01-preview returned RelLifecycleNotEnabledForTenant; the older 2023-09-01-preview worked:

targetScope = 'subscription'

resource serviceGroupMembership 'Microsoft.Relationships/serviceGroupMember@2023-09-01-preview' = {
  name: 'slisreShowcaseBicep' // alphanumeric only, 3-64 chars
  properties: {
    targetId: '/providers/Microsoft.Management/serviceGroups/${serviceGroupId}'
  }
}

The relationship's parent is the member, not the Service Group, so this is subscription-scoped. The SLIs are the opposite case: their parent is the Service Group, which forces a tenant-scoped deployment and an Azure RBAC grant at / that Global Administrator does not confer.

You can express "one SLO across frontend, API and auth" today with one multi-value filter:

{
  "dimensionName": "service",
  "operator": "in",
  "value": "frontend^^api^^upstream-auth-service",
}

in and notin take a ^^-delimited string, not a JSON array. That's documented in one TypeSpec comment and nowhere else.

Turn customer needs into measurable questions​

Each SLI exists because the basic availability number leaves a different customer or business question unanswered. A sixth, latency-windowed, measures the same 500ms bar as mean latency per five-minute window rather than per request, and is covered separately below.

Azure Monitor offers two evaluation methods. Request-based evaluation asks whether the ratio of good requests to total requests meets the target. Window-based evaluation asks whether time intervals meet a quality threshold. Request-based evaluation is usually the better fit for checkout because one failed transaction should count as a failed transaction, even during a quiet period. Window-based evaluation can be useful when you want to smooth short bursts or measure the proportion of time a system remains within a limit. Choose the method with the customer experience and decision in mind, not the metric that happens to be easiest to query.

availability measures the edge at /checkout: good requests over total. journey-availability spans all three tiers with the in filter. An edge-only SLI stays green when the API fails and the frontend serves a cached fallback, while the user's journey is broken.

latency exists because a service that answers every request in nine seconds is up, and useless. Good is the le="0.5" histogram bucket; total is the request count.

external-availability matters because every SLI above is computed from traffic generated inside the cluster. You cannot measure an outage from inside the thing that is unreachable. If DNS breaks or the load balancer misroutes, no request arrives, so there is no error metric and no denominator either. Availability reads 100% during a total outage. That is structural; it needs something outside the workload requesting the service over the real network path.

tier-availability is the same measurement as availability, partitioned by who the customer is rather than by which component served them.

For any SLO, ask which signal moves if DNS breaks. If none does, it measures instrumentation rather than availability. Then ask who needs to act on the signal and what decision it should change. That turns a dashboard metric into a useful service conversation.

Choose signals that represent people, not components​

The demo app is instrumented with the Prometheus client because I wrote it. Most workloads you need SLOs for are not yours to change, and that's fine: the SLI reads Prometheus series from an Azure Monitor Workspace and does not care what produced them.

Prefer an infrastructure-emitted signal even when you can instrument the app. It survives redeploys, language changes and vendor upgrades, and is uniform across every service behind it.

One trap cost me hours: for Managed Prometheus sources, metricNamespace must be customdefault, not prometheus. With the wrong value the metric resolves fine and SLI creation fails with errors about dimensions, which is entirely the wrong place to look.

Measuring users, not components​

An SLI's scope is a dimension filter, not a resource. journey-availability spans three Kubernetes Deployments purely because of:

{
  "dimensionName": "service",
  "operator": "in",
  "value": "frontend^^api^^upstream-auth-service"
}

Add a fourth tier tomorrow and it joins the SLO by emitting the label. Nothing gets registered. The SLO describes the journey; the deployment topology is free to change underneath it.

The partition trick​

The dimensions do not have to be technical. This app emits customer_tier from a request header and propagates it across every hop:

KNOWN_TIERS = frozenset({"premium", "standard", "free"})
DEFAULT_TIER = "standard"

tier = request.headers.get("x-customer-tier", DEFAULT_TIER).lower()
if tier not in KNOWN_TIERS:      # never let a caller mint unbounded series
    tier = DEFAULT_TIER

The SLI partitions on it via spatialAggregation.dimensions:

"spatialAggregation": { "type": "Sum", "dimensions": ["customer_tier"] }

The alert uses the portal's own idiom:

sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") ({__name__="...:good"})

Summing without a label that does not exist aggregates while preserving every real dimension, so one alert rule fires once per partition. One SLI, one rule, and the incident names the affected segment.

Break premium only:

./scripts/inject-fault.sh --mode=errors --rate=100 --tier=premium

The second has already eliminated most of the search space: not a capacity problem, not a dependency outage, and not a deployment affecting everyone. Something is routing premium differently. That is a fault class per-service telemetry cannot see at all: every pod is healthy, every tier is up, and only the signal partitioned by who the user is moves.

This is where the business lens changes the response. A blended 85.02% can trigger concern, but it does not tell a product manager whether premium customers, free customers, or everyone is affected. The partitioned result supports prioritisation, customer communication, and a more honest conversation about differentiated service commitments.

Imagine a premium customer trying to complete checkout during this fault. The pods are healthy and the overall availability number is still above zero, but that customer cannot complete the transaction. The tier-specific SLI turns that experience into an operational fact: investigate the premium path first, communicate the impact accurately, and avoid spending the incident response effort on healthy free and standard traffic.

There are four limits:

1. Azure cannot infer business dimensions. customer_tier exists because the application chose to emit it.
2. Cardinality is the tax. Keep business dimensions closed and small; tenant IDs are usually a bill, not a feature.
3. Targets are per SLI, not per partition. Differentiated targets need one SLI per tier.
4. In-cluster signals stay blind to the front door. Partitioning does not fix what external-availability exists to fix.

The Service Group gives the service a tenant-scoped identity that survives resource churn. Through customProperties.sliId, the agent gets a service-level fact about users rather than a symptom about pods.

Service Groups nest too. A child group joins a parent via properties.parent.resourceId at create time. The differentiated-target SLIs now live in a Customer Segments child group under the parent, without moving the existing six SLIs or their history.

Make the alert answer a business question​

Azure Monitor has a native SLI alerting path. In the portal, enableAlert can configure a baseline alert, fast-burn alert, and slow-burn alert, with an action group defining who is notified and what downstream action runs. That is the shortest path when you want the platform's standard SLI experience.

I used a second path in this project: explicit Microsoft.Insights/metricAlerts resources using Microsoft.Azure.Monitor.PromQLCriteria, scoped to the Azure Monitor Workspace. That gave me infrastructure-as-code control over the PromQL, dimensions, custom properties, and alert names used by the SRE Agent. It is not a replacement for native SLI alerting. Decide which path owns the policy, and avoid enabling both for the same condition unless duplicate notifications are intentional.

They read the SLI's own published metrics, <sli>:Good, <sli>:Total, and <sli>:Value:

(
  ( sum(increase({__name__="ns::<sg>/m::availability:total"}[15m]))
  - sum(increase({__name__="ns::<sg>/m::availability:good"}[15m])) )
  /
  ( sum(increase({__name__="ns::<sg>/m::availability:total"}[15m])) * (1 - 0.99) )
) > 14

I originally recomputed the same signal in parallel Prometheus rules. Those selectors summed across all three tiers and left /healthz in the denominator, making the alert materially less sensitive than the SLO. Both objects looked correct in isolation, and total-outage testing did not catch it. Detection latency was 9m47s versus 5m20s on the same outage.

Alerts derived from the SLI's own output cannot drift from it. That safety property is the technical argument. The business argument is that the alert represents the same commitment people agreed to measure, rather than an engineer's approximation of it.

Use error budgets to guide attention​

Burn rate is how fast you're spending error budget. At a 99% target, a 2% error rate burns at 2x; a 20% error rate burns at 20x. That makes an error budget a decision tool: it connects reliability work to the limited amount of failure the service can afford before customers and the business feel the impact.

The canonical 14x over one hour and 6x over six hours assumes a 30-day budget. On a one-day window with a 0.5% budget, the long window becomes the rate limiter:

Derive the long window from the compliance window. Shortening it to 15 minutes took a 20% error rate from roughly 22 minutes to a measured 8m36s end to end.

A correction: sum_over_time() was wrong here​

The SLI's :good and :total are cumulative across the compliance window, not counters. sum_over_time(...[15m]) therefore sums fifteen cumulative snapshots. It is accumulated damage, not the error rate during those fifteen minutes.

The practical consequence was a complete backend outage that ran for twelve minutes without crossing the 14x threshold. Treat the baseline attainment alert as the fast detector; keep burn-rate alerts for the budget-spend signal. If true short-window sensitivity is required, compute it from raw application counters with rate(), accepting the drift risk this design otherwise avoids.

A second correction: rate() was not the problem​

That conclusion was half wrong. sum_over_time()/rate() returning empty was true of exactly one test: a brand-new SLI queried within its first evaluation cycles. rate() and increase() need at least two samples spanning the lookback window. After a day of samples, rate(), increase() and delta() all returned normal values.

On identical 15-minute fault injections, sum_over_time() took 49 minutes to trip; increase() took 10m06s, most of which was the mandatory five-minute sustained-condition window. Use increase(), not sum_over_time(), and interpret an empty PromQL result as "not enough history yet" until proven otherwise.

Design the handoff from signal to action​

The agent subscribes to Azure Monitor directly through incidentManagementConfiguration.type: AzMonitor, scoped by knowledgeGraphConfiguration.managedResources to the resource group. It is not wired through an action group. That is a different integration path from native SLI alert notifications: the SRE Agent receives the Azure Monitor incident, while an action group can notify people or invoke other automation.

Each alert carries structural context:

{
  "sliId": "/providers/.../serviceGroups/<sg>/providers/Microsoft.Monitor/slis/availability",
  "serviceGroupId": "/providers/Microsoft.Management/serviceGroups/<sg>",
  "alertKind": "fast-burn-rate",
  "burnRate": "14",
  "lookback": "15m",
}

The agent can resolve sliId and learn the target, compliance window, and filters. The incident opens as “the /checkout journey at the frontend edge, committed at 99% over a one-day rolling window, is at 93.4%,” not “a burn-rate rule exceeded 6”. The first version gives an operator or an agent a user-centred problem to investigate.

The handoff also follows a useful service-design pattern: preserve context as the problem moves between people and systems. The alert carries the service, journey, target, customer segment, and time window. The agent can then investigate the likely experience before choosing an implementation-level action.

Correlation is the fastest diagnosis​

Given a Deployment scaled to zero, the agent ran 31+ tool calls, queried Log Analytics, correctly root-caused the missing pods, and scaled the Deployment back up. Injection to recovery was about 20 minutes, roughly five of them detection.

The current SRE Agent guidance recommends starting with Reader access, where writes require approval, and moving to Privileged access only after the team trusts the workflow. That is a good pilot model: begin with investigation and proposed remediation, measure false positives and recovery quality, then grant the smallest write permission for a narrow incident class. The agent also creates or uses Application Insights and persists investigation context, so data retention, cost, region, and who can review that operational memory belong in the service design as well. The SRE Agent security overview describes the related identity, isolation, and telemetry considerations.

On a later run it refused to declare success based only on rollout status: "the rollout completed, but the probe and workspace telemetry do not yet provide a current success sample." kubectl confirmed the outcome it was checking for: 2/2 ready.

Treat trust as part of the service design​

The agent needs Kubernetes access to act, but permission to act is not the same as evidence that it should act. Scope the permissions tightly:

Writer rather than Admin excludes role bindings, so the agent cannot grant itself anything further.

PreToolUse hooks gate individual actions, but only if their matchers bind. Mine shipped with ^(restart_|scale_).*, copied from another framework. The real tool names are PascalCase: RunKubectlWriteCommand, Terminal, and RunInTerminal. The hooks matched zero tools while reporting HooksRun: 2, FinalDecision: pass.

When the agent remediated, it used RunInTerminal, not the obvious kubectl-specific tool. A matcher listing only write tools would have missed it.

A response plan with agentMode: review also did not prevent execution in my test. That contradicted the current product guidance, which describes Review mode as requiring approval for write operations, so I treated it as a behaviour to retest rather than a safety guarantee.

From a design-thinking perspective, the question is not “can the AI fix the incident?” It is “what level of autonomy is appropriate for this incident, this customer impact, and this reversibility?” A scale-up in one namespace may be a bounded experiment. A database migration or a change affecting every tenant is a different service risk and needs a different approval model.

The re-test showed why the underlying control matters​

I wrote that Review was the one control that prevented writes after testing it once. It was not. Repeating the scaled-to-zero fault twice showed the supervised agent, ARM mode: Review, performing the scale operation without approval.

The agent's incident telemetry said IncidentMitigatedByAgent: False both times. Kubernetes audit logs were the only reliable witness: kube-audit-admin named the principal that executed the PATCH .../scale request. Treat that discrepancy as release-specific behaviour to verify in your tenant, not as a supported guarantee either way.

The cause was a hook named require-approval-for-restarts whose content effectively said: deny unless the action is confined to the namespace, reversible, evidence-backed, and not a delete. It did not check whether a human had approved anything. The agent was grading its own homework.

I rewrote the hook to unconditionally deny and re-tested with a fresh fault injection. The autonomous agent performed the fix; the supervised agent investigated and touched nothing. One clean run is evidence, not a guarantee, so keep testing the layer underneath the telemetry.

Test the experience, not only the configuration​

• An enabled monitoring addon does not enable collection. omsAgent.enabled deploys the agent; a data collection rule makes it collect. Without the DCR, Container Insights is empty and nothing reports an error. Managed Prometheus has the same DCE/DCR/DCRA trap.
• Absent data makes burn-rate alerts stop firing. A zero denominator produces NaN, and NaN > threshold is false. Alert explicitly on absence with absent().
• Incident titles reach the agent URL-encoded. availability baseline alert becomes availability%20baseline%20alert, so a plan matching spaces never matches. Name alerts without spaces.
• Overlapping response plans disable the agent. Several matching plans created incidents that were marked handled within a minute while running zero tools. One matching plan produced 31 tool calls and a successful remediation.

A query returning zero rows and a query that can never work are indistinguishable. Establish a positive control before believing an empty result. More broadly, test the experience you intend to protect: inject a fault, observe the customer-facing signal, check the alert context, and verify the action or refusal in the underlying audit trail.

Make reliability legible to decision-makers​

The platform engineer wanted a single pane connecting infrastructure state to user reliability. For most of this project that was two panes: pod health in Log Analytics with KQL, and SLI evaluations in the Azure Monitor Workspace with PromQL.

Workbooks support Prometheus as a data source. The JSON shape is undocumented, but the portal's own workbook uses this form:

{
  "queryType": 16,
  "resourceType": "microsoft.monitor/accounts",
  "crossComponentResources": ["<azure monitor workspace resource id>"],
  "query": "{\"version\":\"PrometheusQueryProvider/1.0\",\"queryText\":\"slo:error_budget:remaining_ratio\",\"type\":\"query_range\"}",
}

The composite panel uses the minimum attainment across every SLI rather than an average. A premium-tier outage should read “we are breaching our worst commitment”, not disappear inside a blended number.

That panel also closes the reporting gap for the engineering manager, who needs credible error-budget reporting but is least likely to browse nine SLI charts. Alerting on a number is not the same as being able to see its trend, and a trend without an agreed decision is not a useful outcome either.

What it still does not do​

• Service Groups are still in public preview. Do not make a production security or compliance boundary depend on Service Group membership, and confirm the current preview terms before adopting the pattern broadly.
• An SLI's definition is portable, but its execution is not permission-free. The managed identity, source and destination workspaces, default data collection rule, and metric collection path all need to be healthy.
• The in-cluster SLIs still measure in-cluster traffic. external-availability covers the public path; the rest are blind to ingress and DNS by construction.
• The tier SLA table is a draft, not a signed contract. Differentiated targets exist and alert independently, but nothing enforces the SLA outside that policy document.
• The one-day compliance window is a demo choice so the budget visibly moves. Never copy evaluationPeriodDays: 1 into production without an explicit decision.
• Native SLI alerts and explicit PromQL metric alerts are two policy paths. Choose one owner for each condition, or document why duplicate alert routes are useful.
• SLOs do not detect outages faster than a probe. On the same outage, the external web test fired in 3m37s and the burn-rate alert in roughly five minutes. SLOs earn their place in the ambiguous middle: deciding whether a 2% error rate is worth waking someone for.

A reliability design checklist​

Start with the user's task and the business promise. Measure the user, not the application. Ask what moves if DNS breaks. If none of the signals move, you have instrumentation, not an SLO.

Choose dimensions that support a decision. Customer tier, journey, region, and tenant can explain impact, but uncontrolled cardinality can make the signal expensive and noisy.

Derive alerts from the SLI, never alongside it. A parallel implementation will drift silently toward being less sensitive than the SLO it enforces, and it will pass testing because total outages trip almost anything.

Design the handoff for the person or agent who must act. Carry the service identity, customer impact, target, and evidence into the incident instead of exposing only a rule name.

Verify guardrails by trying to breach them. Hooks that match nothing, plan modes that do not supervise, and rules that never evaluate all look identical to ones that work. The difference is visible only when you inject a fault and watch what happens.

Finally, treat reliability as a learning loop. Observe the customer experience, interpret the evidence with the people who own the outcome, test the smallest safe intervention, and update the design when reality disagrees with the model.