I was wrong, and the way I was wrong is the useful part.

The trap​

You create an SLI on a service group, and you want it to alert. So you read the API. Microsoft.Monitor/slis at 2025-03-01-preview gives you exactly one alerting property:

"enableAlert": true

That is the whole surface. No threshold, no lookback window, no severity, and no action group. I checked the TypeSpec in azure-rest-api-specs, all seven files. I checked every API version the resource provider reports. There is only one. I checked whether a newer version existed. It does not.

Then I set enableAlert: true on a badly breaching SLI, with attainment at 79.9% against a 99% target and the error budget long since exhausted, and waited. No alert appeared in Alerts Management after fifteen minutes.

I was starting to think that the feature was portal-only and could not be expressed as code, which was weird as the portal is essentially an ARM (Azure Resource Manager) client. That conclusion was wrong.

What the portal actually does​

The portal's Baseline + Alert tab offers fast burn rate, slow burn rate, and action group selection. Since the SLI API has none of those fields, I captured what the portal sends when you select Save.

It updates the SLI, at the same API version I was already using, but only to flip enableAlert. Then it creates three ordinary Microsoft.Insights/metricAlerts resources in the workload resource group:

<sliName> baseline alert
<sliName> fast burn alert
<sliName> slow burn alert

Those resources use api-version=2024-03-01-preview, with odata.type: Microsoft.Azure.Monitor.PromQLCriteria. They scope the query to the Azure Monitor Workspace (targetResourceType: microsoft.monitor/accounts), not to the SLI.

enableAlert is a display flag. The alerts are the mechanism, and they are entirely ordinary ARM resources. Nothing private is involved. The undocumented part is the relationship between the two resource types.

The queries​

The alerts read the SLI's own emitted metrics. The SLI lists them under destinationMetrics as <sli>:Good, <sli>:Total, and <sli>:Value.

For a request-based SLI, the baseline rule checks attainment against the target:

(
  sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") ({"availability:good"})
  /
  sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") ({"availability:total"})
) * 100 < 99

The burn-rate rule divides the observed error ratio by the error budget:

(
  (
    sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") (increase({"availability:total"}[15m]))
    -
    sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") (increase({"availability:good"}[15m]))
  )
  /
  (
    sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") (increase({"availability:total"}[15m])) * (1 - 0.99)
  )
) > 14

Two details are easy to get wrong.

First, the correct function is increase(), not sum_over_time(). This corrects what I originally put in the post. The SLI publishes :good and :total as cumulative totals over the compliance window. They are not counters in the reset-on-restart sense, but they are monotonically increasing.

Early on, rate() against a freshly created SLI returned an empty result. I concluded from that single data point that the series was incompatible with the rate family of PromQL functions. It was not. The likely cause was too few samples. Rate-family functions need at least two samples spanning enough of the window to extrapolate from, and a new SLI cannot provide that.

After the series had been running for hours, rate() and increase() returned normal, consistent values. sum_over_time(x[15m]) sums fifteen cumulative snapshots of an ever-growing total. That gives you a lookback-averaged cumulative ratio, not the error rate during those fifteen minutes.

Measured live, a total outage held burn rate at about 3.5 against a 14x threshold for more than five minutes. A fault confined to one SLI partition took 49 minutes to cross the same threshold. With increase(), the replay crossed at T+5 minutes, and a fresh live fault fired the alert at 10m06s end to end. That includes the mandatory for: PT5M sustained-condition window after the metric crossed the threshold.

If a PromQL function returns an empty result, that means the query has insufficient data at that point. It does not mean the data shape is incompatible. Retest after the series has real volume.

Second, sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") is deliberate. The label does not exist, so the aggregation removes no real dimensions. A partitioned SLI still alerts per partition. It is the portal's own idiom, and copying it avoids changing the alert's scope accidentally.

The metric names also need quoted-selector syntax: {"availability:good"} rather than a bare identifier, because the names contain : and -.

Burnrate exists, but is not published​

Partway through the investigation I found Burnrate declared as a first-class sampling type in kqlmQueryResult.tsp, alongside Good, Total, Uptime, Downtime, and Value. That looked like the answer. The service computes burn rate itself, so surely there must be a metric to alert on.

There is not.

kqlmQueryResult.tsp describes the SLI query result model, not the metric contract. The live workspace publishes :good, :total, and :value for request-based SLIs, plus :uptime, :downtime, and :value for window-based SLIs. It does not publish :burnrate. Even with enableAlert: true, destinationMetrics lists only Good, Total, and Value.

The service computes burn rate internally and exposes it through its query surface, but does not publish it as a metric that an alert can read. The alert rules therefore calculate burn rate from :good and :total.

Four errors between here and a working deployment​

Each of these took a deployment cycle to find.

The template language function 'mul' expects its first parameter to be of type 'Integer'.

I was calculating the baseline threshold as json(sloTarget) * 100 to turn 0.99 into 99. ARM's mul is integer-only. Carry the percentage as its own parameter.

Query-based alert rule doesn't support 'global' location.

Classic metric alerts are global resources, so every example you have copied probably sets location: 'global'. A PromQL-based alert is regional. Set location to the workload's region.

Query-based metric alert rules need a managed identity. Make sure it's configured.

Add identity: { type: 'SystemAssigned' }.

Then, silently, nothing evaluates. That identity needs Monitoring Data Reader (b0d8363b-8ddd-447d-831f-62ca05bff136) on the Azure Monitor Workspace it queries. The rule deploys cleanly without the role assignment.

The Bicep​

This is the important part of the resource. The full implementation is in infra/modules/sli-native-alerts.bicep.

resource fastBurnAlert 'Microsoft.Insights/metricAlerts@2024-03-01-preview' = {
  name: '${sliName} fast burn alert'
  location: location          // NOT 'global'
  identity: {
    type: 'SystemAssigned'    // required, and needs Monitoring Data Reader on the AMW
  }
  properties: {
    enabled: true
    severity: 1
    targetResourceType: 'microsoft.monitor/accounts'
    scopes: [ azureMonitorWorkspaceResourceId ]
    evaluationFrequency: 'PT1M'
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.PromQLCriteria'
      allOf: [
        {
          name: 'SliAlertCriterion'
          query: burnRateQuery
          criterionType: 'StaticThresholdCriterion'
        }
      ]
      failingPeriods: { for: 'PT5M' }
    }
    actions: [ { actionGroupId: pageActionGroupResourceId } ]
    customProperties: {
      serviceGroupId: '/providers/Microsoft.Management/serviceGroups/<sg>'
      sliId: '/providers/Microsoft.Management/serviceGroups/<sg>/providers/Microsoft.Monitor/slis/${sliName}'
      alertKind: 'fast-burn-rate'
      burnRate: '14'
      lookback: '15m'
    }
  }
}

customProperties is not decoration. The portal uses that shape to reconstruct its Baseline + Alert UI from the alert resources. If you want alerts deployed from Bicep to render correctly in the SLI blade, reproduce the shape.

The complete relationship looks like this:

Why the native path matters​

Because I believed SLI alerting was not IaC-able, I built multi-window burn-rate alerting myself in Microsoft.AlertsManagement/prometheusRuleGroups, recomputing the error ratio from raw application counters.

That parallel implementation drifted in the dangerous direction.

My PromQL summed across all three service tiers, so one user request counted once per hop and inflated the denominator roughly threefold. It also left /healthz and /readyz in the denominator, even though those endpoints never fail. Both errors pushed the ratio down, making the alert less sensitive than the SLO it was meant to enforce.

Neither object looked wrong in isolation. The SLI was correct. The alert rule was syntactically valid and fired in testing, because the test fault was a total outage and a total outage trips almost anything. I found the problem by measuring detection latency before and after correcting the selectors. The same outage went from 9m47s to 5m20s.

Alerts derived from the SLI's own published output cannot drift from the SLO they enforce. There is no second implementation to disagree with. That is a stronger property than saving a few lines of Bicep.

What I'd tell Microsoft​

The capability is good. The burn-rate formula is right, the multi-window design is right, and the portal experience is clean.

The gap is that nothing points from the SLI resource to the alert resources. The enableAlert boolean actively misleads: it looks like the alerting switch, and setting it does nothing observable by itself.

A line in the SLI documentation saying that alert configuration is stored as Microsoft.Insights/metricAlerts resources with PromQLCriteria would have saved me a day and a wrong-headed workaround. The four deployment requirements would also make a useful quickstart:

• ARM mul expects an integer
• PromQL metric alerts need the workload's regional location
• PromQL metric alerts need a managed identity
• That identity needs Monitoring Data Reader on the workspace

The lesson I'd keep​

I read the spec, tested the one field it exposed, and concluded that a feature did not exist. The spec was accurate. My inference from it was not.

When a portal can do something an API apparently cannot, the portal is still an ARM client. Twenty seconds in the network tab would have told me what a day of reading TypeSpec did not.

Check what the product does before concluding what it cannot.

The full Bicep for this alerting layer, and the rest of the showcase it belongs to, is public in lukemurraynz/AzureSLI-AzureSREAgent.

Hopefully this article saves you the day I spent looking in the wrong resource type.

References​

• AzureSLI-AzureSREAgent repository
• Native SLI alert implementation
• AzureSLI-AzureSREAgent architecture and reliability chain
• AzureSLI-AzureSREAgent deployment and demo runbook
• Why SLOs are not just alerts

Getting the infrastructure up was the easy part. Getting a single SLI created against Managed Prometheus metrics took considerably longer, and almost none of the reasons were in the documentation. This post is the list of things I wish I had known before I started, in the order they bit me.

If you only read one line: the metric namespace for Managed Prometheus metrics is customdefault, not prometheus. That one value cost me the most time by a wide margin.

The setup​

A three-tier app on AKS, instrumented with the Prometheus client, scraped by Managed Prometheus into an Azure Monitor Workspace (AMW). The SLI reads http_requests_total and compares good requests (status code below 500) against total requests. Standard availability SLI, nothing exotic.

Worth knowing up front, because it shapes everything else:

• An SLI is an extension resource on a tenant-scoped service group, at /providers/Microsoft.Management/serviceGroups/{sg}/providers/Microsoft.Monitor/slis/{name}. That is not the scope an azd Bicep deployment targets, so in practice you apply SLIs with a direct ARM call rather than from your main template.
• The resource type is Microsoft.Monitor/slis, and at the time of writing the only published API version is 2025-03-01-preview, even though the SLI/SLO feature itself is GA.

Gotcha 1: the metric namespace is customdefault​

This is the big one. Every Microsoft example I could find uses an MDM style metric namespace, and the obvious guess for Managed Prometheus metrics is prometheus, especially since the generated query text shows metricNamespace("prometheus") quite happily when you supply it.

Here is the trap. With prometheus, the metric name resolves without complaint, so you get no "metric not found" error. What you get instead is one of these two, depending on whether you declared any dimensions:

QueryErrorCE2001: Name 'service' does not exist in current context

NoPartitioningDimension: Query metadata returned no partitioning dimension,
for account: mac_..., metricName: http_requests_total, metricNamespace: prometheus

Both of those point you at dimensions, which is where I spent hours. I checked the Prometheus endpoint and confirmed all 17 labels were present and queryable. I checked the portal, and the SLI blade happily listed service, namespace, status_code, path, and the rest in its dimension picker. Everything said the dimensions existed, and the API insisted they did not.

The dimensions were fine. The namespace was wrong.

{
  "signalSourceId": "A",
  "metricNamespace": "customdefault",
  "metricName": "http_requests_total",
  "sourceAmwAccountResourceId": "/subscriptions/.../providers/microsoft.monitor/accounts/<amw>",
  "sourceAmwAccountManagedIdentity": "/subscriptions/.../userAssignedIdentities/<identity>",
  "filters": [
    {
      "dimensionName": "status_code",
      "operator": "notstartswith",
      "value": "5"
    }
  ],
  "spatialAggregation": { "type": "Sum", "dimensions": ["service"] },
  "temporalAggregation": { "type": "Average" }
}

Change prometheus to customdefault and the same request that had been failing for hours returns provisioningState: Succeeded.

Note also that an SLI needs at least one partitioning dimension. An empty dimensions array fails with NoPartitioningDimension, so pick something meaningful (I use service).

Gotcha 2: the identity roles go on the DCR, not the workspace​

The documented prerequisites say the SLI identity needs Monitoring Reader on the source workspace, and Monitoring Reader plus Monitoring Metrics Publisher on the destination workspace's default data collection rule (DCR).

I read that, and assigned both roles on the Azure Monitor Workspace resource. That looks right, it passes review, and it fails at SLI creation with:

[DestinationAmwAccountAccessValidator] Access denied to target resource
/subscriptions/.../resourceGroups/MA_<amw>_<region>_managed/providers/Microsoft.Insights/dataCollectionRules/<amw>
for identity /subscriptions/.../userAssignedIdentities/<identity>

The default DCR is not in your resource group. It lives in the workspace's managed resource group, named MA_<amw-name>_<region>_managed, and it shares the workspace's name.

In Bicep you need a separate module scoped to that resource group:

module sliDcrRbac 'modules/sli-dcr-rbac.bicep' = {
  scope: az.resourceGroup('MA_${amwName}_${location}_managed')
  name: 'sli-dcr-rbac'
  params: {
    azureMonitorWorkspaceName: amwName
    sliIdentityPrincipalId: sliIdentity.outputs.principalId
  }
}

with the DCR referenced as existing inside that module, and role assignments scoped to it.

Gotcha 3: the preview call is useless for debugging, the create call is excellent​

There is a sliSignalPreview action that the portal's Validate button uses. It is tempting to gate your deployment on it, which is exactly what I did, and it was a mistake.

For essentially every failure, sliSignalPreview returns this:

{
  "code": "MalformedStructureError",
  "message": "The request properties payload is not in the correct format."
}

No detail, no field name, nothing. Meanwhile the actual create call (PUT .../slis/{name}) returns the generated query, the validator that rejected it, and the offending identifier by name. Same inputs, completely different diagnostic value.

So do not gate creation on a successful preview. Attempt the create and read what comes back. My script now runs the preview as advisory output only.

Gotcha 4: the real error is often in the portal notification pane​

SLI creation runs several validators in sequence, and each one masks the next. When it fails in the portal, the blade shows a generic validation banner and resets the wizard back to Basics, which tells you nothing.

The full error, including the validator name, shows up in the notification pane (the bell icon). The DestinationAmwAccountAccessValidator failure above was visible nowhere else: not in the blade, and not in the response to the preview call.

If both your API calls and the portal appear to fail identically, check the notifications before you conclude anything about the platform. I twice decided this was a platform limitation and twice I was wrong.

Gotcha 5: prerequisites that fail as generic errors​

Two more that produce contentless errors rather than saying what is missing.

The service group needs a default Azure Monitor Workspace. Until Monitor settings are configured on the service group, every SLI call fails. The portal path is service group, then Monitoring, then Monitor settings. Microsoft documents the subscription scoped association:

PUT https://management.azure.com/subscriptions/<sub>/providers/microsoft.monitor/settings/default?api-version=2025-06-03-preview
{ "properties": { "defaultAzureMonitorWorkspace": "<amw resource id>" } }

What is not documented, but works, is the same settings resource at service group scope:

PUT https://management.azure.com/providers/Microsoft.Management/serviceGroups/<sg>/providers/microsoft.monitor/settings/default?api-version=2025-06-03-preview

That one returns a proper resource with an etag, and it means you can keep the whole thing in code instead of clicking through the portal. Treat it as undocumented and re-check it on each new API version.

The metrics have to exist first. An SLI requires its input metric and its partitioning dimensions to already be present in Managed Prometheus at creation time. Create the SLI before the series materialises and validation fails. In a fresh environment that is a real wait: allow for the scrape interval plus two to three minutes of ingestion lag before you try. My deployment script polls the Prometheus endpoint for the metric name and only then creates the SLI.

Gotcha 6: enableAlert looks like the switch, and it lies​

At 2025-03-01-preview, the SLI resource exposes a single boolean, enableAlert. The fast burn and slow burn thresholds and the action group selection that the portal offers on the Baseline + Alert tab have no representation anywhere in the API surface, and there is no SLI specific alert type under Microsoft.AlertsManagement or Microsoft.Insights either. Read the API cold and the reasonable conclusion is that SLI alerting is portal-only, a boolean with nowhere for the real configuration to live.

I believed that for the better part of a day, and started building a parallel implementation in Microsoft.AlertsManagement/prometheusRuleGroups to work around it. Wrong. enableAlert is a display flag. The real configuration lives in ordinary Microsoft.Insights/metricAlerts resources, using Microsoft.Azure.Monitor.PromQLCriteria, that read the SLI's own published :good/:total metrics, and every bit of it deploys from Bicep. I cover the discovery, the queries, and the working resource definition in a companion post: Azure Monitor SLI alerting really is deployable as code, just not where you'd look.

The parallel prometheusRuleGroups implementation I built first is worth knowing about as a cautionary tale, not a recommendation. My selectors summed across all three service tiers and left health-check paths in the denominator, both of which quietly made the alert less sensitive than the SLO it was meant to enforce. Deriving alerts from the SLI's own output instead of recomputing them is the actual fix, and it's the pattern I'd ship.

Gotcha 7: the burn rate windows everybody quotes assume a 30 day budget​

This one is not an Azure quirk at all, which is exactly why it is worth writing down. It cost me more time than any of the API problems above, and it will be sitting in a lot of SLO implementations right now.

Every reference for multi window burn rate alerting gives you the same table: fast burn at 14.4x over 1h and 5m, slow burn at 6x over 6h and 30m. What almost nobody carries along with it is the precondition. Those windows assume a 30 day error budget.

My demo uses a 1-day compliance window, because on 30 days a fresh error budget barely moves and there is nothing to show. I copied the canonical windows in anyway.

The failure mode is not what you would expect. It is not that the alert is noisy, or that it fires late. It is that the long window is the rate limiter, and it makes partial faults invisible. A rolling 1h average starting from zero has to be physically dragged up to the threshold, and how long that takes depends on how big the fault is. Against a 7.2% threshold:

Read that table again, because the shape of it is the point. The alert fires promptly for a total outage and effectively never fires for a realistic partial degradation.

And here is why it survives review: the fault everyone tests with is a total outage. Kill the pod, watch the page arrive, tick the box. The configuration passes every test you throw at it while being unable to detect the degradations you are most likely to actually experience.

I only noticed because I wanted a better demo. A crashing pod is a weak SLO story, since Kubernetes already tells you about a crashing pod. What I wanted was the fault where every pod is Running with zero restarts and users are still failing, because that is the entire argument for SLIs on one screen. When I injected a 20% error rate, nothing fired. My first instinct was that the fault was too small. It was not, the window was too long.

Shortening the fast burn long window from 1h to 15m fixed it. Measured on the live environment: a 20% error rate now pages at Sev1 in 8 minutes 36 seconds, which is faster than the total outage managed under the old windows.

If you take one thing from this post, take this: derive your long window from your compliance window, and test with a partial fault. Compute long_window x threshold / expected_error_rate and ask whether that many minutes is acceptable for the smallest degradation you care about. Copying 1h/5m onto a 1-day budget is as wrong as copying it onto a 1-hour budget, and it fails silently in the direction of missing real incidents.

A related trap while you are testing: a long window keeps carrying the previous incident's errors well after recovery, so a slow burn rule can still be Fired from your last test run. I nearly recorded a stale Sev2 as a successful detection. Filter alerts by startDateTime against your injection time rather than trusting what is on the screen.

Gotcha 8: the signal model is much more capable than the examples suggest​

Every worked example I found uses a single signal source and a signalFormula of "A". That makes the model look thin. It is not, and I built three noticeably better SLIs once I read the TypeSpec instead of the examples.

signalFormula combines multiple sources. Give each source a distinct signalSourceId and reference them in the formula. Dividing a histogram's _sum by its _count gives you mean latency computed inside the SLI, with no recording rule and no change to your app:

"signals": {
  "signalFormula": "A / B",
  "signalSources": [
    { "signalSourceId": "A", "metricName": "http_request_duration_seconds_sum" },
    { "signalSourceId": "B", "metricName": "http_request_duration_seconds_count" }
  ]
}

in and notin take a ^^ delimited string, not a JSON array. This is documented only in a doc comment on the ConditionOperator union, and it is useful rather than trivia:

{
  "dimensionName": "service",
  "operator": "in",
  "value": "frontend^^api^^upstream-auth-service"
}

That is one SLI spanning a whole call chain instead of three separate ones. It matters for design, not just syntax. An edge only availability SLI stays green when something fails deep in the chain and the edge masks it with a cache or a fallback. A journey scoped one does not.

windowUptimeCriteria.target is a float, so sub-second thresholds work fine, and window-based SLIs are worth having alongside request-based ones rather than instead of them. The request-based one answers "what share of requests were good". The window-based one answers "how much of the time was the service bad", and it is the one that stays meaningful in a quiet period where a handful of slow requests can swing a ratio.

One caveat to close that loop, and it is the mistake I made. Because burn rate alerting lives in a parallel system rather than on the SLI resource, nothing keeps the two in step. I ended up with four SLIs, two availability and two latency, and burn rate rules for availability only. The latency SLIs charted perfectly and paged nobody. The list looks symmetrical, which is precisely why nobody catches it. Enumerate your SLIs and your alert rules side by side.

Gotcha 9: enabling the monitoring add-on does not enable monitoring​

This is the one I am most annoyed about, because I hit the same failure mode twice in the same build and still did not recognise it the second time.

Managed Prometheus first. Setting azureMonitorProfile.metrics.enabled = true in Bicep gets you the metrics agent and no metrics. You also need a data collection endpoint, a data collection rule, and an association to the cluster. I knew this, I built it, it worked.

Then Container Insights. omsAgentEnabled: true with the right workspace id. Addon reports enabled. The ama-logs pods run happily. And every single Container Insights table was empty for the entire life of the cluster:

union withsource=T *
| where TimeGenerated > ago(24h)
| summarize n=count() by T

No rows. Not "a bit thin", not "one table missing". Nothing, anywhere, ever. Same root cause as Prometheus: the enable flag provisions the agent, and a data collection rule associated to the cluster is what gives it a job. az aks enable-addons creates that rule for you. Bicep does not.

az monitor data-collection rule association list --resource <cluster-id> --query "[].name" -o tsv

Expect one association per collection path. I had exactly one, the Prometheus one, and no error anywhere in the portal, the CLI, the addon status, or the pod logs to suggest anything was missing.

Take the DCR shape and the stream list from the canonical onboarding template in microsoft/Docker-Provider rather than writing it from memory. The stream names are exact strings and there are twelve of them.

The knock-on effect is worth stating: my workbook had never had data, and my write-up claimed the SRE Agent "cites the AKS pod event as root-cause evidence". It cannot cite a pod event from an empty table. I had asserted that from the design rather than from an observation.

Gotcha 10: an empty panel and an impossible query look identical​

Once Container Insights was actually collecting, one workbook panel stayed empty. The query was fine, in the sense that it parsed and ran and returned zero rows:

InsightsMetrics
| where Namespace == 'container.azm.ms/memory'
| where Name == 'memoryWorkingSetBytes'

There is no container.azm.ms/memory namespace. InsightsMetrics carries node, disk and kube-state metrics. Container CPU and memory live in the Perf table under ObjectName == 'K8SContainer', and to get from there to a namespace you have to join back to KubePodInventory, because Perf.InstanceName is <clusterResourceId>/<podUID>/<containerName>:

let podMap = KubePodInventory
| where Namespace == 'payment'
| distinct ContainerName, Name;
Perf
| where ObjectName == 'K8SContainer' and CounterName == 'memoryWorkingSetBytes'
| extend ContainerName = strcat(tostring(split(InstanceName, '/')[-2]), '/', tostring(split(InstanceName, '/')[-1]))
| join kind=inner podMap on ContainerName
| summarize WorkingSetMiB = avg(CounterValue) / 1024 / 1024 by bin(TimeGenerated, 1m), Name

The lesson generalises well beyond workbooks: a query against the wrong table returns zero rows forever, and zero rows is indistinguishable from "no data yet" or "nothing is broken right now". Before you believe an empty panel, strip the filters and run a positive control:

InsightsMetrics | summarize count() by Namespace, Name | order by count_ desc

That one command would have shown me the real namespaces immediately, and it is the first thing I will run next time.

Gotcha 11: Service Group membership looks tenant-blocked, and is actually an api-version trap​

This one I got wrong, publicly and for weeks, so it is worth walking through properly.

Service groups are pitched as spanning resources across subscriptions, which is exactly what makes them attractive for a service-level SLO. I could not add a single member. The path, each step producing a different error:

1. PUT .../providers/Microsoft.Relationships/serviceGroupMember/<name> returns SubscriptionNotRegistered.
2. az provider register --namespace Microsoft.Relationships succeeds.
3. Retry: HttpRequestPayloadAPISpecValidationFailed. Two things buried in the details array: the name must match ^[a-zA-Z0-9]{3,64}$, so hyphens are rejected, and sourceId is required alongside targetId.
4. Retry with a valid payload: RelLifecycleNotEnabledForTenant, "Relationship lifecycle callbacks are not enabled for tenant."

Undocumented flag, no self-serve way to enable it. I recorded it as a hard limitation and designed around it.

It is not a limitation. Step 4 only happens on 2026-03-01-preview.

# fails: RelLifecycleNotEnabledForTenant
PUT .../serviceGroupMember/slisreShowcase?api-version=2026-03-01-preview
{"properties":{"sourceId":"/subscriptions/<sub>","targetId":"<sg>"}}

# succeeds
PUT .../serviceGroupMember/slisreShowcase?api-version=2023-09-01-preview
{"properties":{"targetId":"<sg>"}}

The older version infers sourceId from the parent scope and provisions in seconds. The newer one demands sourceId explicitly and then refuses on a tenant flag. So the newest preview is the more restrictive one, which is the opposite of the assumption that sent me down this path. I had reached for the newest api-version precisely because the older one seemed to be failing, and in doing so swapped a solvable error for an unsolvable-looking one.

What actually misled me is subtler than the version number. Errors 1, 3 and 4 are all real and all different, so each retry felt like progress toward a wall. A single error repeated would have made me question the request; a sequence of distinct, plausible errors reads like a narrowing path toward a wall.

And once it works, it is ordinary Bicep, because the relationship's parent is the member, not the service group:

targetScope = 'subscription'

resource serviceGroupMembership 'Microsoft.Relationships/serviceGroupMember@2023-09-01-preview' = {
  name: 'slisreShowcaseBicep'          // alphanumeric only
  properties: {
    targetId: '/providers/Microsoft.Management/serviceGroups/${serviceGroupId}'
  }
}

That is a subscription-scoped resource, so it deploys from the same template as everything else. Which produces an odd asymmetry worth knowing:

Same feature area, opposite answers, purely because of which end of the relationship owns the resource.

Does membership matter? Less than the effort suggests, and the reason is worth understanding. An SLI does not reference Azure resources at all: its scope is sourceAmwAccountResourceId plus metricNamespace, metricName, and dimension filters. My journey-availability SLI spans three tiers purely because of service in "frontend^^api^^upstream-auth-service". Membership gives you topology and rollup; it changes no SLI's computation.

What it does buy is letting something else tell you what composes the service, rather than you asserting it. That matters for the SRE Agent, which scopes its knowledge graph to a resource group: if a service actually spans resource groups, the agent investigates one and silently misses the rest.

The lesson, and it is the same one as the alerting post: when an Azure preview API refuses, check whether an older api-version accepts. Newer previews tighten validation and add gates, and the resource provider will happily tell you which versions exist: the error at the wrong version named all of them, and I did not read it.

Gotcha 12: GET returns shapes that PUT rejects​

Round-tripping a resource (GET it, change one field, PUT it back) is the most natural thing to try, and it fails in several places across this stack:

• SLI: the identity.userAssignedIdentities values from GET are rejected on PUT with InvalidIdentityValues. They must be empty objects ({}).
• SRE Agent response plans: GET returns isDeleted, documentType, partitionKey, createdAt, updatedAt; POST rejects all five with Unknown incident filter properties.
• SRE Agent skills and subagents: GET returns array fields as stringified lists ("tools": "['RunAzCliReadCommands']"), while PUT requires real JSON arrays. Sending back what you received yields 400 ... could not be converted to System.Collections.Generic.List.

Strip server-generated fields and rebuild write payloads from the documented shape rather than from a GET response.

Two SLI correctness traps while you are here​

Neither of these is an Azure quirk, they are just easy to get wrong, and both quietly produce an SLI that measures the wrong thing.

Health probe traffic. Your scraped series includes /healthz and /readyz. Those never fail, so leaving them in the denominator dilutes the error ratio and your SLO will not move the way you expect. Filter to the user journey path.

Multi-tier double counting. If every tier emits the same metric name, one user request is counted once per hop. Filter to the edge service, otherwise a three-tier app inflates the denominator threefold.

What actually found the answers​

Worth saying, because the method matters more than the specific values.

Searching the docs and Learn got me nowhere on the namespace. What found it was searching GitHub code for the literal error string, DestinationAmwAccountAccessValidator. That returned exactly one result, a workarounds.md in Toru Makabe's aks-chaos-lab, which documents both the DCR RBAC requirement and the metrics-must-exist-first constraint, and whose SLI Bicep module has metricNamespace defaulting to customdefault. Full credit there, that repo saved me a support case.

If you are stuck on an Azure error that reads like an internal validator name, put the exact string into GitHub code search before you do anything else. Somebody has usually hit it and written it down.

The other thing that paid off was reading the TypeSpec rather than the examples. sliProperties.tsp in Azure/azure-rest-api-specs is about 260 lines and took five minutes, and it is where the ^^ delimiter, the float target, and the multi source formula all are. The examples had led me to believe none of those existed.

And one anti method, since it wasted an hour. When I could not find an API to list the SRE Agent's incidents, I started guessing endpoint paths. Three of them returned HTTP 200, which felt like progress, right up until I looked at the body and found the portal's single page app HTML rather than JSON. A 200 from a SPA host means "this host serves a web app", not "this endpoint exists". If you are guessing paths, you have already left the part of the problem that is solvable by guessing.

What actually worked was asking what the service emits rather than what it exposes. The agent resource has a logConfiguration.applicationInsightsConfiguration.appId, and the agent writes an IncidentActivitySnapshot custom event to it on every incident transition, carrying the response plan id, the autonomy level, the handled timestamp, and the agent's written investigation summary. That answered in one KQL query what an hour of path guessing could not. When an Azure service will not give you an API, check its telemetry before you reach for its UI.

Wrap up​

None of this makes Azure Monitor SLIs a bad feature. The error budget and burn rate experience is useful, and having it native beats maintaining your own burn rate rules. But the preview API surface is sharp in places, and the failure modes are almost all generic errors that point away from the real cause.

The short version for anyone starting out:

• Use customdefault as the metric namespace for Managed Prometheus metrics.
• Put the identity roles on the default DCR in MA_<amw>_<region>_managed, not on the workspace.
• Debug with the create call, not sliSignalPreview.
• Check the portal notification pane for the real error.
• Configure the service group default AMW first, and wait for your metrics to land.
• Don't recompute burn rate alerting yourself. Derive it from the SLI's own :good/:total metrics via Microsoft.Insights/metricAlerts and PromQLCriteria, it's IaC-able, just not where the SLI resource itself suggests.

Hopefully this saves you the afternoon it cost me.

The full Bicep, scripts, and agent configuration for the showcase this was all found while building are public: lukemurraynz/AzureSLI-AzureSREAgent.

References​

• Create service level indicators in Azure Monitor
• Microsoft.Monitor/slis template reference
• Use Azure Monitor issues (default AMW association)
• Service groups overview
• torumakabe/aks-chaos-lab
• lukemurraynz/AzureSLI-AzureSREAgent

That gap is a product and service-design problem as much as a monitoring problem. The people responsible for a payment journey need to know whether checkout works, how quickly it works, and which customer segment is paying the cost of an incident. The on-call engineer needs enough evidence to choose the right first action. The business needs a trustworthy way to decide when an error budget has been spent.

I wanted to see whether an Azure Monitor SLI could give the Azure SRE Agent enough context to investigate an Azure Kubernetes Service failure, rather than just react to a pod alert. I built a small payment service, chose signals around the customer journey, and deliberately broke it.

The result is an infrastructure-as-code deployment that runs with azd up. It measures availability, latency, the full request journey, the public path, and customer tiers. Azure Monitor alerts on those service-level signals, and the Azure SRE Agent investigates the incident and, when allowed, fixes it.

The important result is not the number of Azure resources. It is the chain from customer need to operational decision:

I got several things wrong. The corrections stay in the post because they expose the difference between a design that looks plausible and one that has been tested with the people, decisions, and failure modes it is meant to serve.

Who this is for​

This walkthrough is aimed at teams that share responsibility for a customer-facing service:

• Platform engineers who own Azure Monitor workspaces, identities, collection, and RBAC
• SRE and operations teams who define SLOs, error-budget policy, and incident response
• Product and engineering leaders who need reliability data expressed in terms of customer impact
• Teams evaluating how much operational work an AI agent should investigate, propose, or perform

Start with the service promise​

The arrow from slis to alerts is the design decision I cared about. The alerts read the SLI's own published metrics, not a second implementation built from raw application counters. That keeps the alert and the SLO measuring the same customer promise.

This is the first design-thinking checkpoint: define the outcome before choosing the tool. “A pod is ready” is an implementation fact. “A customer can complete checkout within 500 milliseconds” is an outcome. Both are useful, but they answer different questions and should not be treated as interchangeable.

Give the service a durable identity​

An SLI in Azure Monitor is an extension resource on a tenant-scoped Service Group:

/providers/Microsoft.Management/serviceGroups/<sg>/providers/Microsoft.Monitor/slis/<name>

That scope is the point. A service is rarely one resource group: frontend here, API there, and a database in a shared subscription. A tenant-scoped Service Group gives the service an identity that outlives any particular resource layout, so the SLO follows the customer experience rather than a cluster.

For a platform team, this reduces the cost of change. Teams can move workloads, split deployments, or replace a dependency without making the reliability conversation start again. For a product team, it creates a stable place to discuss whether the service is meeting its promise.

There is an important boundary here: a Service Group is an operational view, not a security boundary. It is currently in public preview, and membership does not grant access to the resources inside it. A resource can belong to multiple Service Groups, which is useful when platform, product, and customer-support teams need different views of the same service. Keep resource RBAC and Service Group RBAC as separate design decisions.

The Service Group does not define what the SLI measures. There is no membership involved. An SLI's scope is sourceAmwAccountResourceId (Azure Monitor Workspace) plus a metric name and dimension filters:

"filters": [
  { "dimensionName": "namespace",   "operator": "eq",            "value": "payment" },
  { "dimensionName": "service",     "operator": "eq",            "value": "frontend" },
  { "dimensionName": "path",        "operator": "eq",            "value": "/checkout" },
  { "dimensionName": "status_code", "operator": "notstartswith", "value": "5" }
]

The Service Group is the SLO's home, not its definition. Membership gives you topology and rollup, but changes nothing about what any SLI computes.

Plan the SLI identity path​

The SLI also has an identity and data-storage path that is easy to miss when the portal hides the plumbing. Azure Monitor reads the source Azure Monitor Workspace and writes the evaluated SLI results to a destination workspace. A user-assigned managed identity needs Monitoring Reader on the source workspace, Monitoring Reader plus Monitoring Metrics Publisher on the destination workspace, and Monitoring Reader on the destination workspace's default data collection rule. The source and destination can be the same workspace, but separating raw telemetry from evaluated reliability data can make ownership and access easier to explain.

This is a useful handoff between platform and product teams. The product team defines what “good” means. The platform team owns the identity, workspace boundaries, collection rules, and retention that make that definition executable.

Membership is ordinary Bicep, with one sharp edge. 2026-03-01-preview returned RelLifecycleNotEnabledForTenant; the older 2023-09-01-preview worked:

targetScope = 'subscription'

resource serviceGroupMembership 'Microsoft.Relationships/serviceGroupMember@2023-09-01-preview' = {
  name: 'slisreShowcaseBicep' // alphanumeric only, 3-64 chars
  properties: {
    targetId: '/providers/Microsoft.Management/serviceGroups/${serviceGroupId}'
  }
}

The relationship's parent is the member, not the Service Group, so this is subscription-scoped. The SLIs are the opposite case: their parent is the Service Group, which forces a tenant-scoped deployment and an Azure RBAC grant at / that Global Administrator does not confer.

You can express "one SLO across frontend, API and auth" today with one multi-value filter:

{
  "dimensionName": "service",
  "operator": "in",
  "value": "frontend^^api^^upstream-auth-service",
}

in and notin take a ^^-delimited string, not a JSON array. That's documented in one TypeSpec comment and nowhere else.

Turn customer needs into measurable questions​

Each SLI exists because the basic availability number leaves a different customer or business question unanswered. A sixth, latency-windowed, measures the same 500ms bar as mean latency per five-minute window rather than per request, and is covered separately below.

Azure Monitor offers two evaluation methods. Request-based evaluation asks whether the ratio of good requests to total requests meets the target. Window-based evaluation asks whether time intervals meet a quality threshold. Request-based evaluation is usually the better fit for checkout because one failed transaction should count as a failed transaction, even during a quiet period. Window-based evaluation can be useful when you want to smooth short bursts or measure the proportion of time a system remains within a limit. Choose the method with the customer experience and decision in mind, not the metric that happens to be easiest to query.

availability measures the edge at /checkout: good requests over total. journey-availability spans all three tiers with the in filter. An edge-only SLI stays green when the API fails and the frontend serves a cached fallback, while the user's journey is broken.

latency exists because a service that answers every request in nine seconds is up, and useless. Good is the le="0.5" histogram bucket; total is the request count.

external-availability matters because every SLI above is computed from traffic generated inside the cluster. You cannot measure an outage from inside the thing that is unreachable. If DNS breaks or the load balancer misroutes, no request arrives, so there is no error metric and no denominator either. Availability reads 100% during a total outage. That is structural; it needs something outside the workload requesting the service over the real network path.

tier-availability is the same measurement as availability, partitioned by who the customer is rather than by which component served them.

For any SLO, ask which signal moves if DNS breaks. If none does, it measures instrumentation rather than availability. Then ask who needs to act on the signal and what decision it should change. That turns a dashboard metric into a useful service conversation.

Choose signals that represent people, not components​

The demo app is instrumented with the Prometheus client because I wrote it. Most workloads you need SLOs for are not yours to change, and that's fine: the SLI reads Prometheus series from an Azure Monitor Workspace and does not care what produced them.

Prefer an infrastructure-emitted signal even when you can instrument the app. It survives redeploys, language changes and vendor upgrades, and is uniform across every service behind it.

One trap cost me hours: for Managed Prometheus sources, metricNamespace must be customdefault, not prometheus. With the wrong value the metric resolves fine and SLI creation fails with errors about dimensions, which is entirely the wrong place to look.

Measuring users, not components​

An SLI's scope is a dimension filter, not a resource. journey-availability spans three Kubernetes Deployments purely because of:

{
  "dimensionName": "service",
  "operator": "in",
  "value": "frontend^^api^^upstream-auth-service"
}

Add a fourth tier tomorrow and it joins the SLO by emitting the label. Nothing gets registered. The SLO describes the journey; the deployment topology is free to change underneath it.

The partition trick​

The dimensions do not have to be technical. This app emits customer_tier from a request header and propagates it across every hop:

KNOWN_TIERS = frozenset({"premium", "standard", "free"})
DEFAULT_TIER = "standard"

tier = request.headers.get("x-customer-tier", DEFAULT_TIER).lower()
if tier not in KNOWN_TIERS:      # never let a caller mint unbounded series
    tier = DEFAULT_TIER

The SLI partitions on it via spatialAggregation.dimensions:

"spatialAggregation": { "type": "Sum", "dimensions": ["customer_tier"] }

The alert uses the portal's own idiom:

sum without ("INCLUDE-ALL-DIMENSIONS-DONT-REMOVE") ({__name__="...:good"})

Summing without a label that does not exist aggregates while preserving every real dimension, so one alert rule fires once per partition. One SLI, one rule, and the incident names the affected segment.

Break premium only:

./scripts/inject-fault.sh --mode=errors --rate=100 --tier=premium

The second has already eliminated most of the search space: not a capacity problem, not a dependency outage, and not a deployment affecting everyone. Something is routing premium differently. That is a fault class per-service telemetry cannot see at all: every pod is healthy, every tier is up, and only the signal partitioned by who the user is moves.

This is where the business lens changes the response. A blended 85.02% can trigger concern, but it does not tell a product manager whether premium customers, free customers, or everyone is affected. The partitioned result supports prioritisation, customer communication, and a more honest conversation about differentiated service commitments.

Imagine a premium customer trying to complete checkout during this fault. The pods are healthy and the overall availability number is still above zero, but that customer cannot complete the transaction. The tier-specific SLI turns that experience into an operational fact: investigate the premium path first, communicate the impact accurately, and avoid spending the incident response effort on healthy free and standard traffic.

There are four limits:

1. Azure cannot infer business dimensions. customer_tier exists because the application chose to emit it.
2. Cardinality is the tax. Keep business dimensions closed and small; tenant IDs are usually a bill, not a feature.
3. Targets are per SLI, not per partition. Differentiated targets need one SLI per tier.
4. In-cluster signals stay blind to the front door. Partitioning does not fix what external-availability exists to fix.

The Service Group gives the service a tenant-scoped identity that survives resource churn. Through customProperties.sliId, the agent gets a service-level fact about users rather than a symptom about pods.

Service Groups nest too. A child group joins a parent via properties.parent.resourceId at create time. The differentiated-target SLIs now live in a Customer Segments child group under the parent, without moving the existing six SLIs or their history.

Make the alert answer a business question​

Azure Monitor has a native SLI alerting path. In the portal, enableAlert can configure a baseline alert, fast-burn alert, and slow-burn alert, with an action group defining who is notified and what downstream action runs. That is the shortest path when you want the platform's standard SLI experience.

I used a second path in this project: explicit Microsoft.Insights/metricAlerts resources using Microsoft.Azure.Monitor.PromQLCriteria, scoped to the Azure Monitor Workspace. That gave me infrastructure-as-code control over the PromQL, dimensions, custom properties, and alert names used by the SRE Agent. It is not a replacement for native SLI alerting. Decide which path owns the policy, and avoid enabling both for the same condition unless duplicate notifications are intentional.

They read the SLI's own published metrics, <sli>:Good, <sli>:Total, and <sli>:Value:

(
  ( sum(increase({__name__="ns::<sg>/m::availability:total"}[15m]))
  - sum(increase({__name__="ns::<sg>/m::availability:good"}[15m])) )
  /
  ( sum(increase({__name__="ns::<sg>/m::availability:total"}[15m])) * (1 - 0.99) )
) > 14

I originally recomputed the same signal in parallel Prometheus rules. Those selectors summed across all three tiers and left /healthz in the denominator, making the alert materially less sensitive than the SLO. Both objects looked correct in isolation, and total-outage testing did not catch it. Detection latency was 9m47s versus 5m20s on the same outage.

Alerts derived from the SLI's own output cannot drift from it. That safety property is the technical argument. The business argument is that the alert represents the same commitment people agreed to measure, rather than an engineer's approximation of it.

Use error budgets to guide attention​

Burn rate is how fast you're spending error budget. At a 99% target, a 2% error rate burns at 2x; a 20% error rate burns at 20x. That makes an error budget a decision tool: it connects reliability work to the limited amount of failure the service can afford before customers and the business feel the impact.

The canonical 14x over one hour and 6x over six hours assumes a 30-day budget. On a one-day window with a 0.5% budget, the long window becomes the rate limiter:

Derive the long window from the compliance window. Shortening it to 15 minutes took a 20% error rate from roughly 22 minutes to a measured 8m36s end to end.

A correction: sum_over_time() was wrong here​

The SLI's :good and :total are cumulative across the compliance window, not counters. sum_over_time(...[15m]) therefore sums fifteen cumulative snapshots. It is accumulated damage, not the error rate during those fifteen minutes.

The practical consequence was a complete backend outage that ran for twelve minutes without crossing the 14x threshold. Treat the baseline attainment alert as the fast detector; keep burn-rate alerts for the budget-spend signal. If true short-window sensitivity is required, compute it from raw application counters with rate(), accepting the drift risk this design otherwise avoids.

A second correction: rate() was not the problem​

That conclusion was half wrong. sum_over_time()/rate() returning empty was true of exactly one test: a brand-new SLI queried within its first evaluation cycles. rate() and increase() need at least two samples spanning the lookback window. After a day of samples, rate(), increase() and delta() all returned normal values.

On identical 15-minute fault injections, sum_over_time() took 49 minutes to trip; increase() took 10m06s, most of which was the mandatory five-minute sustained-condition window. Use increase(), not sum_over_time(), and interpret an empty PromQL result as "not enough history yet" until proven otherwise.

Design the handoff from signal to action​

The agent subscribes to Azure Monitor directly through incidentManagementConfiguration.type: AzMonitor, scoped by knowledgeGraphConfiguration.managedResources to the resource group. It is not wired through an action group. That is a different integration path from native SLI alert notifications: the SRE Agent receives the Azure Monitor incident, while an action group can notify people or invoke other automation.

Each alert carries structural context:

{
  "sliId": "/providers/.../serviceGroups/<sg>/providers/Microsoft.Monitor/slis/availability",
  "serviceGroupId": "/providers/Microsoft.Management/serviceGroups/<sg>",
  "alertKind": "fast-burn-rate",
  "burnRate": "14",
  "lookback": "15m",
}

The agent can resolve sliId and learn the target, compliance window, and filters. The incident opens as “the /checkout journey at the frontend edge, committed at 99% over a one-day rolling window, is at 93.4%,” not “a burn-rate rule exceeded 6”. The first version gives an operator or an agent a user-centred problem to investigate.

The handoff also follows a useful service-design pattern: preserve context as the problem moves between people and systems. The alert carries the service, journey, target, customer segment, and time window. The agent can then investigate the likely experience before choosing an implementation-level action.

Correlation is the fastest diagnosis​

Given a Deployment scaled to zero, the agent ran 31+ tool calls, queried Log Analytics, correctly root-caused the missing pods, and scaled the Deployment back up. Injection to recovery was about 20 minutes, roughly five of them detection.

The current SRE Agent guidance recommends starting with Reader access, where writes require approval, and moving to Privileged access only after the team trusts the workflow. That is a good pilot model: begin with investigation and proposed remediation, measure false positives and recovery quality, then grant the smallest write permission for a narrow incident class. The agent also creates or uses Application Insights and persists investigation context, so data retention, cost, region, and who can review that operational memory belong in the service design as well. The SRE Agent security overview describes the related identity, isolation, and telemetry considerations.

On a later run it refused to declare success based only on rollout status: "the rollout completed, but the probe and workspace telemetry do not yet provide a current success sample." kubectl confirmed the outcome it was checking for: 2/2 ready.

Treat trust as part of the service design​

The agent needs Kubernetes access to act, but permission to act is not the same as evidence that it should act. Scope the permissions tightly:

Writer rather than Admin excludes role bindings, so the agent cannot grant itself anything further.

PreToolUse hooks gate individual actions, but only if their matchers bind. Mine shipped with ^(restart_|scale_).*, copied from another framework. The real tool names are PascalCase: RunKubectlWriteCommand, Terminal, and RunInTerminal. The hooks matched zero tools while reporting HooksRun: 2, FinalDecision: pass.

When the agent remediated, it used RunInTerminal, not the obvious kubectl-specific tool. A matcher listing only write tools would have missed it.

A response plan with agentMode: review also did not prevent execution in my test. That contradicted the current product guidance, which describes Review mode as requiring approval for write operations, so I treated it as a behaviour to retest rather than a safety guarantee.

From a design-thinking perspective, the question is not “can the AI fix the incident?” It is “what level of autonomy is appropriate for this incident, this customer impact, and this reversibility?” A scale-up in one namespace may be a bounded experiment. A database migration or a change affecting every tenant is a different service risk and needs a different approval model.

The re-test showed why the underlying control matters​

I wrote that Review was the one control that prevented writes after testing it once. It was not. Repeating the scaled-to-zero fault twice showed the supervised agent, ARM mode: Review, performing the scale operation without approval.

The agent's incident telemetry said IncidentMitigatedByAgent: False both times. Kubernetes audit logs were the only reliable witness: kube-audit-admin named the principal that executed the PATCH .../scale request. Treat that discrepancy as release-specific behaviour to verify in your tenant, not as a supported guarantee either way.

The cause was a hook named require-approval-for-restarts whose content effectively said: deny unless the action is confined to the namespace, reversible, evidence-backed, and not a delete. It did not check whether a human had approved anything. The agent was grading its own homework.

I rewrote the hook to unconditionally deny and re-tested with a fresh fault injection. The autonomous agent performed the fix; the supervised agent investigated and touched nothing. One clean run is evidence, not a guarantee, so keep testing the layer underneath the telemetry.

Test the experience, not only the configuration​

• An enabled monitoring addon does not enable collection. omsAgent.enabled deploys the agent; a data collection rule makes it collect. Without the DCR, Container Insights is empty and nothing reports an error. Managed Prometheus has the same DCE/DCR/DCRA trap.
• Absent data makes burn-rate alerts stop firing. A zero denominator produces NaN, and NaN > threshold is false. Alert explicitly on absence with absent().
• Incident titles reach the agent URL-encoded. availability baseline alert becomes availability%20baseline%20alert, so a plan matching spaces never matches. Name alerts without spaces.
• Overlapping response plans disable the agent. Several matching plans created incidents that were marked handled within a minute while running zero tools. One matching plan produced 31 tool calls and a successful remediation.

A query returning zero rows and a query that can never work are indistinguishable. Establish a positive control before believing an empty result. More broadly, test the experience you intend to protect: inject a fault, observe the customer-facing signal, check the alert context, and verify the action or refusal in the underlying audit trail.

Make reliability legible to decision-makers​

The platform engineer wanted a single pane connecting infrastructure state to user reliability. For most of this project that was two panes: pod health in Log Analytics with KQL, and SLI evaluations in the Azure Monitor Workspace with PromQL.

Workbooks support Prometheus as a data source. The JSON shape is undocumented, but the portal's own workbook uses this form:

{
  "queryType": 16,
  "resourceType": "microsoft.monitor/accounts",
  "crossComponentResources": ["<azure monitor workspace resource id>"],
  "query": "{\"version\":\"PrometheusQueryProvider/1.0\",\"queryText\":\"slo:error_budget:remaining_ratio\",\"type\":\"query_range\"}",
}

The composite panel uses the minimum attainment across every SLI rather than an average. A premium-tier outage should read “we are breaching our worst commitment”, not disappear inside a blended number.

That panel also closes the reporting gap for the engineering manager, who needs credible error-budget reporting but is least likely to browse nine SLI charts. Alerting on a number is not the same as being able to see its trend, and a trend without an agreed decision is not a useful outcome either.

What it still does not do​

• Service Groups are still in public preview. Do not make a production security or compliance boundary depend on Service Group membership, and confirm the current preview terms before adopting the pattern broadly.
• An SLI's definition is portable, but its execution is not permission-free. The managed identity, source and destination workspaces, default data collection rule, and metric collection path all need to be healthy.
• The in-cluster SLIs still measure in-cluster traffic. external-availability covers the public path; the rest are blind to ingress and DNS by construction.
• The tier SLA table is a draft, not a signed contract. Differentiated targets exist and alert independently, but nothing enforces the SLA outside that policy document.
• The one-day compliance window is a demo choice so the budget visibly moves. Never copy evaluationPeriodDays: 1 into production without an explicit decision.
• Native SLI alerts and explicit PromQL metric alerts are two policy paths. Choose one owner for each condition, or document why duplicate alert routes are useful.
• SLOs do not detect outages faster than a probe. On the same outage, the external web test fired in 3m37s and the burn-rate alert in roughly five minutes. SLOs earn their place in the ambiguous middle: deciding whether a 2% error rate is worth waking someone for.

A reliability design checklist​

Start with the user's task and the business promise. Measure the user, not the application. Ask what moves if DNS breaks. If none of the signals move, you have instrumentation, not an SLO.

Choose dimensions that support a decision. Customer tier, journey, region, and tenant can explain impact, but uncontrolled cardinality can make the signal expensive and noisy.

Derive alerts from the SLI, never alongside it. A parallel implementation will drift silently toward being less sensitive than the SLO it enforces, and it will pass testing because total outages trip almost anything.

Design the handoff for the person or agent who must act. Carry the service identity, customer impact, target, and evidence into the incident instead of exposing only a rule name.

Verify guardrails by trying to breach them. Hooks that match nothing, plan modes that do not supervise, and rules that never evaluate all look identical to ones that work. The difference is visible only when you inject a fault and watch what happens.

Finally, treat reliability as a learning loop. Observe the customer experience, interpret the evidence with the people who own the outcome, test the smallest safe intervention, and update the design when reality disagrees with the model.

The shape of the problem​

Three extra steps sit between "a request arrived" and "code that does something with it." None of the three add behaviour. All three exist because, at some point, the framework's built-in routing did not do what the app needed yet. That was the fastest way to bridge the gap. Years later, the framework has caught up. But hundreds of routes still depend on that bridge, and nobody wants to be the one who breaks it.

"Just rewrite it" is the answer everyone reaches for. It is also the answer almost nobody can actually ship. A rewrite means a feature freeze long enough to matter. It also means one huge merge that is almost impossible to review well. And it creates a regression surface that spans the entire app instead of one corner of it. The option that actually ships is one domain at a time.

That option already has a name, so use it instead of acting like it is new. Martin Fowler wrote about it on martinfowler.com in 2004 as the strangler fig application. The name comes from the way a strangler fig seed grows high in a host tree's branches. Over years, it sends roots down around the trunk until the original tree is no longer holding anything up. Build the new path next to the old one. Move traffic to it one piece at a time. Retire the old piece as nothing depends on it anymore. The general shape is Fowler's. What is specific to a legacy HTTP dispatch layer is the fixed rule set below. It shows what a strangler fig migration looks like when the thing you are strangling is three layers of route-dispatch extra steps in front of hundreds of endpoints.

The migration rules​

1. Freeze the current list of routes as a baseline before changing anything: add a test that checks the current set of registered routes. Then any drop or duplicate during migration fails loudly instead of shipping silently. This one test is what makes everything after it safe. You cannot check "nothing changed for the caller" without first writing down what "unchanged" means.
2. Move one domain's handlers to the framework's built-in pattern for each pass. Not the whole app. Pick one clear slice, such as a single API domain (orders, billing, or whatever your app's real seams are). Replace it with native routers and typed request and response schemas. Share one dependency-injection context instead of spreading ad hoc lookups through the handler.
3. Port the old tests to the new transport, don't write them from scratch. The current test suite shows the real behaviour, edge cases and all. Those cases often came from real incidents. Writing tests from a blank page means finding those edge cases the hard way, in production, again. Porting a dispatch-based test to hit the new HTTP layer directly keeps the same contract while changing the mechanism underneath it. Keep the same inputs and the same status codes and payloads. Only the transport changes.
4. Delete the old domain's legacy modules as soon as the cutover is done. Not "leave them for now in case we need to roll back." The moment the new path is checked and merged, remove the old wiring for that domain in the same change. Two ways to do the same thing are worse than either way alone. A bug fix now has to be made in both places, or it silently does not.

Where the actual time goes​

The router code itself is close to mechanical. The time sink is everywhere the old and new paths disagree about details nobody wrote down:

• Status code mapping: the old dispatch shim might turn a permission error into a 403 and a value error into a 400 through some central error handler. The new built-in router needs the same mapping. If it does not, clients that depend on specific status codes break. That can be easy to miss in a quick manual check and obvious the moment a real client hits it.
• Test harness behaviour that hides real errors: a test client that swallows server exceptions by default can report a clean pass on a handler that is actually throwing a 500. The framework caught the exception before your assertion ever saw the status code. Get this wrong and "port the tests" from rule 3 quietly stops testing anything.
• Request shape differences: a GET that used to pass parameters through a dispatch-layer body now needs those parameters in the query string to fit the framework more cleanly. Each difference is small. There are a lot of them.

A design doc never catches this. A failing test does, one at a time. That is why a domain-by-domain migration matters. A failing test in a 30-route migration tells you something specific. A failing test in a 400-route one huge rewrite tells you almost nothing about where to look.

What this rule set adds to the general pattern​

The general case for a strangler fig migration over a rewrite is well known: ship value steadily, avoid a feature freeze, and catch regressions in a scope small enough to debug. What this rule set adds for a dispatch-layer migration is specific. The baseline from rule 1 turns each migration into a change you can check with a test, not just a smaller one. You are not asking a reviewer to trust that "the behaviour is the same." You are pointing to a test that proves it. That test fails the build the moment parity breaks. This scales in a way careful review never does, and it misses nothing the test actually covers.

None of this is tied to one framework. None of it requires treating the strangler fig approach as a new idea. It is the discipline that makes "boring, steady, and safe" beat "fast, exciting, and scary" when the thing you are strangling is a dispatch layer that hundreds of routes depend on.

I’ve spent a lot of time around orchestration frameworks. Azure Durable Functions doesn’t automatically rewind entire orchestrations when a single step fails; instead, the orchestrator fails and previous steps remain executed, with "rewind" only available as a manual, preview‑level recovery that replays from the failure point. Temporal demands a Kubernetes cluster and a team comfortable with its programming model. And then there’s the dark matter: the cron scripts and PowerShell scheduled tasks that run half the back offices I’ve seen, failing silently at 3am until someone notices the queue depth.

I wanted something that treated recovery as a first-class capability rather than an operational afterthought. Something where an operator could replay one step without touching the rest of the job. Something where compensation wasn't an afterthought but a first-class primitive. Something that ran on Azure's managed platform without requiring Kubernetes, and used managed identity everywhere because I am done explaining connection strings to auditors.

So I built Execution Ledger. It's a Rust multi-service backend deployed on Azure Container Apps, and this post is about why.

The gap: step-level recovery as a first-class primitive​

Most orchestration platforms treat replay as an all-or-nothing operation. Durable Functions rewinds the orchestration to a checkpoint, every step after that point re-executes. Temporal uses continue-as-new to re-run the entire workflow. Manual scripts? Hope your operator knows SQL.

The problem is that in real multi-step jobs, most step failures are surgical. The REST call to the ERP timed out because the ERP was under maintenance. The blob upload failed because a SAS token expired. Re-running every other step, steps that already committed, is wasteful at best and dangerous at worst.

Execution Ledger gives each connector step an idempotency key (derived from run ID + step ID) and lets you target replay at a specific step. Other steps in the run are untouched.

cargo run -p execution-ledger-cli -- job-replay <tenant-id> <run-id> health

That's it. One step. The worker re-executes only health. The other steps (the blob transfer, the queue message, the notification) stay exactly as they were. This makes incident recovery precise rather than all-or-nothing.

Why Rust​

Three reasons, in descending order of importance:

1. No cold starts. Container Apps scale from zero. When a Durable Functions app wakes up after inactivity, C# JIT compiles, JavaScript loads modules, Python imports libraries. Rust compiles to a native binary. Startup is measured in milliseconds, not seconds. When KEDA triggers a scale-out from 1 to 10 replicas because the Service Bus queue just spiked, every new replica starts processing intents immediately.

2. Resource density. A Rust binary serving HTTP with Axum at idle consumes single-digit megabytes of RAM. That matters in Container Apps where you pay per vCPU-second and per GB-second. At 10 worker replicas, the memory delta between Rust and a managed runtime pays for itself monthly.

3. A compiler that actually enforces things. The workspace denies unwrap(), expect(), and panic!() via clippy lints at the crate level. Unsafe code is forbidden. clippy::pedantic is set to warn. These aren't style preferences. They're guardrails. When you build a platform that processes tenant-scoped jobs with Entra-authenticated callers, you want the compiler catching mistakes before they reach production.

// ❌ won't compile — clippy::unwrap_used is deny
let config = load_config().unwrap();

// ✅ must handle the error path
let config = load_config().context("failed to load runtime config")?;

The architecture​

Execution Ledger runs as four Container Apps behind API Management:

The worker is the interesting piece. It uses a tokio semaphore for per-replica bounded concurrency (default 5 concurrent intents). KEDA monitors Service Bus queue depth and scales replicas. Combined, you get ~50 concurrent intent executions at the default ceiling, and you can scale higher with max replicas or the concurrency knob.

Steps within a single intent execute sequentially. This is by design: step 3 might need the file path step 2 produced. Cross-run parallelism comes from horizontal scaling via replicas.

Why Azure Container Apps​

I evaluated three deployment targets:

Container Apps won because KEDA scale rules, managed identity, and private networking come out of the box. The azd integration (azure.yaml at the repo root) means the entire infrastructure (Service Bus, PostgreSQL Flexible Server, Key Vault, App Configuration, APIM, four Container Apps, private DNS zones, Log Analytics) provisions with one command:

azd provision && azd deploy

The Drasi service runs as a singleton because PostgreSQL logical replication slots are inherently single-consumer. The scheduler runs as a singleton because it only enqueues intents; worker replicas do the real work. Both are Container Apps with maxReplicas: 1 in the Bicep template, not special infrastructure.

Why not Durable Functions?​

Execution Ledger isn't trying to replace Azure Durable Functions.

Durable Functions is excellent for event-driven serverless workflows where the orchestration model fits your problem.

Execution Ledger solves a different problem: operational recovery. Every connector step is independently replayable, idempotent, and compensatable. Instead of restarting an orchestration from a checkpoint, operators can replay exactly the failed step while preserving the rest of the execution history.

What I learned​

KEDA on Service Bus works beautifully for Rust workloads. Each replica competes for messages via ServiceBusReceiverClient. No coordination needed. The queue depth threshold of 25 messages per replica provides predictable scaling without flapping.

PostgreSQL connection pool sizing deserves its own section of the README. With N worker replicas each opening a pool, you hit max_connections faster than you think. Azure PostgreSQL Flexible Server B1ms caps at 97 connections. With 10 workers at a steady-state ~7 connections each + 3 API + 1 scheduler + 1 Drasi = 75. Leave headroom.

App Configuration with Key Vault references is the right config pattern. Bootstrap env vars carry only the App Configuration endpoint and environment label. Everything else (queue names, timeouts, toggles) lives in App Configuration by label. Key Vault references (<secret:my-secret>) resolve at startup via managed identity. The process fails closed if any required secret is missing. No connection strings. No access keys. No shared secrets in env vars.

The doc-comment-as-specification pattern works for Rust connector code. Every connector module has a //! header explaining its configuration contract, fail-closed behaviour, and compensation model. Here's the rest connector header:

//! REST and ERP connectors.
//!
//! | Key | Source | Required | Default | Notes |
//! |-----|--------|----------|---------|-------|
//! | `url` | config or input | yes | — | Target URL |
//! | `method` | config or input | no | `POST` | HTTP method |
//! | `timeoutMs` | config or input | no | `30000` | Clamped to 120s max |
//! | `headers` | config | no | — | Custom headers (some denylisted) |
//!
//! ## Fail-closed behavior
//! - Returns `InvalidConfiguration` if `url` is missing.
//! - Returns `InvalidConfiguration` if any denylisted header is present.
//! - Returns `DependencyFailed` if the HTTP request fails.

This doubles as documentation and as a specification for anyone adding a new connector.

Is this for you?​

Execution Ledger fits when you need durable, auditable job orchestration with built-in recovery controls. It's not for serverless event-driven workflows (use Durable Functions) or complex long-running workflows with custom activity patterns (use Temporal). It's for teams running ERP integrations, queue-based processing, blob transfers, or REST orchestration that need to survive failures without manual intervention, and who want the platform to prove it can recover before the first 3am page.

References​

• Execution Ledger on GitHub
• Azure Container Apps documentation
• KEDA scalers for Azure
• Azure PostgreSQL Flexible Server

A prompt is still a dependency. Change it, and downstream behaviour can shift with no compiler warning, no failing build, and no obvious signal in the request pipeline.

Why this fails quietly​

A bad code change usually throws, times out, or fails a test. A bad prompt change degrades tone, drifts off format, starts hallucinating in a new way, or gets subtly less accurate, and every one of those failure modes returns a valid HTTP 200 with plausible-looking text. Nothing in a typical request/response cycle distinguishes a good prompt output from a degraded one. The only way to catch it is to check the output against something, before a customer does.

That is the gap an eval gate closes: a required check between "someone edited a prompt" and "that prompt is what production calls", in the same way CI sits between "someone pushed a commit" and "that commit is what is running".

The pattern​

A few design choices make this work rather than become theater:

• Fail closed: if no eval run exists yet for a prompt version, the gate blocks by default. "Assume it is fine because nobody checked" is exactly the failure mode this is meant to prevent.
• Check the latest run, not any run: a prompt that passed three edits ago does not get to ride on that old pass.
• Expose status without promotion: a read-only status check lets dashboards and operators inspect state without accidentally attempting promotion.
• Return an actionable reason: "Blocked" is not enough. "Blocked: latest eval failed on 3/40 factuality cases" gives the author somewhere to start.

Where this sits on the maturity curve​

This is one rung on a ladder, not the whole ladder:

1. No process: prompt changes go straight to production. Someone notices a regression from a support ticket or a screenshot.
2. Manual review: a second person reads the diff before it ships. Catches obvious problems, misses subtle ones, doesn't scale past a small team.
3. Automated eval gate: a fixed test set runs against every candidate prompt before promotion. Catches regressions against known cases, doesn't catch novel failure modes the test set doesn't cover.
4. Continuous eval in production: sample live traffic, run it back through evaluation, and feed regressions into the same test set that gates step 3. This is where the set gets stronger over time instead of freezing at whatever the team thought to write on day one.

Most teams building on Microsoft Foundry or a similar platform are somewhere between 1 and 2 today. Getting to 3 doesn't need custom infrastructure. It needs wiring together pieces the platform already gives you.

What steps 3 and 4 actually look like on Microsoft Foundry​

Worth being precise here, because it is easy to overstate what is automatic.

Versioning is the unit of promotion, not the prompt text itself. Every save of a prompt-based agent in Foundry creates an immutable version. You route traffic to a specific version and roll back to a prior one if the new one misbehaves. Promotion is traffic moving from version N to N+1. Rollback is moving it back.

The evaluators are built in, not something you write from scratch. Foundry ships evaluators that map directly to common failure modes: Groundedness and Relevance for RAG-style factual drift, Coherence and Fluency for format and tone drift, Intent Resolution / Tool Call Accuracy / Task Adherence for agentic behaviour, and risk and safety evaluators (hate/unfairness, sexual, violence, self-harm, prompt injection) for the failure modes that matter far more than "the wording is a bit off".

The practical CI gate mechanism is a GitHub Action, and it is honest about uncertainty. The microsoft/ai-agent-evals action (preview) takes a candidate version and a baseline version in agent-name:version form, runs both against your dataset with the evaluators you name, and returns scores with confidence intervals and significance testing. That is the difference between "up by 0.02" (likely noise) and "significantly better at p<0.05" (likely real).

The caveat is important: there is no single "promote" API that the platform automatically blocks on your own score threshold. Promotion in Foundry today is a human or CI decision informed by comparison output. So "fail closed" is still your pipeline logic. Do not merge, deploy, or re-route traffic unless the evaluation step passed.

Step 4, continuous evaluation, is largely an observability workflow, not a separate product build. Once a version is live, Foundry can sample production traffic and traces on a recurring schedule, then score with the same evaluators. Same regression question, now asked continuously.

The trap worth naming: the gate becoming a bottleneck teams route around​

A gate that is slow, opaque, or noisy on legitimate changes gets bypassed. Someone finds the emergency override, uses it "just this once", and six months later the override is the normal path and the gate is decoration. Three things help prevent that:

• Fast eval loops: if checking a candidate takes twenty minutes, people batch changes and avoid the gate. Minutes, not tens of minutes, for the common path.
• Clear failure output: a pass/fail flag with no detail teaches people to distrust the gate instead of fixing the prompt.
• An audited escape hatch: emergencies happen. The override should exist, but it should be logged, attributed, and visible.

Treat the prompt like the artifact it is​

Treat a prompt exactly like any other artefact that can change production behaviour without a compiler catching mistakes. Give it a test set, give it a gate, and default that gate to blocking until proven otherwise.

Hopefully this helps you avoid the quiet regressions that only show up after customers do, and gives you a practical path to move from manual review to repeatable evaluation gates.

1. The API surface itself might change under you​

Azure OpenAI currently has two versioning surfaces. The legacy route uses dated strings like POST {endpoint}/openai/responses?api-version=2025-04-01-preview. The newer route, /openai/v1/responses, is designed to remove that treadmill. In preview, it expects the literal api-version=preview rather than a date. At GA, api-version is no longer required.

If your client or agent framework moved to /v1/ but your config still sends a dated legacy value, you get 400 API version not supported. The value can look reasonable while still being wrong for that route.

Two checks matter here. First, rollout behavior is not fully uniform: some resources have accepted a specific dated preview value (2025-11-15-preview) instead of literal preview. Second, /openai/v1/... only exists when the "Next-generation APIs (v1 preview)" feature is enabled on the resource. If it is not enabled, you get a 404 instead of a version error.

2. Sampling parameters your code has always sent​

Reasoning models don't accept the sampling parameters chat models take for granted:

• temperature
• top_p
• presence_penalty
• frequency_penalty
• logprobs / top_logprobs
• logit_bias
• max_tokens (superseded by max_completion_tokens or max_output_tokens, see below)

Code that hardcodes temperature=0.7 because older deployments accepted it will fail against reasoning-tier models. There is no graceful fallback. The request errors.

The durable fix is to make sampling parameters conditional, not assumed. Send your normal set, and if the model rejects a non-essential field, drop it and retry.

3. Reasoning tokens draw from your visible-output budget​

This is the most confusing trap because it usually does not error. It just returns nothing.

Reasoning models spend tokens on internal reasoning before they produce any visible output. Those hidden reasoning tokens count against the same budget as the answer text: max_completion_tokens on the Chat Completions API, or max_output_tokens on the Responses API. Microsoft's own docs put it directly: max_completion_tokens is "an upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens."

If you size that budget like a non-reasoning model, you can get finish_reason: "length" with an empty response. The model spent the whole allowance reasoning and never emitted visible text. This looks like a timeout or parser issue, which makes root cause hard to spot unless you inspect completion_tokens_details.reasoning_tokens.

The fix is a floor, not a formula. Reserve materially more headroom than visible output needs, and treat that as a hard minimum for reasoning-tier deployments.

4. reasoning_effort is a real lever, and its defaults vary by model​

reasoning_effort controls how long the model reasons before responding. Most reasoning models support low, medium, and high. GPT-5 models also add minimal for latency-sensitive paths. Check support per deployment:

• gpt-5.1, gpt-5.2, and gpt-5.1-codex* support none, which explicitly disables reasoning, but on gpt-5.1 you have to pass it explicitly. Unlike earlier reasoning models, it does not default to any particular effort level.
• gpt-5-pro only supports high, and defaults to it even if you omit the parameter.
• o1-mini doesn't support reasoning_effort at all.
• Parallel tool calls aren't supported when reasoning_effort="minimal", which matters if your agent framework relies on parallel tool invocation.

For latency-sensitive interactive paths, moving to low or minimal is often your largest latency lever. Set it explicitly per deployment rather than copying values across models in the same family.

The timeout budget these interact with​

Reasoning latency must fit inside the timeout budget in front of your endpoint. In Azure Container Apps, default (non-premium) ingress timeout is fixed at 240 seconds and is not configurable. If you need longer, the supported option is premium ingress, where you can raise idle timeout (--request-idle-timeout) up to 30 minutes, with a 4-minute minimum. This is an idle timeout (no-activity window), not a hard total-duration cap, but it is the practical control that decides whether high-effort reasoning calls survive.

Before increasing reasoning_effort on an interactive path, verify what sits in front of the endpoint and whether that timeout budget matches real latency at the new effort level.

A short checklist before swapping in a reasoning-tier model​

• Confirm which API surface your client or agent framework targets, and match api-version to that surface, not to habit.
• Strip temperature and the other unsupported sampling parameters from reasoning-model call paths, or make them conditional.
• Raise max_completion_tokens / max_output_tokens with explicit headroom for hidden reasoning tokens, and check completion_tokens_details.reasoning_tokens in a test call before assuming the budget is sized correctly.
• Set reasoning_effort deliberately per deployment rather than inheriting a value from a different model in the family, and check whether your target model supports the value you're setting.
• Check the ingress or gateway timeout in front of the endpoint against the latency a higher effort level will actually produce.

They are easy to miss because older deployments often tolerated defaults that reasoning-tier deployments reject.

The fallback logic was intentional. If a model call failed, the system would return a deterministic template instead of showing the user an error. That works fine when failures are actually rare. But something had gone wrong. The fallback wasn't rare anymore - it was the only path that ever executed. And nobody was watching for it.

The slow collapse​

It wasn't one thing. There were three separate bugs, each independently harmless, each invisible on its own. Stack them together and everything - but it still looks fine from the outside.

The API version thing: Azure OpenAI migrated to a new request surface. The old path uses dated version strings like api-version=2025-04-01-preview. The new /openai/v1/... surface is different—it wants api-version=preview while it's in preview, and eventually the parameter goes away entirely. A client configured with the old dated string hitting the new surface gets a 400 API version not supported. Except it doesn't fail loud. The exception gets caught and swallowed. The fallback fires.

Then there was the parameter mismatch. Reasoning models (o-series, GPT-5, future ones) drop support for temperature, top_p, presence_penalty, and a bunch of other sampling params. Code that hardcodes temperature=0.7 because that was always safe breaks immediately when the deployment switches to one of these models. Again: the request fails, gets caught, fallback executes.

The third one was the config issue. The system used a prefix-trimming config loader. Useful feature: strip App1/ off every key, so code queries message instead of the full App1/message. Except in this codebase, every reader was querying the full untrimmed key. So every read missed. Every read fell back to its hardcoded default. Model routing config, prompt versions, retry budgets—all silently reverted to defaults. No error, because a missing key with a default isn't treated as a failure.

Individually, any one would have been caught in testing. Together, they created a system that looked completely fine while doing none of what it was designed to do. The kind of bug you only catch by accident.

How it stayed invisible​

Each failure was caught and logged—at a verbosity level nobody monitors in production. The fallback was doing its job: catch the exception, return the template, keep the response flowing. From a monitoring dashboard, this looks perfect. 200s coming back, response times good, no error spikes.

Your observability is built to catch failures. Not wrongness. A system that breaks and admits it will get your attention. A system that breaks and lies will not.

The one signal that mattered​

Fast response time plus total_tokens_used: 0 in the telemetry. That's the signature. Real inference against a reasoning model takes real time and burns real tokens. A lot of them. The model spends tokens on reasoning steps you never see. A fallback returns instantly and never touches the model. Zero tokens.

If you're logging token usage per request, this becomes a cheap way to distinguish "actual inference" from "template." Except you have to know to look.

Diagnosing this required getting into the running container and calling the agent manually with the prod credentials. The health check doesn't catch it. Once I fixed the API version problem and the actual model call went through, the system still wasn't working right. The config was still defaulting everything. That took another pass through the config loader to spot.

What I'd do differently​

Validate at the config boundary. Wherever your client config meets the SDK, actually check that the values are legal for the SDK you're using. Not just syntactically valid—actually accepted by this version of this model. A version mismatch should fail at startup, loudly, not hide inside exception handling in production.

Make fallback usage visible. Don't lump fallback executions into your generic error log. Emit a distinct counter every time it happens. Then alert on the rate. A system that falls back once a day is resilient. One that falls back 99% of the time and never mentioned it is busted.

Log token usage and alert on zero. If your SDK exposes usage metrics, log them with every response. Set an alert threshold on responses that used zero tokens (outside of intentional cache hits). It's one of the cheapest correctness checks you can add.

What this changes about well-architected​

The standard resiliency review checks whether your system degrades gracefully. Necessary. Not sufficient for LLM-backed systems. Graceful degradation and silent correctness failure look identical on a dashboard that only tracks uptime and latency.

Add this question instead: can this system be wrong in a way that looks fine? For AI systems, that means: when a model call fails, does the fallback emit a signal? Is anyone actually watching it? Because a monitored fallback is resilience. An unmonitored one is just a way to be confidently wrong.

Microsoft Foundry Hosted Agents went generally available in July 2026, and it is built for that shape.

I built a service (product) that takes standard methodologies and frameworks, then supports them with agents (without giving away too much, I will keep this general). Six agents ran on Microsoft Foundry Hosted Agents, and one ran in a sidecar container.

This post is about the hosting decisions, protocol choices, and what I learnt while building that service, where the agents were built with Microsoft Agent Framework in Python.

It also builds on a few related posts: Get Ahead with Self-Hosted Agents and Container Apps Jobs, Getting Started with Azure Developer CLI (azd), and Running Azure SRE Agent for AKS and Drasi Operations.

In short: this is a production story about choosing the right compute and protocol for each agent shape, not a generic tour of features.

TL;DR​

• Hosted Agents are excellent for bursty or scheduled workloads.
• Use Responses only when you genuinely need conversation history.
• Most task-oriented agents are better suited to Invocations.
• Size compute per agent rather than per application.
• Agents designed around in-process callbacks will not migrate cleanly.

What Microsoft Foundry hosted agents give you​

The model is a VM-isolated sandbox created per session, on demand. It runs for the life of the session, then tears down. Two numbers define the economics:

• 15-minute idle timeout. Compute is deprovisioned after 15 minutes of inactivity, and there is no cost while an agent is not serving requests.
• 30-day maximum session lifetime. Long-running work is a first-class case, not something you fight the platform over.

The part that makes the idle timeout usable is that session contents survive it. Files written under $HOME are preserved when compute is deprovisioned and restored when the session resumes, so an idle gap is not a cold start from nothing.

The identity story is the bit I liked the most for this solution - the platform creates a dedicated Microsoft Entra agent identity for each hosted agent at deploy time, as a service principal the running container uses to call models and tools. You do not wire managed identities by hand. It can reach model inferencing and session storage by default, and for anything external (ie your own storage account) you can assign RBAC to that agent's identity. Six agents means six identities, which shrinks the blast radius of any one of them considerably.

Telemetry is on by default too. The platform injects an Application Insights connection string and OpenTelemetry tracing without custom wiring, alongside a set of reserved variables:

The FOUNDRY_ prefix is reserved, and you should not redeclare any of these in your own configuration.

Pick your protocol, and pick it on interaction shape​

Containers can talk to the Foundry gateway through a protocol library, and this is where my original plan was wrong.

Both protocol libraries are in preview (the hosting platform itself is GA). They work, but the API surface can still change between beta releases.

I initially mapped all six Agents to Responses, because "agent" and "chat" are easy to conflate. That was wrong for half of them. synthesis, simulation, and board-export specific Agents were more controlled and structured: typed JSON in, typed JSON out, it had no conversation and no history to manage. So I moved to Invocations, along with a cadence-driven follow-up. Only a conversational-qa genuinely needed the Responses protocol.

A useful heuristic: choose on interaction shape, not an "agent". If there is no multi-turn history worth the platform managing, Invocations is simpler, a single container can expose more than one protocol by declaring them in the protocols field, so this is not a one-way door.

Sizing is per agent​

Compute is allocated per agent as a CPU and memory pair, so sizing is an agent-by-agent decision rather than a platform-wide one. This is how mine looked:

Because compute scales to zero, the sizing decision is about the shape of a single invocation, not about what you are willing to pay to leave running!

Note the board-export line. I deliberately kept PDF and PowerPoint rendering (which is what this particular agent was helping with) out of the agent container and in a separate service, so the agent stays inside a modest allocation and the rendering can scale on its own terms.

The one agent I could not host​

There was one agent I wasn't able to host in Foundry Hosted Agents: a pre-mortem agent.

It used a run() method that takes a cycle_runner: Callable[[int], Mapping], a Python callback the loop invokes on each cycle. That works fine in-process. It cannot work across the Hosted Agent RPC boundary, because you cannot serialise a Python callable and send it over the wire. The agent had to be reworked into an event-driven model, with cycle steps published through a queue or a change-detection reaction, before it can be hosted at all.

The general form of this: an agent that accepts a function as a parameter is quietly assuming it shares a process with its caller. That assumption is invisible in an architecture diagram, and it survives every unit test, right up until the agent becomes a container behind an RPC boundary. If you are planning a move to hosted compute, make sure to check your agent entry-points first and look for anything that is a callable, an open handle, a live connection, or a shared in-memory object. Those are your migration blockers, and they are much cheaper to find in an afternoon of reading than in a sprint of rework!

Deploying with azd​

I use Azure Developer CLI heavily in my projects - especially personal projects like this one. If you want the wider AZD view, I covered that in Getting Started with Azure Developer CLI (azd). It makes redeployment easy, helps reduce Azure spend by deploying only when needed, and is a good pressure test for what a workload actually needs.

A simple:

azd deploy

This can provision and deploy an entire workload. In my service, it also built container images remotely in Azure Container Registry (so I did not need local Docker), pushed them, created a hosted agent version, created the dedicated Entra agent identity, and assigned the RBAC that identity needed. Each deploy creates a new version, previous versions are preserved, and the latest is active by default. azd ai agent show prints the name, version, protocols, container resources, and environment variables.

Agents were declared as a service in azure.yaml:

services:
  my-agent:
    host: azure.ai.agent
    env:
      MODEL_DEPLOYMENT_NAME: gpt-5-mini
      GITHUB_TOKEN: ${{connections.agent-secrets.credentials.github_token}}

That second variable is worth calling out. Rather than baking a secret into the image or the YAML, you can reference a Foundry project connection with a ${{connections.<name>.<path>}} placeholder, and the platform resolves it at sandbox start. A GET on the agent version returns the literal ${{...}} text, so the resolved secret never comes back through the management API.

A few things worth mentioning that will cost you an afternoon if you miss them:

• Images must be x86_64 (linux/amd64). On Apple Silicon, build with docker build --platform linux/amd64 . or you will produce an image the platform cannot run.
• You need the Foundry Project Manager role at project scope to deploy (recently renamed from Azure AI Project Manager, so both names are still floating about).
• Putting the container registry behind a private endpoint only works for Foundry projects created after 25 June 2026. Older projects need the registry reachable publicly so the platform can pull.
• For the SDK path, azure-ai-projects 2.3.0 or later.

For me, I even used Azure Linux 4 Beta, mostly for one reason: because I could.

Final thoughts​

For bursty and cadence-driven agents, per-session scale-to-zero and a per-agent identity remove a real pile of undifferentiated work: replica counts, warm pools, and identity wiring you would otherwise need to own.

If you are evaluating Hosted Agents today, my recommendations are:

• Start with one task-oriented agent and validate end-to-end behavior in production-like traffic.
• Default to Invocations unless conversation history clearly adds value.
• Keep heavy rendering and binary processing outside the hosted container.
• Design agents around message contracts and events, not callbacks.
• Treat each agent as an independently deployable workload with its own identity and sizing.

Hopefully this helps if you're weighing up Hosted Agents for real workloads.

References​

• Hosted agents in Foundry Agent Service
• Deploy a hosted agent
• Manage hosted agent sessions
• Introducing the new hosted agents in Foundry Agent Service
• Azure Update 563546: Agent Harness GA

This is what happened, why, and how I got out of it.

Update (2026-07-23): Microsoft has clarified the add-on behavior and support matrix. If you use the AKS add-on path, ALB controller versions are pinned by cluster version + Managed Gateway API bundle support. See the official matrix: Supported Kubernetes versions for Gateway API bundle versions.

What I saw​

kubectl get gateway -n wc2026
# NAME             CLASS                ADDRESS   PROGRAMMED   AGE
# wc2026-gateway   azure-alb-external             Unknown      12h

Front Door endpoint returning 504:

curl -s -o /dev/null -w "%{http_code}" https://wc2026-api-etckbtg6fxesbzgs.z02.azurefd.net/health/ready
# 504

The gateway existed. The AGC traffic controller was provisioned, but the ALB controller had never reconciled the Gateway resource - it was sitting there doing nothing.

Incident timeline (original run)​

The sequence below is the exact chronology from my original incident run.

The deployment sequence looked fine on paper:

1. Deploy AGC traffic controller via Bicep -- succeeds
2. Deploy Gateway API + HTTPRoute manifests -- applied
3. Enable ALB controller add-on: az aks update --enable-application-load-balancer
4. Error: Preview feature ApplicationLoadBalancerPreview not registered
5. Register feature, propagate provider -- add-on installs
6. ALB controller pods crash-loop:
   "no matches for kind 'ReferenceGrant' in version 'gateway.networking.k8s.io/v1'"
7. Root cause in my install path: ALB controller (v1.11.1) expected newer Gateway API CRDs
  than what my managed Gateway API install exposed at the time
8. AKS managed Gateway API add-on bundle on my cluster lagged the controller expectations
9. Try to upgrade CRDs manually -- denied:
   "managed Gateway API disallows modifying Gateway CRDs"
10. Try to disable managed Gateway API to install CRDs manually -- denied:
    "Application Load Balancer add-on requires managed Gateway API"

That last step is where it locks up completely. The ALB controller needs newer CRDs than the managed Gateway API provides. The managed Gateway API blocks you from upgrading the CRDs yourself. And you cannot remove the managed Gateway API because the ALB controller add-on depends on it.

Every door was closed.

What I did​

The only exit for this cluster was to pull both managed add-ons out and own the installation via Helm:

# Disable both managed add-ons
az aks update --resource-group rg-mvp --name wc2026-fanintel-aks \
  --disable-application-load-balancer
az aks update --resource-group rg-mvp --name wc2026-fanintel-aks \
  --disable-gateway-api

# Install Gateway API CRDs v1.5.1 manually
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml

# Install ALB controller via Helm
helm install alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \
  --version 1.11.3 \
  --set cluster.name=wc2026-fanintel-aks \
  --set cluster.resourceGroup=rg-mvp

With the right CRD version in place, the controller installed cleanly and started reconciling the Gateway resource straight away.

This Helm path is supported (not a hack) when you want ALB controller + Gateway API versions independent of cluster-version-based add-on pinning.

After the fix​

kubectl get pods -n azure-alb-system
# alb-controller-*   Running

kubectl get gateway -n wc2026
# wc2026-gateway   azure-alb-external   <ip>   Programmed   True

curl -s -o /dev/null -w "%{http_code}" https://wc2026-api-etckbtg6fxesbzgs.z02.azurefd.net/health/ready
# 200

Gateway programmed, Front Door responding. The tradeoff is that you are now managing the ALB controller version yourself - no automatic add-on upgrades, and you lose the managed identity wiring the add-on configures for you. Worth keeping in mind when you plan upgrades.

Post-update clarification (2026-07-23)​

Microsoft has now clarified that when you use the AKS add-on path, you should not have to manually manage Gateway API/ALB controller compatibility matrices. Add-on versions are pinned by cluster Kubernetes version and Managed Gateway API bundle support.

Correction from Microsoft PM: ALB Controller v1.11.x targets Gateway API v1.5.1, and v1.5.1 CRDs are recommended with v1.11+ for full functionality. Latest mappings are published in the ALB controller release notes.

Practical guidance:

• If you want managed lifecycle and automatic compatibility pinning, use AKS add-ons and follow the documented support matrix.
• If you want the latest Gateway API + ALB controller independent of cluster version, Helm is a supported self-managed path.

What I learned​

The core failure mode in my incident was managed-component version coupling during rollout timing. The key lesson for me is to choose the operating model early:

• Managed add-ons for lifecycle simplicity and built-in compatibility pinning.
• Helm for independent version control, with explicit ownership of upgrades and identity wiring.

Hopefully this saves you a few hours of digging through crash loops and error messages.

References​

• Application Gateway for Containers documentation
• ALB controller Helm installation
• ALB controller release notes
• Managed Gateway API support matrix
• Gateway API CRDs