I spent the better part of a day convinced that Azure Monitor SLI alerting could not be deployed as infrastructure as code. I wrote a parallel implementation to work around it.

I was wrong, and the way I was wrong is the useful part.

Azure Monitor now does SLIs and SLOs natively, with error budgets and burn rate alerting, which is good news if you have been hand-rolling multi-window burn rate rules out of metric alerts for years. I have been building a demo that pairs Azure Monitor SLIs with the Azure SRE Agent on AKS, all deployed with the Azure Developer CLI, and I wanted every part of it in infrastructure as code.

Getting the infrastructure up was the easy part. Getting a single SLI created against Managed Prometheus metrics took considerably longer, and almost none of the reasons were in the documentation. This post is the list of things I wish I had known before I started, in the order they bit me.

If you only read one line: the metric namespace for Managed Prometheus metrics is customdefault, not prometheus. That one value cost me the most time by a wide margin.

Most Kubernetes monitoring tells you that a pod restarted. It rarely tells you whether a customer noticed, which customers were affected, or whether the business promise is now at risk.

That gap is a product and service-design problem as much as a monitoring problem. The people responsible for a payment journey need to know whether checkout works, how quickly it works, and which customer segment is paying the cost of an incident. The on-call engineer needs enough evidence to choose the right first action. The business needs a trustworthy way to decide when an error budget has been spent.

I wanted to see whether an Azure Monitor SLI could give the Azure SRE Agent enough context to investigate an Azure Kubernetes Service failure, rather than just react to a pod alert. I built a small payment service, chose signals around the customer journey, and deliberately broke it.

The result is an infrastructure-as-code deployment that runs with azd up. It measures availability, latency, the full request journey, the public path, and customer tiers. Azure Monitor alerts on those service-level signals, and the Azure SRE Agent investigates the incident and, when allowed, fixes it.

The important result is not the number of Azure resources. It is the chain from customer need to operational decision:

I got several things wrong. The corrections stay in the post because they expose the difference between a design that looks plausible and one that has been tested with the people, decisions, and failure modes it is meant to serve.

Every codebase that has lasted long enough builds up at least one layer like this. Request stubs call into a wiring dict: a lookup table of small lambda functions, one per route, that exists only to find the right handler. Those lambdas call translation functions that move data between an old dispatch shape and what the framework now wants on its own. It worked when it was written. Every new feature since has paid a cost: it goes through three extra steps to do something the framework could do directly if you let it.

Execution-Ledger is an open-source durable workflow orchestrator I built in Rust on Azure Container Apps. It handles step-level idempotency, targeted replay, and compensation for long-running jobs that span external systems. This post is about why I chose Rust, why I chose ACA, and what I learned along the way.

I’ve spent a lot of time around orchestration frameworks. Azure Durable Functions doesn’t automatically rewind entire orchestrations when a single step fails; instead, the orchestrator fails and previous steps remain executed, with "rewind" only available as a manual, preview‑level recovery that replays from the failure point. Temporal demands a Kubernetes cluster and a team comfortable with its programming model. And then there’s the dark matter: the cron scripts and PowerShell scheduled tasks that run half the back offices I’ve seen, failing silently at 3am until someone notices the queue depth.