Skip to content
Agentic SDLC

Observability with AI coding agents: instrumentation briefs, read-only telemetry and alerts for Claude Code and Codex

A coding agent adds monitoring to a slow export job in twenty minutes. A week later the metrics bill has tripled, a CPU alert wakes someone at 3 a.m. for nothing, and the real outage pages nobody. This playbook gives agents an instrumentation brief, read-only access to telemetry and alert rules that a person on call approves.

Sydla Journal cover: observability with AI coding agents, showing an error budget chart crossing a 14.4x burn-rate line, a checklist of span, histogram, bounded labels and runbook, and an on-call owner's approval

Monitoring in twenty minutes. Then the bill and the pager arrived

The ticket says: the invoice PDF export is slow and sometimes fails, add monitoring. A coding agent wraps the job in a timer, adds a metric with the customer id, the user id and the file name as labels, logs every request body for good measure and creates an alert on CPU above 80%. The merge request is small, tidy and green.

A week later three things have happened. The metrics bill has tripled, because a label per user turned one metric into forty thousand time series. The CPU alert has paged the on-call engineer four nights in a row for batch jobs that were working as designed. And when the export queue actually stalled on Tuesday, nothing paged at all, because a stalled queue uses no CPU. The logs also contain customer addresses that were never supposed to leave the database.

None of this is the agent being careless. Nobody told it which question the monitoring should answer, what the service level objective is, which labels are allowed or who owns the pager. That context is scattered: the ticket in Jira, the runbook in Confluence, the on-call notes in Notion, the job's screens and error states in Figma, the architecture sketch on a Miro board and the status page graphics in Canva. Six tools, six context hops, up to six AI credit meters. Our AI credit fatigue piece adds up what that costs.

This playbook sorts observability work into lanes by risk, gives the agent a seven-field instrumentation brief, connects it to telemetry read-only, and keeps alert rules and anything that touches production with the people who carry the pager.

What changed for observability and agents in 2026

This was the year observability vendors plugged their data straight into coding agents. The facts worth knowing:

  1. Datadog's MCP server is generally available. Since 9 March 2026 it gives Claude Code, Codex, Cursor, GitHub Copilot and other clients live logs, metrics and traces. You choose toolsets such as apm, logs or alerting, and some tools write: creating a monitor makes a draft that sends no notifications, and admins can switch write tools off for the whole organization. There is a fair-use limit of 50 tool calls per 10 seconds and no monthly cap, but the Bits chat tools spend AI credits.
  2. Grafana shipped an MCP server and an agent CLI. On 27 July 2026 Grafana Labs made the Grafana Cloud MCP server and gcx generally available. The MCP server exposes dashboards, alert rules, incidents and data sources; gcx manages dashboards and alert rules as code, with agent-friendly output for Claude Code, Copilot and Cursor.
  3. Honeycomb and Sentry host their own. Honeycomb's hosted MCP server signs in through the browser instead of API keys, and Sentry runs a hosted server for errors and issues. Most observability tools you already pay for can now answer an agent's questions directly.
  4. Coding agents are observable too. Claude Code exports OpenTelemetry metrics and events when CLAUDE_CODE_ENABLE_TELEMETRY is set: token usage, estimated cost, lines changed, commits and pull requests. Prompt and tool content stay out unless you opt in, and exporter settings in a repository's .claude/settings.json are ignored, so a cloned repo cannot redirect your telemetry.
  5. OpenTelemetry's GenAI conventions are still moving. The gen_ai attributes remain in Development status and moved to their own repository in June 2026. Pin the version you emit and expect renames.

The short version: an agent can now read production telemetry as easily as it reads your code. That is a gift for debugging and a risk for everything else, which is why the access level belongs in the brief.

Step 1: sort observability work into four lanes

Reading a trace and muting a production alert are both observability work, and they carry very different risks. A person picks the lane when the ticket is written:

  1. Read. The agent queries logs, traces and metrics to answer a question or debug a bug. Read-only tokens, no writes, findings attached to the ticket. The agent can do this alone.
  2. Instrument. The agent adds spans, metrics and structured logs to code. It goes through normal code review and must be proven in staging with real queries, not only unit tests.
  3. Alert. New or changed alert rules, dashboards and SLOs, kept as code. The person who will be paged approves every rule and its runbook.
  4. Act. Anything that changes production: muting, scaling, rolling back, deleting data or dashboards. The agent may propose the action with its evidence. A person does it.
Four observability lanes for AI coding agents. Read: query logs, traces and metrics read-only, the agent can do it alone. Instrument: add spans, metrics and logs, proven in staging, normal code review. Alert: rules as code, burn-rate based, backtested, approved by the on-call owner. Act: muting, scaling or rolling back production is proposed by the agent and done by a person. Below, the six-tool stack at $83.85 per user per month against Sydla at $19.
Agents read freely. People own the pager.

The lanes map onto MCP permissions. Give the agent's MCP connection only the toolsets its lane needs, and leave write tools off unless the lane is Alert, and even then only for drafts. Our MCP servers guide covers scoping, and the prompt injection guide explains why log lines written by users are untrusted input for an agent.

Step 2: the seven-field instrumentation brief

For Instrument and Alert work, add these seven fields next to the acceptance criteria:

  1. 1. The question. What should someone be able to answer after this ships? For example: is the export slower than usual, for whom, and at which step?
  2. 2. Service level objective. The user-facing promise, such as 99.5% of exports finish within 60 seconds over 30 days. Alerts come from this, not from CPU or memory.
  3. 3. Signals and names. Which spans, metrics and log events, named after your conventions and OpenTelemetry's where they exist. One histogram for duration beats five gauges.
  4. 4. Label budget. The allowed labels and their maximum values: plan, region, status, step. Never user id, email, request id or file name on a metric; those go on traces or logs.
  5. 5. Data rules. What must never be logged: personal data, tokens, request bodies, file contents. Sampling for high-volume paths.
  6. 6. Alert and runbook. Which condition pages, which opens a ticket, who owns it, and the runbook link that goes in the alert. No alert without a runbook.
  7. 7. Done means. The signals visible in staging, queried by the agent with the results attached, the alert backtested on the last 30 days, and the on-call owner's approval recorded.

Field 4 matters most with agents. A model asked to make a metric useful will add every identifier it can find, because more labels look more helpful. Cardinality is invisible in a diff and expensive on the invoice. Our acceptance criteria guide explains why agents need limits they can check, and the PRD template shows where the brief sits.

Step 3: alerts from SLOs, not from symptoms

Agents write alert rules quickly, so the quality bar has to be explicit. Google's SRE Workbook gives a tested default for a 99.9% SLO, using burn rate: how fast you are spending the error budget compared with a steady month.

  1. Page at 14.4x burn over both the last hour and the last 5 minutes. That is 2% of the monthly budget gone in an hour.
  2. Page at 6x burn over both the last 6 hours and the last 30 minutes, 5% of the budget.
  3. Open a ticket at 1x burn over both the last 3 days and the last 6 hours, 10% of the budget.

The short window makes the alert stop soon after the problem stops; the long window keeps one bad minute from waking anyone. Ask the agent to backtest every new rule against the last 30 days of data through the read-only MCP connection and attach how many times it would have fired. A rule that would have paged twelve times last month without an incident is not ready. Pair it with the incident postmortem playbook for what happens after the page, and the performance regression guide for budgets on latency.

Where observability context lives today, and what it costs

The agent's raw material is spread across the usual stack, before the observability tool itself. Here is one paid seat of each next to Sydla. Compare them one by one on our comparison pages.

One paid seat per tool, monthly USD list prices before tax, checked on 14 November 2026. AI limits checked on 20 November 2026.
Tool and planSeat / monthBuilt-in AI billingObservability context it holds
Jira Standard$9.70Rovo credits: 25 per user per month on Standard, 70 on Premium, 150 on Enterprise; $0.01 per extra credit from 3 December 2026The ticket and the incident follow-ups
Confluence Standard$7.15Rovo, same credit poolRunbooks and SLO definitions
Notion Plus$12.00Notion AI and Custom Agents on Notion creditsOn-call notes and handovers
Figma Professional, Full seat$20.003,000 AI credits a monthError and slow states users see
Miro Starter$10.00Create with AI on Miro creditsArchitecture and data flow sketches
Canva Business$25.00Canva AI usage per memberStatus page and incident graphics
Six-tool total$83.85Up to six meters, plus your coding agent and observability toolSix places to look
[Sydla](/)$19.00None: your own Claude Code or Codex plan pays for the model, with no markupOne work item: brief, SLO, states, runs and evidence

That is $64.85 per user per month less than the six-tool stack, before any AI overage. Your observability platform stays where it is: the agent reaches it through its own MCP server. The pricing section has the details.

Walkthrough: brief, UX Studio, your agent in Sydla Desktop, review

Here is how an instrumentation change moves through Sydla on one work item, with the coding agent you already pay for.

  1. Brief on the work item. The ticket carries the lane, the acceptance criteria and the seven instrumentation fields, written once and read by every agent and person.
  2. States in UX Studio. What users see when the job is queued, running, slow, failed or retried. Each state the user can see should be measurable, so the states tell the agent where spans and status labels belong. Our design handoff guide explains why states beat screenshots.
  3. Run in Sydla Desktop with your own agent. Sydla Desktop launches the Claude Code or Codex CLI on your machine, signed in with your own Anthropic or OpenAI account. The agent reads the work item through the Sydla MCP server and, if you connect it, your observability tool through its own MCP server with read-only toolsets. It adds the signals, deploys to staging, queries them back, backtests the alert and attaches the results to the run. There are no Sydla AI credits and no token markup. The Desktop guide walks through the first run.
  4. Review and sign-off. A reviewer checks the label budget and that nothing sensitive is logged; the on-call owner approves the alert and the runbook, and the result is recorded on the work item before merge. Our AI pull request review guide covers the rest of the review.
An observability change in four steps. Brief: the work item holds the lane, the SLO, the signals and the alert rule. States: UX Studio holds the user-facing states the signals must cover, including slow and failed. Run: your own Claude Code or Codex in Sydla Desktop adds spans and metrics, reads staging telemetry through a read-only MCP server and backtests the alert. Review: the on-call owner checks cardinality, logged data, the alert and the runbook, then signs off.
Signals in the code. Pages with a person.

Agents add signals and read data. Rules in the brief keep the bill and the pager sane. People who carry the pager decide what wakes them up.

A filled-in example: the invoice PDF export job

Here is the opening ticket, rewritten as an instrumentation brief. The service and numbers are illustrative:

  1. Lane: Instrument, plus Alert.
  2. Question: Are exports slower or failing more than usual, for which plan and region, and at which step: queue, render or upload?
  3. SLO: 99.5% of exports ready within 60 seconds, over 30 days.
  4. Signals: a span per step under one export trace; a duration histogram and a result counter (ok, failed, timed out); queue age as a gauge.
  5. Label budget: plan (4 values), region (3), step (3), result (3). Customer and user ids on spans only.
  6. Data rules: no request bodies, addresses or file contents in logs; one structured log line per failed export with the trace id.
  7. Alert: page at 14.4x and 6x burn with the export runbook linked; ticket at 1x; queue age above 10 minutes opens a ticket.

In this example the agent's first run does what the agent in the opening did: a user id label and a CPU alert. The staging check catches it, because the agent's own query shows about 40,000 series for one metric, and the backtest shows the CPU rule firing on 19 nights without a single failed export. The agent moves identifiers to spans, switches to the burn-rate rules, and the new backtest fires twice, both on the real stall from last Tuesday. The reviewer removes one leftover debug log with a customer email. The on-call owner rewrites the first line of the runbook and approves. Our bug report template shows how to brief the next export failure, with the trace id from the log line attached.

Frequently asked questions

Can AI coding agents add monitoring and alerts?

Yes, when the brief is specific. Claude Code and Codex add OpenTelemetry spans, metrics and structured logs reliably, and can write alert rules as code. Without an SLO, a label budget and data rules they tend to add high-cardinality labels, log sensitive data and alert on CPU instead of user impact, so a person on call should approve every alert.

Should a coding agent have access to production telemetry?

Read-only access is usually safe and very useful for debugging: Datadog, Grafana, Honeycomb and Sentry all offer MCP servers. Limit the agent to the toolsets it needs, turn write tools off, and treat log content as untrusted input, since users can write text that ends up in logs.

What is an SLO burn-rate alert?

An alert that fires when the error budget of a service level objective is being spent faster than normal. Google's SRE Workbook recommends, for a 99.9% SLO, paging at a 14.4x burn rate over 1 hour and 5 minutes, paging at 6x over 6 hours and 30 minutes, and opening a ticket at 1x over 3 days and 6 hours.

Can I monitor Claude Code itself?

Yes. Claude Code exports OpenTelemetry metrics and events when CLAUDE_CODE_ENABLE_TELEMETRY is set to 1, including token usage, estimated cost, lines of code, commits and pull requests. Prompts and tool content are not exported unless you opt in.

How much does Sydla cost compared with Jira, Confluence, Notion, Figma, Miro and Canva?

Sydla costs $19 per user per month. One paid seat each of Jira Standard, Confluence Standard, Notion Plus, Figma Professional Full, Miro Starter and Canva Business costs $83.85 per user per month, so the saving is $64.85 per user per month. You bring your own Claude Code or Codex account, so your Anthropic or OpenAI plan pays for the model with no markup.

Keep reading.

All articles

ONE WORKSPACE. THE ENTIRE SDLC.

Your next project.
One $19 subscription.

Keep using your Claude Code and Codex accounts. Bring the work into Sydla.

Get SydlaUSD / user / month · Billed monthlyProvider subscriptions remain separate.