Observability with AI coding agents: instrumentation briefs, read-only telemetry and alerts for Claude Code and Codex
A coding agent adds monitoring to a slow export job in twenty minutes. A week later the metrics bill has tripled, a CPU alert wakes someone at 3 a.m. for nothing, and the real outage pages nobody. This playbook gives agents an instrumentation brief, read-only access to telemetry and alert rules that a person on call approves.

Monitoring in twenty minutes. Then the bill and the pager arrived
The ticket says: the invoice PDF export is slow and sometimes fails, add monitoring. A coding agent wraps the job in a timer, adds a metric with the customer id, the user id and the file name as labels, logs every request body for good measure and creates an alert on CPU above 80%. The merge request is small, tidy and green.
A week later three things have happened. The metrics bill has tripled, because a label per user turned one metric into forty thousand time series. The CPU alert has paged the on-call engineer four nights in a row for batch jobs that were working as designed. And when the export queue actually stalled on Tuesday, nothing paged at all, because a stalled queue uses no CPU. The logs also contain customer addresses that were never supposed to leave the database.
None of this is the agent being careless. Nobody told it which question the monitoring should answer, what the service level objective is, which labels are allowed or who owns the pager. That context is scattered: the ticket in Jira, the runbook in Confluence, the on-call notes in Notion, the job's screens and error states in Figma, the architecture sketch on a Miro board and the status page graphics in Canva. Six tools, six context hops, up to six AI credit meters. Our AI credit fatigue piece adds up what that costs.
This playbook sorts observability work into lanes by risk, gives the agent a seven-field instrumentation brief, connects it to telemetry read-only, and keeps alert rules and anything that touches production with the people who carry the pager.
What changed for observability and agents in 2026
This was the year observability vendors plugged their data straight into coding agents. The facts worth knowing:
- Datadog's MCP server is generally available. Since 9 March 2026 it gives Claude Code, Codex, Cursor, GitHub Copilot and other clients live logs, metrics and traces. You choose toolsets such as apm, logs or alerting, and some tools write: creating a monitor makes a draft that sends no notifications, and admins can switch write tools off for the whole organization. There is a fair-use limit of 50 tool calls per 10 seconds and no monthly cap, but the Bits chat tools spend AI credits.
- Grafana shipped an MCP server and an agent CLI. On 27 July 2026 Grafana Labs made the Grafana Cloud MCP server and gcx generally available. The MCP server exposes dashboards, alert rules, incidents and data sources; gcx manages dashboards and alert rules as code, with agent-friendly output for Claude Code, Copilot and Cursor.
- Honeycomb and Sentry host their own. Honeycomb's hosted MCP server signs in through the browser instead of API keys, and Sentry runs a hosted server for errors and issues. Most observability tools you already pay for can now answer an agent's questions directly.
- Coding agents are observable too. Claude Code exports OpenTelemetry metrics and events when CLAUDE_CODE_ENABLE_TELEMETRY is set: token usage, estimated cost, lines changed, commits and pull requests. Prompt and tool content stay out unless you opt in, and exporter settings in a repository's .claude/settings.json are ignored, so a cloned repo cannot redirect your telemetry.
- OpenTelemetry's GenAI conventions are still moving. The gen_ai attributes remain in Development status and moved to their own repository in June 2026. Pin the version you emit and expect renames.
The short version: an agent can now read production telemetry as easily as it reads your code. That is a gift for debugging and a risk for everything else, which is why the access level belongs in the brief.
Step 1: sort observability work into four lanes
Reading a trace and muting a production alert are both observability work, and they carry very different risks. A person picks the lane when the ticket is written:
- Read. The agent queries logs, traces and metrics to answer a question or debug a bug. Read-only tokens, no writes, findings attached to the ticket. The agent can do this alone.
- Instrument. The agent adds spans, metrics and structured logs to code. It goes through normal code review and must be proven in staging with real queries, not only unit tests.
- Alert. New or changed alert rules, dashboards and SLOs, kept as code. The person who will be paged approves every rule and its runbook.
- Act. Anything that changes production: muting, scaling, rolling back, deleting data or dashboards. The agent may propose the action with its evidence. A person does it.
The lanes map onto MCP permissions. Give the agent's MCP connection only the toolsets its lane needs, and leave write tools off unless the lane is Alert, and even then only for drafts. Our MCP servers guide covers scoping, and the prompt injection guide explains why log lines written by users are untrusted input for an agent.
Step 2: the seven-field instrumentation brief
For Instrument and Alert work, add these seven fields next to the acceptance criteria:
- 1. The question. What should someone be able to answer after this ships? For example: is the export slower than usual, for whom, and at which step?
- 2. Service level objective. The user-facing promise, such as 99.5% of exports finish within 60 seconds over 30 days. Alerts come from this, not from CPU or memory.
- 3. Signals and names. Which spans, metrics and log events, named after your conventions and OpenTelemetry's where they exist. One histogram for duration beats five gauges.
- 4. Label budget. The allowed labels and their maximum values: plan, region, status, step. Never user id, email, request id or file name on a metric; those go on traces or logs.
- 5. Data rules. What must never be logged: personal data, tokens, request bodies, file contents. Sampling for high-volume paths.
- 6. Alert and runbook. Which condition pages, which opens a ticket, who owns it, and the runbook link that goes in the alert. No alert without a runbook.
- 7. Done means. The signals visible in staging, queried by the agent with the results attached, the alert backtested on the last 30 days, and the on-call owner's approval recorded.
Field 4 matters most with agents. A model asked to make a metric useful will add every identifier it can find, because more labels look more helpful. Cardinality is invisible in a diff and expensive on the invoice. Our acceptance criteria guide explains why agents need limits they can check, and the PRD template shows where the brief sits.
Step 3: alerts from SLOs, not from symptoms
Agents write alert rules quickly, so the quality bar has to be explicit. Google's SRE Workbook gives a tested default for a 99.9% SLO, using burn rate: how fast you are spending the error budget compared with a steady month.
- Page at 14.4x burn over both the last hour and the last 5 minutes. That is 2% of the monthly budget gone in an hour.
- Page at 6x burn over both the last 6 hours and the last 30 minutes, 5% of the budget.
- Open a ticket at 1x burn over both the last 3 days and the last 6 hours, 10% of the budget.
The short window makes the alert stop soon after the problem stops; the long window keeps one bad minute from waking anyone. Ask the agent to backtest every new rule against the last 30 days of data through the read-only MCP connection and attach how many times it would have fired. A rule that would have paged twelve times last month without an incident is not ready. Pair it with the incident postmortem playbook for what happens after the page, and the performance regression guide for budgets on latency.
Where observability context lives today, and what it costs
The agent's raw material is spread across the usual stack, before the observability tool itself. Here is one paid seat of each next to Sydla. Compare them one by one on our comparison pages.
| Tool and plan | Seat / month | Built-in AI billing | Observability context it holds |
|---|---|---|---|
| Jira Standard | $9.70 | Rovo credits: 25 per user per month on Standard, 70 on Premium, 150 on Enterprise; $0.01 per extra credit from 3 December 2026 | The ticket and the incident follow-ups |
| Confluence Standard | $7.15 | Rovo, same credit pool | Runbooks and SLO definitions |
| Notion Plus | $12.00 | Notion AI and Custom Agents on Notion credits | On-call notes and handovers |
| Figma Professional, Full seat | $20.00 | 3,000 AI credits a month | Error and slow states users see |
| Miro Starter | $10.00 | Create with AI on Miro credits | Architecture and data flow sketches |
| Canva Business | $25.00 | Canva AI usage per member | Status page and incident graphics |
| Six-tool total | $83.85 | Up to six meters, plus your coding agent and observability tool | Six places to look |
| [Sydla](/) | $19.00 | None: your own Claude Code or Codex plan pays for the model, with no markup | One work item: brief, SLO, states, runs and evidence |
That is $64.85 per user per month less than the six-tool stack, before any AI overage. Your observability platform stays where it is: the agent reaches it through its own MCP server. The pricing section has the details.
Walkthrough: brief, UX Studio, your agent in Sydla Desktop, review
Here is how an instrumentation change moves through Sydla on one work item, with the coding agent you already pay for.
- Brief on the work item. The ticket carries the lane, the acceptance criteria and the seven instrumentation fields, written once and read by every agent and person.
- States in UX Studio. What users see when the job is queued, running, slow, failed or retried. Each state the user can see should be measurable, so the states tell the agent where spans and status labels belong. Our design handoff guide explains why states beat screenshots.
- Run in Sydla Desktop with your own agent. Sydla Desktop launches the Claude Code or Codex CLI on your machine, signed in with your own Anthropic or OpenAI account. The agent reads the work item through the Sydla MCP server and, if you connect it, your observability tool through its own MCP server with read-only toolsets. It adds the signals, deploys to staging, queries them back, backtests the alert and attaches the results to the run. There are no Sydla AI credits and no token markup. The Desktop guide walks through the first run.
- Review and sign-off. A reviewer checks the label budget and that nothing sensitive is logged; the on-call owner approves the alert and the runbook, and the result is recorded on the work item before merge. Our AI pull request review guide covers the rest of the review.
Agents add signals and read data. Rules in the brief keep the bill and the pager sane. People who carry the pager decide what wakes them up.
A filled-in example: the invoice PDF export job
Here is the opening ticket, rewritten as an instrumentation brief. The service and numbers are illustrative:
- Lane: Instrument, plus Alert.
- Question: Are exports slower or failing more than usual, for which plan and region, and at which step: queue, render or upload?
- SLO: 99.5% of exports ready within 60 seconds, over 30 days.
- Signals: a span per step under one export trace; a duration histogram and a result counter (ok, failed, timed out); queue age as a gauge.
- Label budget: plan (4 values), region (3), step (3), result (3). Customer and user ids on spans only.
- Data rules: no request bodies, addresses or file contents in logs; one structured log line per failed export with the trace id.
- Alert: page at 14.4x and 6x burn with the export runbook linked; ticket at 1x; queue age above 10 minutes opens a ticket.
In this example the agent's first run does what the agent in the opening did: a user id label and a CPU alert. The staging check catches it, because the agent's own query shows about 40,000 series for one metric, and the backtest shows the CPU rule firing on 19 nights without a single failed export. The agent moves identifiers to spans, switches to the burn-rate rules, and the new backtest fires twice, both on the real stall from last Tuesday. The reviewer removes one leftover debug log with a customer email. The on-call owner rewrites the first line of the runbook and approves. Our bug report template shows how to brief the next export failure, with the trace id from the log line attached.
Frequently asked questions
Can AI coding agents add monitoring and alerts?
Yes, when the brief is specific. Claude Code and Codex add OpenTelemetry spans, metrics and structured logs reliably, and can write alert rules as code. Without an SLO, a label budget and data rules they tend to add high-cardinality labels, log sensitive data and alert on CPU instead of user impact, so a person on call should approve every alert.
Should a coding agent have access to production telemetry?
Read-only access is usually safe and very useful for debugging: Datadog, Grafana, Honeycomb and Sentry all offer MCP servers. Limit the agent to the toolsets it needs, turn write tools off, and treat log content as untrusted input, since users can write text that ends up in logs.
What is an SLO burn-rate alert?
An alert that fires when the error budget of a service level objective is being spent faster than normal. Google's SRE Workbook recommends, for a 99.9% SLO, paging at a 14.4x burn rate over 1 hour and 5 minutes, paging at 6x over 6 hours and 30 minutes, and opening a ticket at 1x over 3 days and 6 hours.
Can I monitor Claude Code itself?
Yes. Claude Code exports OpenTelemetry metrics and events when CLAUDE_CODE_ENABLE_TELEMETRY is set to 1, including token usage, estimated cost, lines of code, commits and pull requests. Prompts and tool content are not exported unless you opt in.
How much does Sydla cost compared with Jira, Confluence, Notion, Figma, Miro and Canva?
Sydla costs $19 per user per month. One paid seat each of Jira Standard, Confluence Standard, Notion Plus, Figma Professional Full, Miro Starter and Canva Business costs $83.85 per user per month, so the saving is $64.85 per user per month. You bring your own Claude Code or Codex account, so your Anthropic or OpenAI plan pays for the model with no markup.
































