Purchasing Guides / Production Alert Investigation

Production Alert Investigation

A purchasing guide to buying a workflow that investigates each production alert across your metrics, logs, deploys and code, and gives the on-call engineer evidence-linked likely causes.

Yearly running cost

Assuming 1,800 alerts a year

By hand: $67.5K / year; 900 hours of work
Hosted service: $11.7K – 15.2K+ / year
This workflow: $648 / year; Machine usage only; setup, hosting and review are extra.

What should it do when an alert fires?

Gather the metrics, logs, recent deploys and code changes behind an alert and give the on-call engineer an evidence-linked first investigation.

Investigate and reportRead-only investigation notes for each alert$1,224 – 3,360 setupAdd approved runbook stepsListed actions after the on-call engineer confirms$1,944 – 5,400 setup

Your on-call engineer decides what to do. No changes to production without their approval.

What is included

Receive alerts from your paging tool, run read-only queries against your metrics, logs, deploy history and code, and post likely causes ranked with linked evidence and the queries used.

Start with your alert source, read-only access to your monitoring tools and code host, existing runbooks and a set of past incidents with confirmed causes. The runbook option adds a short list of approved actions that run only after the on-call engineer confirms.

How reliable should the investigation be?

A confident answer is not enough. Choose how often the real cause should be among its likely causes, how much of what it says must be backed by evidence and how many alerts it must cover.

StandardA useful first investigation, with uncertain findings marked for the engineer.Confirmed cause found: ≥70%Findings with evidence: ≥95%IncludedStrictA second check on each finding and more replayed incidents.Confirmed cause found: ≥80%Findings with evidence: ≥98%Setup +$288 – 840My own targetsAgree your own acceptance requirements with the provider.Quote separately

How these standards are measured

Replay held-out past incidents with known causes, plus noisy alerts, missing data and alerts that look alike but have different causes. A likely cause without linked evidence counts as wrong, and any change made without approval fails the release.

Targets for your selected standard
What is checked	Target
Confirmed cause foundReplayed incidents where the cause your team confirmed afterwards is among the top likely causes, divided by all replayed incidents.	≥70%
Findings with evidenceStatements in the note that link to a query result, log line, deploy or code change which supports them, divided by all statements.	≥95%
Alerts investigated on timeAlerts with a complete note posted within the speed target, divided by all alerts received during the test.	≥95%
Changes with approvalEvery production change must have a recorded confirmation from the on-call engineer before it runs. Any exception fails the release.	≥100%

A likely cause is a lead for the engineer, not a diagnosis. Each finding must point to the query or record that supports it, and missing data is reported as missing.

How quickly should the first investigation arrive?

Choose how quickly the on-call engineer sees the first investigation note after an alert fires. Time to fix the problem is separate.

Within 5 minutesIncludedWithin 3 minutesSetup +$216 – 600Within 90 secondsSetup +$432 – 1,200

Timing details

Time from receiving the alert to a posted note, including monitoring queries, model calls, queueing and retries. Slow monitoring queries count against the target. Confirm alert volume and data sources with your provider.

The target applies to at least 95% of agreed test runs, with 3 in progress at a time.

How much do you want to spend per alert?

Choose the machine budget for querying your tools and reasoning over the results. Tighter budgets may run fewer queries per alert.

Up to $1.50IncludedUp to $0.75Setup +$144 – 480Up to $0.40Setup +$288 – 720

Cost details

Includes model calls, retries and a shared hosting allocation. Your monitoring, logging and paging subscriptions, any query or data-scan charges they bill, and engineers' time are separate. Noisy alerts and repeated runs also cost money.

Reference machine cost: $0.56 – 0.96 per alert at 150 alerts a month. The selected cap is a target to test, not a replacement for this estimate.

Where do you want it to run?

Run it in your cloud or on your own server. Choose whether log lines and metric summaries may go to an approved AI service.

Run it onAI model access

Data and access details

Runs in a cloud account you control, with access controls and logs.

Only query results needed for the investigation go to the selected external model, with secrets masked. Agree access and retention first.

Use read-only credentials for monitoring tools and code, and mask secrets and customer data in log lines before any model call. Private model only keeps telemetry on your hardware; the workflow still reads your own monitoring tools.

How do you want to use it?

Use your incident channel, your current AI agent or a dedicated on-call agent. You can choose more than one.

My incident channelEach alert gets an investigation note in your Slack or Microsoft Teams incident channel, with links to the evidence.IncludedMy existing AI agentLet your agent start an investigation, read the findings and ask follow-up questions.Setup +$72 – 240A dedicated on-call agentUse a dedicated agent that on-call engineers can question about an alert and its evidence.Setup +$144 – 480

Anything else your provider should know?

Optional. Your choices are included automatically.

Alerts per monthYour setup budget (USD)

Your alerts, monitoring tools or runbooks

Common questions

Do I need this if my monitoring tool already has an AI investigator?

Not necessarily. If your monitoring vendor's AI investigation covers your tools and its per-investigation price fits your alert volume, use it. Buy a workflow when your telemetry spans several tools, must stay in your environment or needs your own runbooks.

Will it fix incidents by itself?

No. The base option only reads and reports. The runbook option can run a short list of agreed steps, each after the on-call engineer confirms.

What if the cause is not in the data it can see?

It says which data it checked and what is missing, and leaves the cause open. It should not invent a cause to fill the note.

Can I run it without an external AI service?

Yes. Choose a private model and supply your own hardware. Measure investigation quality and running cost during the pilot.