Skip to content
This page View Markdown Open in ChatGPT Open in Claude

Health Check AI Reporting

Your APIs answer thousands of calls a day. Analytics counts every one of them, but nobody reads the dashboard at 3 a.m. Health Check AI Reporting reads it for you. On a schedule, it looks at the numbers, checks them, and asks AI to explain what it finds. Then it mails you, when you want to be told.

In short

  • A report looks at the numbers of a window of time : the last hour, the last 24 hours, the last 7 days…
  • Its checks find problems : error rates, latency, traffic that fell, servers short of memory.
  • An AI model reads the numbers and the checks, and writes what needs attention and how to fix it.
  • The mail goes every time, when something is wrong, or only for critical problems.
  • Every run is kept in the History, every call to a model in the AI calls, with its tokens and its cost.
Where The root user : Dashboard → Health Check AI Reporting. An admin : Dashboard → Operations → Health Check AI Reporting.
Whose numbers The root user : every admin, or one admin it picks. An admin : its own account.
Needs A key in AI Integration for the analysis, and the mail server of the root user (Root settings → Security Settings → SMTP) for the mails.
Schedules Every few hours, every day, some days of the week, once a month, or cron, in your time zone. At most every 15 minutes.
Costs Each report shows its tokens and its cost before it runs. A run with nothing to send asks no model.

How a run works

flowchart LR
    A[Schedule] --> B[Read the numbers of Analytics]
    B --> C[Run the checks]
    C --> D{Send a mail?}
    D -- no --> E[History : nothing to send]
    D -- yes --> F[Ask the AI model]
    F --> G[Mail : numbers, checks, analysis]
    G --> H[History and AI calls]
  1. The numbers. The report reads what Analytics collected for its window : every call, the kinds of API, the busiest, slowest and failing APIs, every server, every admin (for the root user), and the window before.
  2. The checks. They run with no AI and cost nothing. They decide the verdict : healthy, needs a look, or critical.
  3. The decision. A report which sends on problems only stops here when every check passes. No model is asked, no token is spent.
  4. The analysis. The model gets the numbers as short tables and the problems the checks found. It writes a summary, what needs attention, what goes well, and up to five actions.
  5. The mail. It holds four numbers beside the window before, the checks, the analysis, the lists of APIs and the servers, and what the model cost.

Every server looks at the reports every 30 seconds. The first one which takes a moment runs it : one run per moment, however many servers there are.

Make a report

Press New report, or start from one of the three ideas of an empty page :

Idea Runs Looks at Mails
A daily summary every day at 08:00 the last 24 hours every time, with the analysis
Tell me when something is wrong every hour the last hour on problems, not the same ones twice in 6 hours
Wake me for critical problems every 15 minutes the last 15 minutes on critical problems only

The editor has five parts.

What

Setting What it does
Looks at The window of numbers : the last 15 minutes, hour, 3, 6, 12 or 24 hours, 7 or 30 days.
Whose numbers The root user : every admin, or one admin.
What it holds The parts of the numbers : the health of Analytics, the kinds of API, the busiest, slowest and failing APIs, the most time in all, every API, the servers, the admins, the window before. Fewer parts cost fewer tokens.
Rows of a list How many APIs each list holds, 3 to 50.
Every API, at most How many APIs the list of every API holds, 10 to 100.

When

The schedules of Automatic Backup : every few hours, every day, some days of the week, once a month, or an expression of cron. Each one has its time zone. The editor says the schedule in words, and how many runs the next 30 days hold.

Checks

Each check has a threshold. 0 turns it off.

Check Default A finding
Error rate of all calls warning over 2 %, critical from 10 % the share of the calls which failed
p95 of all calls warning over 1,500 ms, critical from 4,000 ms 95 % of the calls were faster than this
One API errors over 10 %, p95 over 3,000 ms an API which fails or is slow
Counted from 20 calls an API with fewer calls raises no alarm
Server errors from 1 answer of 5xx the server failed calls
Traffic fell by 60 %, grew by 300 % compared with the window before
Servers memory or disk over 85 %, delay over 200 ms a server short of room, or saturated
Health of Analytics on processes which do not answer, mixed versions, a full pool of connections

The verdict is critical when one finding is critical, needs a look when one is a warning, healthy otherwise.

Mail

Setting What it does
To The addresses, separated by commas. Up to 20.
Send a mail Every time, When something is wrong (a warning or worse) or Only critical problems.
The same problems Problems told a moment ago are not told again before this many minutes. New problems are told at once.
Say when the problems are gone One mail when every check passes again.
Ask AI when everything is fine too Off, a mail with no problem holds the numbers and the checks only, and costs no token.
Subject starts with [Production], to tell your environments apart.

AI & prompt

Setting What it does
Ask AI to explain the numbers Off, the mail holds the numbers and the checks only.
Provider and model One of AI Integration, or the default one.
Key An admin : its own key, the one the root user shares, or its own first.
Longest answer 0 uses the setting of the provider.
Tone Concise (300 words), Detailed (600 words, every problem with its numbers) or For managers (200 words, the impact first).
Language The language the model writes in.
Prompt The prompt of API Maker, Add my instructions after it, or My own prompt.

A prompt of your own has placeholders :

Placeholder Becomes
{{data}} the numbers, as short tables
{{checks}} the problems the checks found, one a line
{{verdict}} HEALTHY, DEGRADED or CRITICAL
{{window}} the time the report covers, in words
{{scope}} every admin, or one admin
{{language}} the language of the report
{{report}} the name of the report

A prompt without {{data}} gets the data after it : a model can not explain numbers it is not given. Show the prompt of API Maker shows the prompt API Maker writes, and Start my own prompt from it copies it into the field.

instructions added to the prompt of API Maker
Mention the payment APIs first.
Compare with the targets of our SLA : 99.9 % success, p95 under 800 ms.
Say which team owns each API : /orders is the shop team, /pay the payments team.

Tokens and cost before it runs

The bottom of the editor estimates a run as you write it : the tokens the model reads, the most it may write, the cost of a run at the prices of the model, and the cost of a month of the schedule.

≈ 3.4k tokens in · up to 8k out · $0.0013 to $0.02 a run · 720 runs a month at most · $0.94 to $12.19

The month is the most it could cost : a report which mails on problems only asks the model only when a check finds one. Every mail ends with what its run cost, and the AI calls say the real numbers.

Preview makes the mail with the numbers and the checks, and costs no token. Preview with AI asks the model too. Neither sends anything : send a preview from its window when you like it.

Run now

Run now on a report runs it at once and sends its mail, whatever its checks say. Pause keeps the report without its schedule.

The AI Report of Analytics

The Analytics page has an AI Report button. It asks the model about the time range the page shows, and the admin the root user picked. Choose a model, a depth (Quick, or Every API) and a language. The report shows in the window : send it by mail from there, or press Schedule it to make a report of it.

The tabs of the page

Reports

Every report is a tile : its last verdict, when it runs next, who it mails and when, its model, and how its last run went.

History

Every run of every report : when, the verdict, whether a mail went, the model, its tokens and its cost, and what started it. Filter by report, verdict, mail and start. A row opens on its numbers and its checks. See the mail shows the mail as it was made, and Send again sends it to the addresses you write.

Token usage

The calls, the tokens read and written, and the cost of a period : by day, by purpose (reports, previews, tests, Analytics), by model and by report. The root user also sees the calls admins made with its keys.

AI calls

Every call to a provider. Filter by period, provider, purpose, status and key, or search the errors, models and reports. A row opens on its tokens, the estimate beside them, the prices, the model which answered, the tries and the error. Load more reads older calls. The clock sets how long calls and runs are kept : 90 days by default.

What is saved

What Where
The reports am_ai_report_schedules of the main database.
The runs, with their mails am_ai_reports of the database of the logs (or the main database).
The calls to the providers am_ai_calls, same place.
How long they are kept am_ai_settings of the main database.

None of them is part of git : reports belong to an installation, as its backups do.

Mail is not set up yet ?

The root user sets the mail server in Root settings → Security Settings → SMTP. Until then, reports still run : read them in the History.