Everything you need to go from signup to a working AI endpoint check with alerts. Takes about ten minutes.

1. Add an API credential

Go to Credentials → Add credential and paste an OpenAI or Anthropic API key. The key is validated with a minimal test call, encrypted at rest, and shown only by its last four characters from then on. Your key is used exclusively to run the checks you configure — see Security.

2. Create an endpoint

Under Endpoints, define what to call: the provider (OpenAI-compatible or Anthropic), base URL, model name, and default parameters like temperature. Point the base URL at the provider directly, or at your own API if it speaks the same protocol — that way you monitor your production stack, not just the provider.

3. Write a golden prompt

Under Prompts, create the message your check sends on every run. Good golden prompts have stable, checkable answers:

System: You are the support assistant for Acme. Answer only
questions about Acme products. Reply in JSON:
{"intent": "...", "answer": "..."}

User: What is your refund policy?

Use Test prompt to run it once and see the live response before you commit to assertions.

4. Create the AI monitor and add assertions

Monitors → Add monitor → AI endpoint. Pick your endpoint and prompt, then attach assertions. A typical set for the prompt above:

valid_json — the response must parse as JSON.
json_path_equals — path intent, expected refund_policy.
contains — the answer must mention "30 days".
not_contains — must not mention a competitor or the phrase "I don't know".
latency_under — 10,000 ms.
cost_under — keep per-check spend bounded.
llm_judge_score — rubric "Score 1-5 on whether this correctly explains a 30-day refund policy", threshold 4.

Every assertion form has Test against current response, so you can iterate without waiting for the next scheduled run. LLM responses are noisy by nature: raise the monitor's retry count on flaky prompts so a single bad sample re-runs before an incident opens.

5. Route the alerts

Add an alert channel under Settings → Alerts (email, Slack webhook, or Discord webhook) and attach rules to the monitor: alert on down, degraded, or recovered, with an optional delay for escalation. Duplicate alerts are suppressed and recoveries are batched automatically.

HTTP and SSL monitoring

Monitors → Add monitor → HTTP. URL, expected status codes, optional headers. Failures retry with backoff before an incident opens, and Cloudflare challenge pages are classified as degraded rather than down. SSL expiry is extracted automatically from HTTPS checks.

Cron / heartbeat monitoring

Monitors → Add monitor → Heartbeat. Set the expected interval and grace period, then have your job ping the generated URL:

# end of your cron job
curl -fsS https://ping.uokay.app/<your-uuid>

No ping within interval + grace opens an incident and alerts you.

Import existing Promptfoo evals

Already have a promptfoo config? Monitors → Add monitor → AI eval suite accepts your existing YAML — prompts, providers, and asserts are previewed and imported as a scheduled suite.

Questions

Email [email protected] — replies from the person who built the product.

Documentation