Check Managed Inference endpoint health
Check the status, success rate, and latency of a Managed Inference endpoint from the web interface or the CLI.
Check the health and performance of a Managed Inference endpoint from the web interface or the CLI.
Prerequisites
You need the following before you start:
- A running Managed Inference Job with a serving endpoint. See Create a Managed Inference Job.
- Access to the CosmicAC web interface, for the web method.
- The CosmicAC CLI installed and configured, for the CLI method. See Install the CLI.
Steps
Check the endpoints
The web interface and the CLI both report the recorded health of every deployed endpoint. CosmicAC probes the replicas on a schedule, every five minutes by default. Both methods show the result of the last probe. The web interface re-reads the results every 30 seconds while the page is open. The CLI reads once each time you run it.
Only the web interface can trigger a probe on demand, with Run health check. The CLI has no equivalent, so it shows the last scheduled result rather than a new measurement.
In the left sidebar, open Model Health. To probe every endpoint, click Run health check at the top of the page. To probe one endpoint, click Health check on its card.
Clicking the button doesn't update the figures on screen. They update on the next refresh, within 30 seconds.
CosmicAC probes a replica by sending it a real inference request, so a health check uses serving capacity. Run health check probes every endpoint at once. On a large deployment, one click sends a request to every replica you run. If you only need one endpoint, the card's Health check probes just that endpoint's replicas.
To trigger a probe from your own tooling, see Run health checks.
Read the results
Each endpoint reports its health and performance:
- Status shows
Healthy,Degraded, orDown. For what each value means, see Model health. - Success rate shows the share of requests that succeeded.
- Traffic shows the total requests handled.
- Failures shows how many of those requests failed.
- Avg response shows the average response time in milliseconds. The CLI prints it as Latency (avg).
- Last health check shows when CosmicAC last probed the endpoint. The CLI doesn't print it.
- Last updated shows when CosmicAC generated the reading. The CLI prints it as Timestamp.
The web interface and the CLI cover different time ranges, so the same endpoint can report different numbers in each. In the web interface, choose a range at the top of the Model Health page, next to Run health check. The choices are 1H, 6H, 24H, 7D, and 30D. By default, the page shows 7D. The CLI has no range option and always reports the last hour.
These figures count the health probes that CosmicAC sends, not only the requests your users make. An endpoint that serves no user traffic still records traffic here. How much depends on your probe interval, which is five minutes by default. To change it, see Model health settings.
The CLI also prints a per-replica table, with the replica ID, status, traffic, failures, and average latency for each replica. Use it to find an unhealthy replica. The web interface shows a replica health count on each endpoint card.
CosmicAC measures these metrics itself. The inference proxy times every request it forwards to the model server, records whether it succeeded, and writes the result to a store on disk. The metrics describe how the model server behaved, but CosmicAC doesn't read them from it, so they survive a restart of either one.
To chart these metrics, alert on them, or combine them with metrics from outside CosmicAC, run an observability stack such as Prometheus.
Logs are a separate system. CosmicAC collects logs for Managed Inference Jobs only when K8S_LOG_STORE_ENABLED is true. See Observability and logs.