observability

git.splent.io/splent-io/splent_feature_observability

v0.3.0 service operations

About

Errors to GlitchTip, Sentry or email, and alerts by email, Matrix or Telegram on unhandled errors, failed readiness and degraded services

Provides

Services
ObservabilityService

Commands

  • $ splent feature:observability status

    Where the errors go and who the alerts reach.

  • $ splent feature:observability test

    Send a test alert to every channel and a test error to the backend.

Run them in your workspace once the feature is installed in the selected product.

Configuration

Setting Default What it does
OBSERVABILITY_BACKEND none Where the errors go: glitchtip or sentry (both through the Sentry SDK and OBSERVABILITY_DSN), email (one mail per kind of error to OBSERVABILITY_EMAIL_TO, through splent_feature_mail), or none (the log).
OBSERVABILITY_DSN empty The DSN of the GlitchTip or Sentry project, https://<key>@<host>/<id>.
OBSERVABILITY_ENVIRONMENT empty The environment the reports are filed under. Empty means production, development or testing, after the app's profile.
OBSERVABILITY_RELEASE empty The release the reports are filed under. Empty means <product>@<version>, the version the image was built with.
OBSERVABILITY_SAMPLE_RATE 1.0 The share of errors sent to GlitchTip or Sentry, from 0 to 1.
OBSERVABILITY_TRACES_SAMPLE_RATE 0.0 The share of requests traced for performance, from 0 to 1. Off by default: a trace carries every query of the request.
OBSERVABILITY_ALERT_CHANNELS empty Who is told when something breaks: email, matrix and telegram, any of them. Empty alerts nobody; the backend still gets the errors.
OBSERVABILITY_ALERT_ON errors,readiness,degraded What is alerted on: errors (an unhandled error in a request), readiness (a check of /readyz failing, and recovering) and degraded (what a feature reports with report_degraded).
OBSERVABILITY_ALERT_DEDUP_MINUTES 60 Minutes during which the same alert is not sent again. The next one says how many were held back.
OBSERVABILITY_READINESS_GRACE_SECONDS 60 Seconds a readiness check fails before it is alerted on, so a deploy that waits a few seconds for its database alerts nobody.
OBSERVABILITY_EMAIL_TO empty The addresses of the email backend and of the email channel.
OBSERVABILITY_MATRIX_HOMESERVER empty The Matrix homeserver of the account that posts the alerts.
OBSERVABILITY_MATRIX_ROOM_ID empty The room the alerts go to, !id:server, and the access token of that account. Both in the deployment's .env.
OBSERVABILITY_MATRIX_TOKEN empty No description yet.
OBSERVABILITY_TELEGRAM_BOT_TOKEN empty The bot that posts the alerts, and the chat they go to. Both in the deployment's .env.
OBSERVABILITY_TELEGRAM_CHAT_ID empty No description yet.
OBSERVABILITY_HTTP_TIMEOUT 5 Seconds a Matrix or Telegram request may take.
OBSERVABILITY_STATE_DIR empty Where the processes of a container agree on what was alerted. Empty means a directory in the system's temporary directory.

These are the defaults. A product changes them in its own pyproject.toml, under [tool.splent.config], or in .env.deploy in production.

Dependencies

No hard dependencies.

In product lines

Extension points

Services
ObservabilityService
Routes
A refinement can add routes to its blueprint

Other features fill its hook slots. A refinement can override its templates and services or extend its models without forking it.

Compatibility

Python
>=3.13
SPLENT framework
>=1.30.0
Python packages
sentry-sdk[flask]>=2.20,<3requests>=2.33,<3