mirror of https://github.com/makeplane/plane.git synced 2026-09-01 19:48:42 +02:00

Files

Sriram Veeraghanta b6e1f24f7c fix(api/observability): address PR review — leak, fork-safety, bounded shutdown, gate unification

Review fixes for #9419:

- Pin headers on the license telemetry exporter so it can no longer inherit
  OTEL_EXPORTER_OTLP_HEADERS and ship an operator's APM credential to
  telemetry.plane.so. An empty dict is falsy and does not suppress the
  exporters' env fallback, so the value has to be non-empty.
- manage.py: apply --settings before the first plane.* import, which otherwise
  materializes LazySettings from the default module and silently ignored the
  flag (breaking the local runserver entrypoint).
- Defer OTLP exporter creation to worker_process_init on the Celery prefork
  pool; the gRPC channel is created eagerly and is not fork-safe.
- Bound interpreter-exit flushing: shutdown_on_exit=False on both providers plus
  an atexit hook on the existing bounded flush_otel(). With an unreachable
  collector a manage.py command now exits in ~3.5s instead of ~63s.
- Stamp a per-process service.instance.id so gunicorn workers and prefork
  children stop exporting colliding cumulative http.server.* streams.
- Resolve instrumentors lazily and isolate each import, and take
  opentelemetry-instrumentation-httpx[instruments] so a missing httpx cannot
  crash every entrypoint with OTEL_ENABLED=0.
- Add a shared is_otel_active() (enabled AND any endpoint var, signal-specific
  included) used by setup, both settings modules and celery, so the log schema
  never changes in a process that exports nothing.
- Emit the boot banner through an explicit stderr handler and name the
  observability loggers in LOGGING so disable_existing_loggers stops silencing
  export errors.
- Strip whitespace in _protocol(); treat blank env values as unset so compose's
  ${VAR:-} interpolation no longer defeats the sampler defaults.
- Deployments: give migrator the OTel env, spell out the sampler defaults, and
  ship OTEL_ENABLED commented out on AIO, where plane.env is exported after
  container env and clobbered it.
- Docs: correct the emitted-metrics list, the log-schema gate, and the filelog
  pipeline (Docker envelope needs a second json_parser; trace context needs
  trace_parser, not move).
- Tests: fix the conftest env leak (monkeypatch.delenv, delete-only teardown)
  and cover the new gating, defaults, deferred providers and logging config.

Claude-Session: https://claude.ai/code/session_01BpGkdpVLNQ3Ziqcd6zK2qV

2026-08-30 11:50:21 +05:30

otel-collector.yaml

…

README.md

…

README.md

OpenTelemetry for Plane self-hosters

Plane's API server can emit OpenTelemetry traces, HTTP metrics, and trace-correlated logs over OTLP. Point it at any OTEL-compatible backend (Jaeger, Tempo, Datadog Agent, Honeycomb, Grafana Cloud, …) and start debugging slow endpoints.

Quickstart

Run an OTEL Collector pointing at your backend of choice. See otel-collector.yaml in this folder for a starting config.
Set two environment variables on the API container (and, for task telemetry, the worker and beat-worker containers):
```
OTEL_ENABLED=1
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
```
Restart the API/worker containers. That's it.

What you get

A server span per HTTP request, with http.route, http.method, http.status_code, http.target, and duration.
A span per Celery task with celery.action / celery.task_name / celery.state. Traceparent is propagated through the queue, so a request that enqueues a task is linked to that task's execution span in the same trace.
Child spans for every Postgres query, Redis op, and outbound requests / httpx call inside that request or task.
HTTP metrics: http.server.duration histogram and http.server.active_requests. (These are the only two DjangoInstrumentor emits — it does not produce request/response size histograms.)
JSON logs on stdout with trace_id / span_id / service_name fields added only when OTel is active — that is, OTEL_ENABLED is truthy and an OTLP endpoint is set. Any other combination leaves the existing log schema untouched, so you never get the extra fields without the traces to match them. Point the collector's filelog receiver at your container log directory to link logs ↔ traces.

Environment variables

Var	Default	Purpose
`OTEL_ENABLED`	`0`	Plane gate. Must be `1` (or `true`/`yes`/`on`).
`OTEL_SERVICE_NAME`	`plane-api`	Service identifier in your APM backend
`OTEL_EXPORTER_OTLP_ENDPOINT`	(required)	Your collector's OTLP receiver. The signal-specific `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` / `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` are honored too.
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	`grpc` or `http/protobuf`
`OTEL_EXPORTER_OTLP_HEADERS`	(unset)	For SaaS backends needing auth headers
`OTEL_ENVIRONMENT`	(unset)	Sets `deployment.environment.name` (falls back to `SENTRY_ENVIRONMENT`)
`OTEL_TRACES_SAMPLER`	`parentbased_traceidratio`	Standard OTEL sampler
`OTEL_TRACES_SAMPLER_ARG`	`0.1`	10 % head sampling. Set to `1.0` to capture every request.
`OTEL_RESOURCE_ATTRIBUTES`	(unset)	Extra resource attrs: `service.version=...`

If OTEL_ENABLED=1 but no endpoint var is set, the API logs a single WARNING at boot and continues without instrumentation — no silent local-host default, and the log schema is left unchanged.

What's not instrumented yet

The live (Node.js) collaboration server. It has no OTEL bootstrap yet and isn't covered by this Django-side setup.

Troubleshooting

No spans showing up. Confirm OTEL_ENABLED=1 is in the API container's env, not just the host shell. Check API logs for the OpenTelemetry configured INFO line at boot.
connection refused floods. These are dropped batches. The opentelemetry.* loggers are pinned to WARNING (and kept alive across Django's dictConfig, which would otherwise disable them) so the errors still reach your logs, but the underlying gRPC retries keep happening. Fix the collector reachability or unset OTEL_ENABLED.
trace_id is empty in logs. Either you're outside a request/task or the sampler dropped the trace. Drop OTEL_TRACES_SAMPLER_ARG to 1.0 while debugging.