It is 3 a.m. and the pager goes off. The dashboard says the database is fine. The logs are on another server, in another tool. The traces, if anyone set them up, are in a third. Twenty minutes pass before anyone knows which service is actually slow.
A unified observability stack fixes that: metrics, logs and traces in one place, linked, so the alert takes you straight to the slow request and the log lines that explain it. With open source, the whole stack can run on your own servers, even in networks that never touch the internet.
It has five parts:
- OpenTelemetry to collect metrics, logs and traces from your applications in one standard format
- Prometheus (or Mimir at larger scale) to store metrics
- Loki to store logs
- Tempo to store traces
- Grafana to query, link and alert on all three
Every part is open source, speaks open protocols and runs on plain Linux servers or Kubernetes, including air-gapped networks. You instrument your applications once with OpenTelemetry, and you can change any backend later without touching application code.
Below: the architecture, a collector configuration you can copy, how to size storage, and how to make alerts people trust.
Why teams build it on-prem
- Cost that grows with hardware, not with hosts or gigabytes. Commercial SaaS tools price per host, per gigabyte ingested or per user. On your own storage, more data means more disk.
- Data location. Logs often contain personal data, IP addresses and tokens. Keeping them in your own data centre makes GDPR, NIS2 and customer audits simpler.
- No proprietary agents. OpenTelemetry is a CNCF standard. Your instrumentation is not tied to one vendor.
- Networks that cannot reach the internet. Industrial sites, finance and government environments often cannot send telemetry to a cloud service at all.
The trade-off is that you run it yourself: upgrades, capacity, backups and on-call for the monitoring system.
The architecture
| Layer | Component | What it does |
|---|---|---|
| Instrumentation | OpenTelemetry SDKs, Prometheus exporters | Apps emit metrics, logs and traces; exporters cover servers, databases and network devices |
| Collection | OpenTelemetry Collector (or Grafana Alloy) | Receives OTLP, batches, filters and enriches data, and routes each signal to its backend |
| Metrics | Prometheus; Mimir or Thanos for long-term and multi-site | Stores time series and evaluates alert rules |
| Logs | Loki | Stores logs indexed by labels, with object storage for retention |
| Traces | Tempo (or Jaeger) | Stores traces, also on object storage |
| Visualisation and alerts | Grafana, Alertmanager | Dashboards, links between signals, alert routing |
| Storage | Local disks, plus S3-compatible object storage such as MinIO or Ceph | Long-term retention for Loki, Tempo and Mimir |
What makes it unified is linking the signals, not keeping them in one database. A trace carries a trace ID. Logs written during that request carry the same ID. Grafana uses this to jump from a slow request in a dashboard to its trace, and from the trace to the exact log lines.
A working OpenTelemetry Collector configuration
The Collector receives all three signals over OTLP and sends each one to the right backend. This example uses the Collector Contrib distribution and the native OTLP endpoints in Prometheus 3 and Loki 3:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
batch: {}
exporters:
otlphttp/prometheus:
endpoint: http://prometheus:9090/api/v1/otlp
otlphttp/loki:
endpoint: http://loki:3100/otlp
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
service:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/loki]
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp/tempo]
Notes for production:
- Start Prometheus with
--web.enable-otlp-receiverso it accepts OTLP metrics. - Turn on TLS between the collectors and the backends.
insecure: trueis only for a lab. - Run collectors in two tiers once you have more than a few hosts: an agent on each node, and a small gateway pool that does batching, filtering and routing.
- Use the
memory_limiterprocessor everywhere, so a burst of telemetry cannot crash the collector.
Sizing storage
Prometheus. The Prometheus documentation gives a simple formula:
disk needed = retention time (seconds) × samples ingested per second × bytes per sample
Prometheus compresses samples to about 1 to 2 bytes each. The biggest factor is how many time series you create, not how many hosts you have. Labels with unbounded values, such as user IDs, request IDs or full URLs, multiply series and are the most common reason a Prometheus server runs out of memory. Keep local retention short (two to four weeks is common) and use Mimir or Thanos with object storage for anything longer.
Loki. Loki indexes only labels and keeps log lines compressed in object storage. Keep labels few and low-cardinality (service, environment, cluster, level), and search the content with LogQL instead of adding labels for it.
Tempo. Tempo stores traces in object storage without a heavy index. Sampling decides the volume. Keep all errors and slow requests, and a percentage of normal ones, with tail sampling in the collector.
Alerts people trust
A stack with hundreds of noisy alerts is worse than no stack, because people learn to ignore it. Four rules keep alerts useful:
- Alert on symptoms users feel: error rate, latency and availability against a service level objective (SLO). Keep cause-level alerts, such as CPU or disk, for dashboards and tickets.
- Every alert has an owner and a runbook link. If nobody would act on it at 3 a.m., it should not page.
- Route by severity. Pages go to on-call, warnings go to a team channel, and everything else goes to a daily summary.
- Review alerts monthly. Delete or tune any alert that fired without anyone needing to act.
Where Zabbix fits
Prometheus is built for cloud-native systems that expose metrics over HTTP. Zabbix is stronger for network devices (SNMP), VMware, Windows servers and classic IT. Many teams run both and use Grafana to show them side by side, so nobody has to replace a monitoring system that already works.
Moving from Datadog, New Relic or Splunk
- Instrument with OpenTelemetry first. Most vendors accept OTLP, so you can send the same data to the old tool and the new stack in parallel.
- Rebuild the dashboards and alerts that people use. Usage data in the old tool shows which ones those are. Most organisations use a small fraction of what they have.
- Run both for a few weeks, compare, then switch off the old tool service by service.
If the main problem is a Datadog bill, start with our guide on how to reduce Datadog costs: many bills drop a lot before any migration.
Frequently asked questions
What is a unified observability platform? One place where metrics, logs and traces are collected, linked by shared identifiers such as trace IDs, and searched together. With open source this is usually Prometheus or Mimir, Loki, Tempo and Grafana, fed by OpenTelemetry.
Can the stack ingest OpenTelemetry without proprietary agents? Yes. The OpenTelemetry Collector receives OTLP from your applications and sends it to Prometheus, Loki and Tempo, which all accept open formats.
Does it run fully air-gapped? Yes. All components are self-hosted, and container images and packages can be mirrored to an internal registry. See air-gapped monitoring for how we build and update stacks that never touch the internet.
Prometheus or Mimir? Start with Prometheus. Move to Mimir (or add Thanos) when you need long retention, high availability across sites, or more series than one server handles comfortably.
How long does a first useful stack take? A first stack with key dashboards and alerts for the most important services typically takes a few weeks. Coverage then grows service by service.
We design, build and run open-source observability stacks with Prometheus, Grafana, Loki, Tempo and OpenTelemetry, in your cloud or on-prem. See monitoring and observability for how we work, or Kubernetes if the stack runs on your clusters.
