logo

Building an On-Prem Observability Stack with Prometheus, Grafana, Loki and OpenTelemetry

How to build a unified, open-source observability stack on your own servers: OpenTelemetry, Prometheus, Loki, Tempo and Grafana, with sizing and alerting.

YR
Yasir Raza · Senior Solutions Architect
6 min read · November 17, 2026
Key takeaways
  • OpenTelemetry collects, Prometheus stores metrics, Loki stores logs, Tempo stores traces and Grafana ties them together.
  • Everything is open source, runs on your own servers or Kubernetes, and works fully air-gapped.
  • A ready-to-use Collector config sends all three signals to the right backend.
  • Series cardinality, not host count, decides how much Prometheus storage you need.

It is 3 a.m. and the pager goes off. The dashboard says the database is fine. The logs are on another server, in another tool. The traces, if anyone set them up, are in a third. Twenty minutes pass before anyone knows which service is actually slow.

A unified observability stack fixes that: metrics, logs and traces in one place, linked, so the alert takes you straight to the slow request and the log lines that explain it. With open source, the whole stack can run on your own servers, even in networks that never touch the internet.

It has five parts:

  • OpenTelemetry to collect metrics, logs and traces from your applications in one standard format
  • Prometheus (or Mimir at larger scale) to store metrics
  • Loki to store logs
  • Tempo to store traces
  • Grafana to query, link and alert on all three

Every part is open source, speaks open protocols and runs on plain Linux servers or Kubernetes, including air-gapped networks. You instrument your applications once with OpenTelemetry, and you can change any backend later without touching application code.

Below: the architecture, a collector configuration you can copy, how to size storage, and how to make alerts people trust.

Why teams build it on-prem

  • Cost that grows with hardware, not with hosts or gigabytes. Commercial SaaS tools price per host, per gigabyte ingested or per user. On your own storage, more data means more disk.
  • Data location. Logs often contain personal data, IP addresses and tokens. Keeping them in your own data centre makes GDPR, NIS2 and customer audits simpler.
  • No proprietary agents. OpenTelemetry is a CNCF standard. Your instrumentation is not tied to one vendor.
  • Networks that cannot reach the internet. Industrial sites, finance and government environments often cannot send telemetry to a cloud service at all.

The trade-off is that you run it yourself: upgrades, capacity, backups and on-call for the monitoring system.

The architecture

Layer Component What it does
Instrumentation OpenTelemetry SDKs, Prometheus exporters Apps emit metrics, logs and traces; exporters cover servers, databases and network devices
Collection OpenTelemetry Collector (or Grafana Alloy) Receives OTLP, batches, filters and enriches data, and routes each signal to its backend
Metrics Prometheus; Mimir or Thanos for long-term and multi-site Stores time series and evaluates alert rules
Logs Loki Stores logs indexed by labels, with object storage for retention
Traces Tempo (or Jaeger) Stores traces, also on object storage
Visualisation and alerts Grafana, Alertmanager Dashboards, links between signals, alert routing
Storage Local disks, plus S3-compatible object storage such as MinIO or Ceph Long-term retention for Loki, Tempo and Mimir

What makes it unified is linking the signals, not keeping them in one database. A trace carries a trace ID. Logs written during that request carry the same ID. Grafana uses this to jump from a slow request in a dashboard to its trace, and from the trace to the exact log lines.

A working OpenTelemetry Collector configuration

The Collector receives all three signals over OTLP and sends each one to the right backend. This example uses the Collector Contrib distribution and the native OTLP endpoints in Prometheus 3 and Loki 3:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch: {}

exporters:
  otlphttp/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/loki]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]

Notes for production:

  • Start Prometheus with --web.enable-otlp-receiver so it accepts OTLP metrics.
  • Turn on TLS between the collectors and the backends. insecure: true is only for a lab.
  • Run collectors in two tiers once you have more than a few hosts: an agent on each node, and a small gateway pool that does batching, filtering and routing.
  • Use the memory_limiter processor everywhere, so a burst of telemetry cannot crash the collector.

Sizing storage

Prometheus. The Prometheus documentation gives a simple formula:

disk needed = retention time (seconds) × samples ingested per second × bytes per sample

Prometheus compresses samples to about 1 to 2 bytes each. The biggest factor is how many time series you create, not how many hosts you have. Labels with unbounded values, such as user IDs, request IDs or full URLs, multiply series and are the most common reason a Prometheus server runs out of memory. Keep local retention short (two to four weeks is common) and use Mimir or Thanos with object storage for anything longer.

Loki. Loki indexes only labels and keeps log lines compressed in object storage. Keep labels few and low-cardinality (service, environment, cluster, level), and search the content with LogQL instead of adding labels for it.

Tempo. Tempo stores traces in object storage without a heavy index. Sampling decides the volume. Keep all errors and slow requests, and a percentage of normal ones, with tail sampling in the collector.

Alerts people trust

A stack with hundreds of noisy alerts is worse than no stack, because people learn to ignore it. Four rules keep alerts useful:

  1. Alert on symptoms users feel: error rate, latency and availability against a service level objective (SLO). Keep cause-level alerts, such as CPU or disk, for dashboards and tickets.
  2. Every alert has an owner and a runbook link. If nobody would act on it at 3 a.m., it should not page.
  3. Route by severity. Pages go to on-call, warnings go to a team channel, and everything else goes to a daily summary.
  4. Review alerts monthly. Delete or tune any alert that fired without anyone needing to act.

Where Zabbix fits

Prometheus is built for cloud-native systems that expose metrics over HTTP. Zabbix is stronger for network devices (SNMP), VMware, Windows servers and classic IT. Many teams run both and use Grafana to show them side by side, so nobody has to replace a monitoring system that already works.

Moving from Datadog, New Relic or Splunk

  1. Instrument with OpenTelemetry first. Most vendors accept OTLP, so you can send the same data to the old tool and the new stack in parallel.
  2. Rebuild the dashboards and alerts that people use. Usage data in the old tool shows which ones those are. Most organisations use a small fraction of what they have.
  3. Run both for a few weeks, compare, then switch off the old tool service by service.

If the main problem is a Datadog bill, start with our guide on how to reduce Datadog costs: many bills drop a lot before any migration.

Frequently asked questions

What is a unified observability platform? One place where metrics, logs and traces are collected, linked by shared identifiers such as trace IDs, and searched together. With open source this is usually Prometheus or Mimir, Loki, Tempo and Grafana, fed by OpenTelemetry.

Can the stack ingest OpenTelemetry without proprietary agents? Yes. The OpenTelemetry Collector receives OTLP from your applications and sends it to Prometheus, Loki and Tempo, which all accept open formats.

Does it run fully air-gapped? Yes. All components are self-hosted, and container images and packages can be mirrored to an internal registry. See air-gapped monitoring for how we build and update stacks that never touch the internet.

Prometheus or Mimir? Start with Prometheus. Move to Mimir (or add Thanos) when you need long retention, high availability across sites, or more series than one server handles comfortably.

How long does a first useful stack take? A first stack with key dashboards and alerts for the most important services typically takes a few weeks. Coverage then grows service by service.

We design, build and run open-source observability stacks with Prometheus, Grafana, Loki, Tempo and OpenTelemetry, in your cloud or on-prem. See monitoring and observability for how we work, or Kubernetes if the stack runs on your clusters.

YR
Written by
Yasir Raza
Co-founder · Senior Solutions Architect

Yasir designs the platforms our clients run on: Kubernetes, cloud and on-premises infrastructure, security baselines and the automation that keeps them consistent. He leads architecture and security reviews on our projects.

Want a second opinion on your setup?
Talk to the engineers who wrote this. 30 minutes, no sales pitch.
Book a call

Keep reading

Observability
Datadog Bill Too High? How to Cut It, and When to Move to Open Source
Where Datadog costs come from, the fixes that cut the bill inside Datadog, and when a hybrid or full move to Prometheus, Grafana and OpenTelemetry pays off.
Read article →
Nextcloud
The November 2026 Outlook Outage: How to Keep Working When Microsoft 365 Goes Down
What happened in the 12–14 November Outlook and Microsoft 365 outage, and how a Nextcloud setup you control keeps teams working and data private.
Read article →
Odoo
Odoo Implementation Cost in 2026: What Drives the Price and How to Estimate It
What an Odoo implementation costs and why: licences, effort by scope, data migration, custom code, hosting and support, plus how to check any quote.
Read article →

Next step

Talk to an engineer about Observability

Send a few details, and pick up to 3 times if you would like a call. We reply within one business day.

You talk to the expert who would do the work

Written scope and price after the call

No obligation

info@opensource.consulting
+1 514 577 8599