Grafana LGTM Stack — Operational Monitoring

Platform 12 of 12·OS Layer 10 (Observability Layer)·Required in all deployment modes

Scope of this page. Reference material for the Grafana LGTM Stack: what it is, what it is normally used for, and how and why Synedrix OS uses it. It is descriptive, not normative — where this page and a canonical spec in docs/os/ disagree, the canonical spec wins. Platform responsibilities are set by Tech Stack; the metric definitions, SLOs and alert routing this platform renders are specified in Observability Metrics Spec §14–15.

What the Grafana LGTM Stack is link

The Grafana LGTM Stack is not one product. It is five separately released open-source projects from Grafana Labs that are designed to compose into a single observability backend. LGTM is the initialism of the four backend projects — Loki, Grafana, Tempo, Mimir — with Alloy as the collector that feeds them.

Grafana’s own documentation for the reference distribution names the set directly: the docker-otel-lgtm project “provides an open source backend for OpenTelemetry in a Docker image” and includes “an OpenTelemetry Collector, Grafana, Loki, Mimir, and Tempo.”

Licences were verified individually against each project’s LICENSE file on its official repository; they are not uniform across the stack. See Deployment and licensing.

The pillars framing link

Grafana’s documentation uses a four-pillar framing, not the more commonly cited three: “Metrics, logs, traces, and profiles form the pillars of observability.” Each is given a distinct job:

Synedrix OS adopts three of the four. Pyroscope is not a canonical platform and continuous profiling is not an OS concern at this version.

Query languages link

Each backend keeps its own query language, and all three are exposed through the same Grafana surface:

The shared vocabulary matters operationally: a label selector learned for metrics transfers to logs, which is what makes pivoting from an alert to the underlying log line a single step rather than a context switch.

OpenTelemetry is the ingestion standard link

Every backend in the stack accepts OpenTelemetry natively, which is why the stack can be treated as a destination rather than as an instrumentation framework:

• Alloy is “an OpenTelemetry Collector distribution” and gathers “metrics, logs, traces, and profiles in one unified solution,” sending them “to Grafana Cloud, a self-managed Grafana stack, or other observability platforms.”
• Mimir ingests “Prometheus or OpenTelemetry metrics.”
• Loki “natively supports ingesting OpenTelemetry logs over HTTP” at the /otlp endpoint.
• Tempo accepts OTLP over gRPC and HTTP, alongside Jaeger, Zipkin and Kafka receivers, and “only requires an object storage to operate.”

Storage model link

Loki, Tempo and Mimir all target object storage rather than local disk. Loki “does not index the contents of the logs, but only indexes metadata about your logs as a set of labels,” then compresses log data “in chunks in an object store such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS).” Tempo’s storage backend “should be one of ‘gcs’, ‘s3’, ‘azure’ or ’local’.” This is the property that makes the stack viable at OS scale without a dedicated storage tier per signal.

What the Grafana LGTM Stack is normally used for link

The cases Grafana’s own documentation leads with:

• Infrastructure and service dashboards — turning “time-series database (TSDB) data into insightful graphs and visualizations,” with each backend attached as a data source.
• Alerting on production systems — Grafana Alerting “allows you to learn about problems in your systems moments after they occur,” letting an operator “create, manage, view, and adjust alert rules to alert on your metrics data or log entries from multiple data sources.”
• Notification routing — choosing “how, when, and where to send your alert notifications,” with contact points as the delivery abstraction.
• Cross-signal correlation — Grafana queries “metrics, logs, and traces no matter where they are stored,” so a metric spike, the log lines behind it and the trace that produced it are reachable from one place.
• Long-term Prometheus-compatible metric retention — Mimir as horizontally scalable, multi-tenant storage behind a Prometheus-compatible read API.
• Telemetry collection and routing — Alloy as a single agent scraping Prometheus endpoints and receiving OTLP, forwarding to whichever backends the deployment declares.

The common thread is that the LGTM Stack owns whether the system is running correctly — resource pressure, error rates, latency, saturation, availability — not what the system decided or why.

How Synedrix OS uses it link

The Grafana LGTM Stack is the operational monitoring surface of Synedrix OS and sits at Layer 10 (Observability Layer) of the twelve-layer taxonomy, alongside Langfuse and PostHog. It is required in all three deployment modes.

It is the platform that most directly serves the OS invariant that every workflow must be observable end-to-end. Layer 10 exists to “make execution measurable, explainable, reviewable,” and it explicitly must not own business decisions or domain state.

flowchart LR
  n1([Grafana LGTM Stack]):::term
  subgraph What it owns
    n2[Infrastructure health]:::agent
    n3[Service dashboards]:::agent
    n4[Alerting]:::agent
    n5[SLOs]:::agent
    n6[Threshold breaches and alert routing]:::agent
    n7[Correlation carriage in alerts]:::agent
  end
  n1 --> n2
  subgraph What it does not own
    n8[Is the model layer behaving]:::warn
    n9[Langfuse]:::term
    n8 --> n9
    n10[Is the product working for users]:::warn
    n11[PostHog]:::term
    n10 --> n11
    n12[What is the raw high-volume time-series truth]:::warn
    n13[QuestDB]:::term
    n12 --> n13
    n14[What is the audit-grade record]:::warn
    n15[Supabase PG17]:::term
    n14 --> n15
  end
  n1 --> n8
  subgraph Why the Grafana LGTM Stack
    n16[OpenTelemetry-native throughout]:::term
    n17[Fully self-hostable]:::term
    n18[One query surface over three signals]:::term
    n19[Mimir removes the retention ceiling]:::term
  end
  n1 --> n16

What it owns link

Per Tech Stack, this platform owns:

• Infrastructure health
• Service dashboards
• Alerting
• SLOs

Concretely, in Synedrix OS terms:

• Infrastructure and service health for every platform in the stack — n8n main and worker processes, the Redis queue depth, Supabase, QuestDB, Neo4j, Qdrant, the LangGraph runtime, both Hookdeck gateways.
• Dashboard rendering for the canonical metric families in Observability Metrics Spec §7. That spec requires dashboards to be “organized per metric family (§7.1–7.10) rather than per agent, so cross-agent regressions in a single metric family remain visible even when no single agent crosses its own threshold.”
• Threshold breaches and alert routing — §14 of that spec assigns “threshold breaches, alert routing, dashboard panels” to this platform, and §15 fixes the default SLO thresholds and routing targets for retrieval_f1_at_k, hallucination_probability, cost_drift_pct, tool_success_rate, guardrail_false_positive_rate and trust_trend_30d.
• Correlation carriage in alerts — every alert “MUST carry the same correlation identifiers as the triggering event” so an on-call responder can pivot from the alert into the underlying Langfuse trace or the underlying Supabase record without re-deriving context.

What it does not own link

Layer 10 has three platforms and Layer 7 has two that are frequently confused with it. The boundaries are ownership boundaries, not preferences:

Traces are not an audit trail. This distinction is load-bearing. A Tempo trace is sampled, retention-bounded and shaped for latency debugging; the lineage record in Neo4j and the append-only invocation start, tool-call, and outcome records in Supabase are neither. Principle 7 requires the system to answer what happened, why it happened and under whose authority — those answers come from the event substrate, Neo4j and Supabase. Losing a trace is an operational inconvenience; losing a lineage record is a governance failure. Do not let a Grafana panel stand in for either.

How it is operated link

Alloy is the single ingestion path. Alloy runs as the collector, scraping Prometheus-format endpoints from the platform containers and receiving OTLP from application code, then forwarding metrics to Mimir, logs to Loki and traces to Tempo. Grafana attaches all three as data sources. Nothing writes to a backend directly.

OpenTelemetry is the common wire format across layers. This is what keeps the observability story coherent across the Layer 3 / Layer 4 split:

• n8n supports OpenTelemetry tracing via environment variables on self-hosted instances, which is the supported path for feeding Layer 3 execution telemetry into Layer 10 rather than reading it out of n8n’s own UI.
• LangGraph and LangChain emit traces to Langfuse over OpenTelemetry or the native exporter, per Tech Stack §13. Those spans stay in Langfuse; what reaches Mimir is the aggregate.
• OS events defined in the Event Taxonomy Standard carry trace_id and span_id in the envelope, and the Observability Metrics Spec §6 requires the same. A shared trace identifier is what allows a Grafana alert, a Langfuse span and a Supabase invocation row to be recognised as the same event.
• The Outcome Spec makes the same split explicit: the os.outcome.* event payload carries outcome_id, source, correlation_id, status and duration_ms, and the observability layer reads those events — the full outcome record stays in Supabase.

Events are the source, dashboards are the consumer. Observability Metrics Spec §14 is unambiguous: events “MUST be emitted regardless of which platform ultimately persists the derived value; the event bus is the single normalized ingestion path, and platform-specific write paths are downstream consumers, not sources.” A metric that exists only because a Grafana panel computes it is out of contract.

Why the Grafana LGTM Stack link

• It answers a different question from the other Layer 10 platforms. Three observability platforms is only defensible if each owns a distinct question. System health, model-layer behaviour and product behaviour are genuinely different failure modes with different on-call responses, and collapsing them produces dashboards nobody trusts.
• OpenTelemetry-native throughout. Every backend ingests OTLP, so instrumentation is not coupled to the vendor. If a backend is replaced, the emitting code does not change — which is the same reasoning that put OpenTelemetry on the n8n and LangGraph paths.
• Object storage instead of a per-signal storage tier. Loki, Tempo and Mimir all target S3-compatible object storage, so all three backends can run in Mode 2 and Mode 3 against one local object store rather than three specialised databases.
• Fully self-hostable. All five components run as containers or on Kubernetes with no licence key and no callback to a vendor, which is what Principle 11 (Open by default, sovereign by design) and Mode 3 require.
• One query surface over three signals. PromQL, LogQL and TraceQL share label semantics and are exposed through one Grafana instance, so the metric-to-log-to-trace pivot is a UI action rather than a tool change. That is the difference between an alert an operator can act on and an alert that starts an investigation.
• Mimir removes the retention ceiling. A plain Prometheus server is single-node and local-disk-bound. Mimir provides “horizontally scalable, highly available, multi-tenant, long-term storage” behind the same Prometheus query API, so retention becomes an object-storage cost decision rather than an architectural one.

Deployment and licensing link

Self-hosting. All five components are distributed as official container images and run under Docker, Docker Compose or Kubernetes. The reference implementation in synedrix-os-stack-self-hosted runs Alloy, Loki, Tempo, Mimir and Grafana as containers with MinIO as the object store; Alloy scrapes the platform containers and remote-writes to Mimir. No Prometheus server container is deployed in that stack — Mimir is the metrics backend and Alloy is the scraper.

Licences — verify per component, not per stack. Grafana Labs relicensed its core projects from Apache 2.0 to AGPLv3 in 2021, and the stack is not uniformly licensed today:

What AGPLv3 means here, stated factually. AGPLv3 extends the GPL’s source-availability obligation to network use: where GPLv3 is triggered by distributing a modified binary, AGPLv3 §13 is additionally triggered by making a modified version available to users over a network, requiring that those users be offered the corresponding source of the modified version. Four of the five components in this platform are AGPLv3.

Three consequences follow for Synedrix OS, and none of them are resolved by this page:

1. Running unmodified upstream images is the low-friction path. AGPLv3’s network clause attaches to modified versions. Deploying official Grafana, Loki, Tempo and Mimir images unchanged, configured through their own configuration files, does not create a source-offer obligation for Synedrix OS code.
2. Forking or patching a component changes that. Any patched build of Grafana, Loki, Tempo or Mimir that is reachable over a network by users triggers the §13 obligation for that component’s corresponding source.
3. “Open source” is not one commitment. Core Principles Principle 11 (Open by default, sovereign by design) is satisfied by every component here — all five are genuinely open source and fully self-hostable, which is a stronger position than the fair-code platforms elsewhere in the stack. But AGPLv3 and Apache 2.0 impose materially different obligations on a commercial product, and a stack-wide licence summary such as “AGPL/Apache” hides which component carries which.

Open item. The canon does not agree on what this platform is. Tech Stack §11 names it “Grafana + Prometheus” and records self-hosting as “Grafana (AGPL 3.0), Prometheus (Apache 2.0)”; Observability Metrics Spec §14 repeats “Grafana + Prometheus” and states that “Prometheus scrapes or receives the aggregate metrics”; Deployment Model Spec lists Grafana and Prometheus as separate inventory rows. Against that, canonical/architecture.yaml — the machine-readable facts layer, and the generator source for the reference architecture — and Layer Taxonomy Table both name “Grafana LGTM Stack” with the note “Operational monitoring (Grafana, Alloy, Mimir, Loki, Tempo),” and the reference implementation deploys exactly those five components with no Prometheus server. These are not two names for one thing. Prometheus and Mimir are related but distinct: Mimir is Grafana’s horizontally scalable, multi-tenant long-term storage for Prometheus-format metrics, and it replaces the storage and query role a Prometheus server would play, while Alloy replaces the scrape role. A description naming Prometheus also omits Loki and Tempo entirely, which means the prose specs currently describe a metrics-only platform where the facts layer and the deployed stack describe a three-signal one. This page uses “Grafana LGTM Stack” because the facts layer and the deployed reality agree. Reconciling the three prose documents — including the licence line, since Loki, Tempo and Mimir are AGPLv3 and Alloy is Apache 2.0 — requires an explicit decision and edits to those documents. It is not resolved here.

Official documentation link

• Grafana documentation home — what Grafana is; querying, visualisation and alerting
• Grafana Alerting — alert rules, contact points, notification routing
• Grafana Loki documentation — components and LogQL
• Loki overview — label-only indexing and object-storage chunks
• Send OpenTelemetry logs to Loki — the native /otlp endpoint
• Grafana Tempo documentation — tracing backend and TraceQL
• Tempo — traces and telemetry — Grafana’s pillars-of-observability framing
• Tempo configuration — OTLP/Jaeger/Zipkin receivers and storage backends
• Grafana Mimir documentation — long-term Prometheus and OpenTelemetry metric storage
• Mimir querier — PromQL evaluation on the read path
• Grafana Alloy documentation and Alloy introduction — the OpenTelemetry Collector distribution
• Docker OpenTelemetry LGTM — Grafana’s own reference composition of the stack
• PromQL basics — the query language Mimir implements
• Licence files: Grafana · Loki · Tempo · Mimir · Alloy

Related Synedrix OS documents link

• Synedrix OS Tech Stack — normative platform responsibilities; §11 covers operational monitoring (see the Open item above).
• Observability Metrics Spec — canonical metric families, the storage tiering table (§14) and the default SLO and alert-routing table (§15) this platform renders.
• Layer Taxonomy Table — Layer 10 definition, what it must not own, and the platform mapping that names the LGTM Stack.
• Core Principles — Principle 7 (Explainability and audit by design) and Principle 11 (Open by default, sovereign by design).
• Deployment Model Spec — the three deployment modes and the open-source component inventory.
• Event Taxonomy Standard — the event envelope and correlation identifiers that alerts and dashboards must carry.
• Outcome Spec — what the observability layer reads from os.outcome.* events, and what stays in Supabase.

Synedrix OS · Platform reference · Grafana LGTM Stack