Layer 10 of 12·Slice: Governance and observability·Platform: Langfuse, PostHog, Grafana LGTM Stack

Scope of this page. Reference material for the Observability layer: what it owns, what it must not own, and how Synedrix OS implements it today. It is descriptive, not normative — where this page and the Layer Taxonomy Table disagree, the taxonomy table wins.

What this layer is

Observability makes execution measurable, explainable, and reviewable. It is the layer that answers what happened, how well did it perform, and can it be reconstructed later — for every workflow, every agent invocation, every tool call, and every memory operation across the platform — without itself deciding whether an action was permitted or owning the business state that action touched.

This layer is realized by three non-overlapping platforms rather than one, because “is this healthy” is genuinely three different questions with three different failure modes and three different on-call responses. Langfuse answers the model-level question — what did the model do, and how good was the output. PostHog answers the behavioral question — how is the product or workflow actually being used, and did a change help. The Grafana LGTM Stack answers the infrastructure question — is the system itself running correctly. All three consume the same canonical metric definitions and event envelope from the Observability Metrics Spec, so a correlation identifier minted once is traceable across all three rather than requiring the value to be recomputed, and possibly disagree, in more than one place.

A distinction that must not be blurred: a trace is not an audit trail. Observability data — traces, logs, dashboards — is sampled and retained on an operational schedule for debugging and quality measurement. An audit trail must be complete and durable, and one with gaps in it is not an audit trail. That record is produced by the Audit & Lineage Agent (os-agt-008), which sits in this layer’s Observability family and is strictly read-only, reconstructing lineage over Neo4j and Supabase rather than over sampled trace data.

flowchart LR
  n1([Observability]):::term
  subgraph What it owns
    n2[Logs]:::agent
    n3[Traces]:::agent
    n4[Metrics]:::agent
    n5[Run health]:::agent
    n6[Decision lineage]:::agent
    n7[Alerts]:::agent
  end
  n1 --> n2
  subgraph What it must not own
    n8[Whether a specific action is permitted]:::warn
    n9[Governance Controls Layer]:::term
    n8 --> n9
    n10[Domain state ownership]:::warn
    n11[Memory Data Layer]:::term
    n10 --> n11
    n12[Workflow sequencing retries and approval routing]:::warn
    n13[Orchestration Layer]:::term
    n12 --> n13
    n14[Domain KPI vocabulary and targets]:::warn
    n15[Domain Pack Layer]:::term
    n14 --> n15
  end
  n1 --> n8
  subgraph Interfaces with
    n16[Human Interface Goal/Intent]:::check
    n17[Orchestration Agent Runtime]:::check
    n18[Agent Registry Tool Abstraction]:::check
    n19[Memory Data]:::check
    n20[Governance Controls]:::check
    n21[Domain Pack Business Instance]:::check
  end
  n1 --> n16

What it owns

Per the Layer Taxonomy Table, Observability owns the following:

  • Logs — collected through Alloy and stored in Loki, queried with LogQL.
  • Traces — LLM and agent trace spans captured in Langfuse; infrastructure traces stored in Tempo and queried with TraceQL.
  • Metrics — the eleven canonical metric families the Observability Metrics Spec defines: retrieval quality, groundedness and hallucination, cost and drift, semantic output validity, response sampling and human review, multi-turn coherence, tool reliability, risk-aware decision alignment, policy and guardrail effectiveness, user trust and satisfaction trend, and the memory control plane.
  • Run health — infrastructure and service dashboards for every platform in the stack, rendered in Grafana against Mimir-stored metrics.
  • Decision lineage — the read-only reconstruction of workflow execution traces, agent invocation records, policy decisions, and tool call logs into a sourced, explainable run history.
  • Alerts — threshold breaches and their routing, carrying the same correlation identifiers as the event that triggered them so an on-call responder can pivot directly into the underlying trace or record.

What it must not own

This layer measures and explains; it must never become the layer that decides an outcome or holds the canonical record of business state.

ConcernCorrect owner
Whether a specific action is permittedGovernance & Controls Layer (Policy Guard Agent)
Domain state ownership — the canonical business recordMemory & Data Layer (Supabase PG17)
Workflow sequencing, retries, and approval routingOrchestration Layer
Raw high-throughput time-series truth and rolling temporal aggregatesMemory & Data Layer (QuestDB)
Domain KPI vocabulary and targetsDomain Pack Layer defines the vocabulary; Business Instance Layer sets the targets and alert thresholds

Interfaces with

Observability’s taxonomy row lists its interface as all other layers, and that breadth is deliberate rather than a shorthand for “everything eventually gets logged.” The invariant that every workflow must be observable end-to-end means the same correlation identifiers — trace_id, span_id, correlation_id — are carried in the canonical event envelope emitted by every layer, not sampled from a subset of them:

  • Human Interface & Goal/Intent — human decisions and goal decompositions emit lifecycle events that PostHog reads to answer adoption and funnel questions: which goals are actually pursued, where a human-reviewed step stalls.
  • Orchestration & Agent Runtime — every n8n workflow run and every LangGraph invocation emits traces over OpenTelemetry, the shared wire format that keeps one observability plane coherent across the Layer 3 / Layer 4 split established by ADR-OS-001.
  • Agent Registry & Tool Abstraction — tool reliability metrics (tool_success_rate, p95_latency_ms) and risk-alignment scores measured here feed back as promotion gates and authorization inputs at those layers.
  • Memory & Data — the memory control plane (§7.11 of the Observability Metrics Spec) is measured with the same metric families as any other invocation: recall mediation, consolidation lag, fabric growth and staleness, and promotion/erasure rates, so a memory operation is as observable as an agent invocation.
  • Governance & Controls — policy decisions and tool-authorization events (agent.tool.unauthorized, agent.tool.suspended) are emitted here so that Governance & Controls can consume canonical metrics like cost_drift_flag and risk_alignment_score as policy inputs, closing a feedback loop between the two layers rather than making it one-directional.
  • Domain Pack & Business Instance — domain packs emit domain-specific metrics and logs into this same canonical substrate rather than a parallel one, and business instances set the alerting rules and thresholds on top of it, per the Responsibility Matrix’s “Observability standards” row.

How Synedrix OS implements it

The Observability Metrics Spec is this layer’s dedicated canonical spec, and it standardizes four things. First, a canonical event envelope — event_type, event_version, a trace block (trace_id/span_id/parent_span_id) and a subject block (agent_id/workflow_id/invocation_id/conversation_id/user_id) — that every observability event must use, so correlation is structural rather than convention. Second, the eleven metric families listed above, governed by six normative principles: events must be emitted before or alongside derived aggregates, every event carries stable correlation identifiers, invocation and lifecycle records are append-only, metric field names are reused exactly across tables/events/dashboards/policy rules, invocation-scoped and conversation-scoped metrics are never collapsed into each other, and structural validation is kept distinct from semantic validation. Third, a persistence model spanning seven tables (agent_invocations, agent_invocation_outcomes, tool_invocations, evaluation_results, response_reviews, conversation_metrics, guardrail_metrics) that store audit-grade metric values. Fourth, a platform-and-storage tiering rule (§14): those seven tables live in Supabase PG17 as canonical relational truth; rolling trend series derived from them (cost_drift_pct over time, trust_trend_30d, tool-success-rate windows) live in QuestDB rather than being recomputed ad hoc; raw LLM trace spans live in Langfuse and must carry the same trace_id as the correlating metric event; and threshold breaches, alert routing, and dashboard panels render through Grafana against Mimir-stored aggregates. The spec’s default SLO table (§15) fixes concrete thresholds — for example, an alert when the 7-day rolling retrieval_f1_at_k average drops below 0.5, a page when the hallucination_flag rate exceeds 5% of finalized responses in a rolling hour, and a promotion-blocking gate whenever a risk_alignment_score failure is reported for a medium/high/critical risk-tier agent — and each alert must carry the same correlation identifiers as the triggering event.

Each of the three platforms then realizes one slice of that spec. Langfuse captures prompt versioning, LLM/agent trace spans, token and cost attribution, and evaluation scoring, ingesting over OpenTelemetry so that both n8n AI Agent nodes and LangGraph invocations land in one correlated trace view regardless of which engine produced the call — the retrieval, groundedness, cost, and semantic-validation events defined in §9 of the metrics spec correlate 1:1 with a Langfuse trace span for exactly this reason. PostHog turns workflow and agent lifecycle events into behavioral analysis: which workflows are actually run, where a governed workflow’s triggered → validated → approved → executed funnel drops off, and it also carries the feature-flag mechanism that lets a Business Instance progressively enable a capability without a redeploy. The Grafana LGTM Stack (Grafana, Alloy, Loki, Tempo, Mimir) is the operational-monitoring surface: Alloy is the single collector that scrapes and receives OpenTelemetry data and forwards it to Loki (logs), Tempo (infrastructure traces) and Mimir (long-term Prometheus-compatible metrics), with Grafana as the shared query and alerting surface across all three via PromQL, LogQL, and TraceQL.

Platform mapping

Three platforms are mapped to this layer, each owning a distinct question about system behavior rather than overlapping coverage.

PlatformOwns
LangfuseLLM/agent trace capture, prompt versioning, token and cost attribution, model-call latency and error visibility, evaluations and quality scoring.
PostHogWorkflow and agent behavioral analytics, feature flags, A/B experiments, cross-domain business decision analytics.
Grafana LGTM StackInfrastructure health, service dashboards, alerting, and SLOs across Grafana, Alloy, Mimir, Loki, and Tempo.

OS agents at this layer

Four OS-owned agents are registered at this layer, spanning two families: three Intelligence agents that produce reports and signals, and the Observability-family Audit & Lineage Agent, which is held to a stricter invariant than the others.

Agent IDNameDecision scopeWhat it does
os-agt-005Analytics AgentrecommendAggregates cross-domain operational metrics from QuestDB, Supabase, PostHog, and Langfuse into structured performance reports and anomaly flags. It writes nothing except its own report records and anomaly-event emissions, and it never modifies source data or triggers execution workflows directly.
os-agt-006Forecasting AgentrecommendGenerates forward-looking projections from historical time-series and operational data across domains, using QuestDB for history and Qdrant for similar historical pattern retrieval. Its outputs are informational — acting on a forecast is always the responsibility of the requesting workflow or a human operator.
os-agt-007Anomaly Detection AgentrecommendContinuously monitors operational signals, workflow metrics, and event streams across all active domains, and emits structured anomaly events when a pattern falls outside configured normal bounds. It observes and signals only — remediation is always a separately triggered, separately governed workflow.
os-agt-008Audit & Lineage AgentrecommendCorrelates workflow execution traces, agent invocation records, policy decisions, and tool call logs into structured run lineage, answering explainability queries with a sourced reconstruction of what happened, why, and under whose authority. It is strictly read-only by design and must never appear in any agent’s tools_allowed list — it may never modify, redact, or amend audit records; any amendment to audit state is a human operator action with its own audit trail.

Layer principles that apply

  • Layer Principle 1 — Shared before specific. The canonical metric families, event envelope, and correlation identifiers are a shared OS substrate that every domain pack consumes rather than reimplementing its own metric variants — the Observability Metrics Spec states this explicitly as its Layer boundary.
  • Layer Principle 2 — Controls before execution. Observability is named alongside governance in this principle for a reason: execution paths must pass through both rather than bypassing either for convenience, which is why policy decisions and tool-authorization outcomes are themselves observable events, not side channels invisible to this layer.

Synedrix OS · Layer reference · Observability