Observability

Purpose link

The observability layer (Layer 10) of Synedrix OS is responsible for making every agent execution measurable, explainable, and reviewable. It does not own business decisions or domain state; it owns the signals that prove the system behaved as intended. Layer 10 sits alongside Layer 9 (Governance & Controls) in the “governance and observability” topology slice, and it draws on two Layer 7 stores — QuestDB for temporal aggregation and Neo4j for graph-based lineage — to answer questions that raw trace data alone cannot resolve.

The layer is required in all three deployment modes (cloud-connected, on-premises, and air-gapped). It is composed of three platform surfaces and two Layer 7 dependency stores, each owning a non-overlapping concern:

Every surface in this layer reads from the event substrate (Layer 8). Events are the single normalized ingestion path; platform-specific write paths are downstream consumers, not sources.

The observability invariant link

Core Principle 7 (Explainability and audit by design) requires the system to answer three questions about any action: what happened, why it happened, and under whose authority it happened. Layer 10 is the architectural home of those answers.

This requirement produces two structural invariants:

1. Traces are not an audit trail. A Langfuse trace is sampled, retention-bounded, and shaped for latency debugging. The lineage record in Neo4j and the append-only invocation start, tool-call, and outcome records in Supabase are neither sampled nor retention-bounded. Losing a trace is an operational inconvenience; losing a lineage record is a governance failure. The Audit & Lineage Agent (os-agt-008) is strictly read-only and may never modify, redact, or amend audit records.
2. Correlation identifiers are never lost. Every metric event, every trace span, and every alert carries a complete set of traceability identifiers — trace_id, span_id, parent_span_id from the Observability Metrics Spec’s event envelope (§6), plus correlation_id and causation_id from the Event Taxonomy Standard’s 15-field envelope (§2). An alert that cannot pivot directly into the underlying trace and the underlying Supabase record without re-deriving context is out of contract.

Canonical event envelope and metric families link

Event envelope link

Events feed all four observability surfaces from a single normalized ingestion path. The layer draws on two canonical envelopes:

• Observability Metrics Spec envelope (§6): The metric-specific envelope that wraps each metric-producing event with trace identifiers (trace_id, span_id, parent_span_id), producer metadata, and subject context. This envelope correlates metric events to the trace spans that produced them.

The observability envelope (defined in §6 of the Observability Metrics Spec) carries:

{
  "event_type": "string",
  "event_version": "v0.1",
  "event_id": "uuid",
  "occurred_at": "ISO-8601 timestamp",
  "producer": { "layer": "os", "component": "agent|workflow|service", "component_id": "string" },
  "trace": { "trace_id": "uuid", "span_id": "uuid", "parent_span_id": "uuid|null" },
  "subject": {
    "agent_id": "uuid|null", "workflow_id": "uuid|null", "invocation_id": "uuid|null",
    "conversation_id": "uuid|null", "tool_invocation_id": "uuid|null", "user_id": "string|null"
  },
  "payload": {}
}

Standardized event fields enable end-to-end monitoring in event-driven architectures and support consistent metric aggregation across producers and consumers. Every metric-producing event MUST precede or accompany its derived aggregates (principle: event first), and correlation identifiers MUST be propagated to every observation, not only the root trace.

Event-to-platform flow link

The event bus is the single normalized ingestion path; every observability surface receives data from it:

graph LR
    E["Event Bus (Layer 8)"] -->|15-field envelope| L["Langfuse (Layer 10)"]
    E -->|OTLP / metric events| G["Grafana LGTM (Layer 10)"]
    E -->|lifecycle events| P["PostHog (Layer 10)"]
    E -->|metric samples| Q["QuestDB (Layer 7)"]
    E -->|outcome events| N["Neo4j (Layer 7)"]
    L -->|trace spans + metadata| G
    Q -->|trend series| G
    P -->|flag state| G
    G -->|dashboard + alerts| R["Operations"]
    N -->|lineage graph| O["Audit & Lineage Agent"]
    L -->|evaluations| O
    style E fill:#f9f9f9
    style L fill:#e8f5e9
    style G fill:#e8f5e9
    style P fill:#e8f5e9
    style Q fill:#fff3e0
    style N fill:#fff3e0

Metric families link

The Observability Metrics Spec §7.1–7.11 defines eleven canonical metric families that every observability surface in the stack reads from the same names:

Every metric in these families is stored as an event first (§9 of the spec), and derived aggregates land in their platform of record through downstream consumers of the event bus. Dashboards MUST be organized per metric family rather than per agent, so cross-agent regressions in a single metric family remain visible even when no single agent crosses its own threshold.

Persistence model link

Audit-grade invocation records, evaluation results, response reviews, conversation metrics, and guardrail metrics are append-only relational rows in Supabase (§8 of the Observability Metrics Spec). The two-table pattern for agent lifecycle is:

• agent_invocations — inserted before execution; holds immutable identity and start-time dimensions (tenant_id, invocation_id, agent_id, workflow_id, conversation_id, user_id, started_at, correlation_id).
• agent_invocation_outcomes — inserted after execution; holds invocation status, completion time, and terminal metric fields joined by (tenant_id, invocation_id). A missing outcome means the invocation is in progress or awaiting timeout recovery.

Rolling trend series derived from these tables (cost_drift_pct over time, trust_trend_30d, tool_success_rate windows, retrieval quality trend) are written to QuestDB and MUST NOT be recomputed ad hoc from Supabase for dashboard use. The metrics_samples table with designated timestamp, day partitioning, and 90-day TTL feeds materialized views (mv_<metric_name>_<window>) that provide incremental refresh on every new sample.

Langfuse — LLM engineering link

Langfuse is the LLM engineering platform of Synedrix OS. It owns:

• Prompt versioning and label management
• LLM and agent trace capture (nested observations)
• Token and cost attribution
• Model-call latency and error visibility
• Evaluations and quality scoring (scores on traces, observations, sessions, and dataset runs)
• Nested trace debugging

One observability plane across Layer 3 and Layer 4 link

After ADR-OS-001 split the Agent Runtime (Layer 4) out of Orchestration (Layer 3), agent reasoning happens in two places: simple acyclic agents may still run as n8n AI Agent nodes, while iterative, consensus, reflect-and-revise and hierarchical patterns run in LangGraph. Both emit to Langfuse. The same Layer 10 platform used by n8n agents also handles LangGraph traces, so a single governed workflow — n8n trigger, policy gate, LangGraph reasoning step, outcome persistence — produces one correlated view of every model call it made, regardless of which engine made it.

The correlation model is key: Langfuse’s OTel attribute mapping is where the OS correlation identifiers land — langfuse.trace.metadata.* for the filterable top-level keys, langfuse.session.id and langfuse.user.id for the grouping attributes. Retrieval, groundedness, cost and semantic-validation events correlate 1:1 with a Langfuse trace span and MUST carry the same trace_id so a metric flag can be traced back to the exact LLM call that produced it.

Langfuse ingests via OpenTelemetry (OTLP over HTTP/JSON or HTTP/protobuf, not gRPC) at /api/public/otel, authenticated with Basic Auth. Ingestion is asynchronous and non-blocking; short-lived processes must explicitly call flush() before exiting. Langfuse’s storage components (Postgres, ClickHouse, Redis, S3) are infrastructure internal to the observability platform — they are NOT the OS’s Supabase and Redis.

Langfuse vs LangSmith link

ADR-OS-001 evaluated LangChain, LangGraph and LangSmith as three independent decisions and adopted the first two while rejecting LangSmith. The deciding axis is Principle 11 (Open by default, sovereign by design) + non-overlapping responsibilities: LangSmith is proprietary SaaS, incompatible with Mode 3 (air-gapped), and duplicates Langfuse. LangSmith MAY be used only as an optional developer-time convenience in cloud-connected Mode 1, never as a required platform and never in Modes 2 or 3.

Operational notes link

• Self-hosted Langfuse runs with no internet access; the only outbound call in a default OSS deployment is a cached version check that fails gracefully when blocked.
• Langfuse’s own Postgres and Redis are not the OS’s Postgres and Redis. Supabase PG17 is canonical relational truth; Redis is the OS coordination bus.
• Playground and LLM-as-a-Judge evaluators require an LLM endpoint. In Modes 2 and 3 that is the local inference layer (Ollama/vLLM) exposing an OpenAI-compatible API. Local and fine-tuned models need custom model definitions in Langfuse for cost inference to work.

Open items. Langfuse is open core, not uniformly MIT: ee/, web/src/ee/ and worker/src/ee/ are under a commercial Enterprise License. OSS telemetry is on by default and requires TELEMETRY_ENABLED=false to disable; if an EE licence key is ever introduced, telemetry cannot be disabled at all, which would break the Mode 3 invariant. The path by which n8n traces reach Langfuse depends on n8n’s own OpenTelemetry support or a gateway, since there is no native n8n tracing integration. These are implementation decisions not yet resolved in the canonical stack documents.

Grafana LGTM Stack — Operational monitoring link

The Grafana LGTM Stack is the operational monitoring surface of Synedrix OS. It is not one product but five composable open-source projects:

What it owns link

• Infrastructure and service health for every platform in the stack
• Dashboard rendering for the canonical metric families from the Observability Metrics Spec
• Threshold breaches and alert routing (per §15 of that spec)
• SLO management

Concretely, it covers: n8n main and worker processes, Redis queue depth, Supabase, QuestDB, Neo4j, Qdrant, the LangGraph runtime, and both Hookdeck gateways. Every alert MUST carry the same correlation identifiers as the triggering event so an on-call responder can pivot from the alert into the underlying Langfuse trace or the underlying Supabase record without re-deriving context.

OpenTelemetry is the ingestion standard link

Alloy runs as the single ingestion path, scraping Prometheus-format endpoints from the platform containers and receiving OTLP from application code, then forwarding metrics to Mimir, logs to Loki, and traces to Tempo. Grafana attaches all three as data sources. Nothing writes to a backend directly. OpenTelemetry is the common wire format across layers:

• n8n supports OpenTelemetry tracing via environment variables on self-hosted instances.
• LangGraph and LangChain emit traces to Langfuse over OpenTelemetry; aggregate metrics reach Mimir.
• OS events from the Event Taxonomy Standard carry trace_id and span_id.

Storage targets object storage (S3-compatible) rather than local disk, which makes the stack viable at OS scale without a dedicated storage tier per signal.

Licence implications link

Four of the five components are AGPLv3; only Alloy is Apache 2.0. AGPLv3’s network clause (§13) triggers when a modified version is made available over a network. Running unmodified upstream images is the low-friction path; forking or patching any component changes that obligation.

Open item. The canon documents (Tech Stack, Observability Metrics Spec, Deployment Model Spec) inconsistently name this platform as “Grafana + Prometheus” while the facts layer (canonical/architecture.yaml) and the deployed stack use “Grafana LGTM Stack.” A reconciliation of these prose documents — including the licence lines — is needed but not resolved here.

PostHog — Behavioural analytics link

PostHog is the behavioural analytics platform of Synedrix OS. It owns:

• Workflow analytics (funnel-shaped lifecycle tracking: triggered → validated → agent-invoked → approval requested → approved → executed)
• Feature flags (Layer 12 progressive rollout mechanism)
• A/B experiments
• Business decision analytics

Events reach PostHog through the event bus with canonical correlation identifiers. The event is the workflow or agent lifecycle event; the distinct ID is the identity analysis is grouped by. Choosing what plays “person” is a design decision per domain pack.

Feature flags as a governed rollout mechanism link

A flag is how a capability is progressively enabled for a concrete runtime without a redeploy. Two rules apply to any flag used this way:

1. A flag read MUST declare a fail-safe. If PostHog is unreachable or the flag is unavailable, the workflow must route to the safe branch.
2. A flag is NOT a policy engine. It decides whether a capability is switched on for this instance; it never decides whether a specific action is permitted — that is the Policy Guard Agent (os-agt-002).

Platform boundaries link

QuestDB — Temporal aggregation link

Although QuestDB sits at Layer 7 (Memory & Data Layer), it is an essential observability substrate: it holds the rolling trend series that dashboards and SLOs depend on. Its role is authoritative for how a measured value moved over time, and for nothing else.

QuestDB owns the metrics_samples table (with designated timestamp, metric_name, metric_family, tenant_id, value, dims, trace_id, and 90-day TTL) and materialized views named mv_<metric_name>_<window> (e.g. mv_trust_trend_30d, mv_tool_success_rate_60m). Dashboards MUST read trends from QuestDB or materialized views over it, not recompute them ad hoc from Supabase.

The OS signals that are time-series and therefore land here include:

• Agent execution durations (invocation latency distributions per agent and per workflow)
• Token and cost over time (total_tokens, total_cost_usd, cost_drift_pct against baseline)
• Tool latency and reliability (p95_latency_ms, tool_success_rate windows per tool_name/tool_version)
• Event volumes (emission rates from the Event Taxonomy Standard lifecycle events)
• Quality trends (retrieval quality, groundedness, hallucination probability, semantic validity, coherence scores)
• Trust and SLO measurements (trust_score_30d_avg, trust_trend_30d, rolling windows behind SLO thresholds)

Retrieval spans MUST be exported to Langfuse nested under the requesting agent’s trace. Memory fabric observability metrics (growth rate, records by tier, staleness ratio, consolidation lag) are also aggregated here from the memory.* event stream.

Retention is expressed at the table level via PARTITION BY plus TTL, dropping old partitions as directory operations rather than expensive deletes.

Open item. QuestDB’s Apache 2.0 core offers no built-in high availability or disaster recovery for the temporal layer — replication is Enterprise only. A single-node QuestDB is a single point of failure for every trend series, SLO window, and materialized view. Options include accepting single-node operation with backup-and-restore, treating temporal data as reconstructible from the event bus, or licensing Enterprise.

Neo4j — Decision lineage link

Neo4j sits at Layer 7 but is a foundational component of Layer 10’s explainability mandate. It holds the traversable projection of lineage from the Outcome Spec, answering the “why it happened and under whose authority” questions that Core Principle 7 requires.

Lineage is a governance invariant link

The Audit & Lineage Agent (os-agt-008) queries Neo4j by outcome_id, correlation_id, and causation_id to reconstruct causal chains. The agent is strictly read-only by design and its systems_allowed includes Neo4j (read). The enforcing invariant is explicit: os-agt-008 must never appear in any agent’s tools_allowed list — it is a query target, not a callable tool. No agent can launder a write to lineage through the audit agent; lineage is written by the outcome path and read by one read-only agent.

What is in the graph link

The outcome envelope is the unit of lineage. Links in Neo4j are written as source_outcome_id → target_outcome_id with relation types: invoked, contributed-to, escalated-to, governed-by. The source of truth is agent_lineage_links in Supabase, projected into Neo4j. Losing the graph loses query capability, not evidence.

Node types beyond the outcome include Agent, Workflow, and Tool (per ADR-OS-007), with relationships such as PRODUCED, CAN_ESCALATE_TO, AUTHORIZED_FOR, OWNS_WORKFLOW, and TOUCHED (from NAMS).

correlation_id is the partition key for all traversals; causation_id is the edge that names each direct predecessor. The ordering rule matters because timestamps do not reliably order a causal chain across async boundaries — the causation_id chain does. Escalation-path reachability (every agent’s path terminates at os-agt-001 or os-agt-004) is a traversal predicate over the union of all declared paths, checked in a single query rather than one agent at a time.

Neo4j also backs the graph memory tier through the NAMS client (neo4j-labs/agent-memory). NAMS models long-term memory as a POLE+O graph (Person, Organization, Location, Event, plus other objects) and records agent reasoning as AgentStep and ToolCall nodes joined to entities they affected by :TOUCHED edges. NAMS writes go through the control plane only — the Memory Curator (os-agt-010) after the Memory Access Broker (os-agt-012) resolves scope.

Open items. Community Edition lacks role-based access control (the os-agt-008 read-only guarantee is enforced only at the OS-layer tool, not the database), no Prometheus metrics endpoint, and offline-only backup (rebuild-from-Postgres is the routine runbook). Enterprise-only features include RBAC, online backup, and the metrics endpoint — governance features, not scale features. Community Edition’s GPLv3 copyleft is material to any deployment model where Synedrix OS is delivered as a product. NAMS is an experimental Neo4j Labs project with community support.

Platform boundaries and non-overlap link

Layer 10 has three platforms and two Layer 7 stores. The boundaries are ownership boundaries, not preferences:

Three boundaries deserve particular care:

• Langfuse vs PostHog: Both platforms have opinions about agent quality. Langfuse owns the LLM level (trace span, prompt version, token cost, evaluation score). PostHog owns the behavioural level (workflow completion, abandonment, human override rate). A hallucination score is a Langfuse fact. “Operators reject this agent’s proposals 40 % of the time” is a PostHog fact.
• Langfuse vs Neo4j: Langfuse traces are evidence of how a model call executed. The Outcome Envelope, its os.outcome.* events, and Neo4j lineage are the record of what the system decided. The two are structurally different.
• Grafana vs all others: A Grafana panel may display an aggregate derived from any backend, but it is never where that data is stored. Grafana is the operational surface, not the source of record.

Evaluation gates and promotion link

The observability metric families feed directly into the evaluation framework. Agents without declared evaluation suites must not graduate from draft to active status in the registry. The default gate classification per family:

Default SLO/alert thresholds (§15 of the Observability Metrics Spec):

Each alert MUST carry the same correlation identifiers as the triggering event so the on-call responder can pivot into the underlying trace or record without re-deriving context.

How the layer integrates with other layers link

Layer 10 does not exist in isolation. Its interfaces with every other layer define the end-to-end traceability property:

• Layer 3 (Orchestration): n8n workflows emit lifecycle events to the event bus; n8n’s OTel tracing feeds Alloy into the LGTM Stack. The os.workflow.* event families (workflow.started, workflow.completed, workflow.failed) feed PostHog funnels.
• Layer 4 (Agent Runtime): LangGraph agents emit traces to Langfuse via OTel or the native exporter. LangChain agents in n8n AI Agent nodes do the same. Every agent invocation produces an os.agent.* event.
• Layer 7 (Memory & Data): QuestDB holds the temporal aggregation of every metric produced in the system. Neo4j holds the graph projection of lineage from agent_lineage_links in Supabase. The graph tier is written through NAMS via the control plane.
• Layer 8 (Event): The observability layer consumes the same 15-field envelope every other consumer does. Emission is not optional for the lifecycle events in the Event Taxonomy Standard. The event bus is the single normalized ingestion path.
• Layer 9 (Governance): Policy Guard and HITL Gateway decisions are recorded as events that flow into both PostHog (behavioural) and Neo4j (lineage). Guardrail triggers and false-positive rates feed back to the governance loop to decide whether a guardrail moves from monitor to enforce mode.
• Layer 10 (Observability): The layer’s own agents — Analytics (os-agt-005), Forecasting (os-agt-006), Anomaly Detection (os-agt-007), and Audit & Lineage (os-agt-008) — read from the observability platforms to produce reports, scores, and lineage reconstructions. Their decision scope is recommend; they report, they do not act.
• Layer 12 (Business Instance): Business instances configure thresholds, enable capabilities through feature flags, and select which dashboards and alerts to activate. They do not redefine canonical metric names or event envelopes.

Deployment notes link

All three Layer 10 platforms and both Layer 7 dependency stores are required in all three deployment modes (Mode 1: cloud-connected, Mode 2: on-premises, Mode 3: air-gapped). Each platform’s licence and self-hosting characteristics are documented on its platform page. The stack-wide deployment reference is the Deployment Model Spec.

Several open items require explicit architectural decisions before production deployment:

• Langfuse open core / EE licence and telemetry configuration
• Neo4j Community vs Enterprise licence for governance features (RBAC, metrics, online backup)
• QuestDB single-node availability vs replication licensing
• PostHog self-hosted scale limits and Cloud-only features
• “Grafana LGTM Stack” vs “Grafana + Prometheus” nomenclature reconciliation in prose docs
• The n8n-to-Langfuse trace path

These are tracked in the respective platform pages and should be resolved through ADR before production rollout.