Observability Metrics Spec

1. Document Control link

• Document name: synedrix-os-observability-metrics-spec-v1.2
• Layer: Synedrix OS Platform (Layer 1)
• Status: Draft candidate
• Scope: Canonical metric definitions, event payloads, and persistence rules for OS-wide agent observability
• Normative language: The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as binding specification language
• Supersedes: v1.1. v1.2 makes §8.2–8.6 sketches tenant-led; v1.1 adds QuestDB trend DDL; v1.0 moves invocation completion and metric fields to agent_invocation_outcomes.

2. Purpose link

This specification defines the canonical observability metrics required for Synedrix OS to measure retrieval quality, groundedness and hallucination risk, execution cost drift, semantic output validity, user-facing review sampling, multi-turn coherence, and tool reliability at OS scope. These metrics are defined as first-class platform concerns so that domain packs consume a consistent observability substrate rather than creating domain-specific metric variants.

This specification complements the event taxonomy, agent database schema, runtime policy, and agent evaluation specifications by standardizing how metric events are emitted, how metric values are stored, and how downstream policy consumers interpret metric flags.

3. Authority and Layer Boundary link

This document governs Layer 1 — Synedrix OS Platform only. Domain packs, including Synedrix Trading, MUST consume these metrics and MAY define local thresholds or workflow reactions, but MUST NOT redefine canonical metric names, alter OS-owned event envelopes, or add columns to OS-owned tables from a domain context.

This split preserves a reusable OS substrate and aligns with event-driven observability practice, where common telemetry schemas and correlation identifiers are standardized centrally to support auditability, aggregation, and policy reuse across systems.

4. Objectives link

Synedrix OS observability MUST satisfy the following objectives:

• Measure retrieval quality per call using graded metrics rather than binary success states.
• Estimate response groundedness and hallucination risk using evidence-linked scoring rather than heuristic pass/fail logic.
• Track execution cost per invocation against a rolling baseline and raise drift flags when cost increases exceed policy thresholds.
• Validate structured outputs at both structural and semantic levels because JSON conformance alone does not guarantee meaningful correctness.
• Sample user-facing responses for human review using random selection and policy-targeted escalation to improve quality assurance coverage.
• Score coherence across full conversations rather than only at single-turn scope.
• Measure tool reliability per tool and per version because aggregated success rates can mask unstable dependencies.

5. Normative Principles link

The following principles apply to all metrics defined by this specification:

1. Event first: Raw metric-producing events MUST be emitted before or alongside derived aggregates.
2. Correlation by default: Every metric event MUST include stable correlation identifiers for agent, workflow, invocation, conversation, and trace when applicable.
3. Append-only evaluation: Invocation records, lifecycle records, and compliance-related events MUST remain append-only; evaluators SHOULD write new results rather than rewriting history.
4. Canonical naming: Metric field names MUST be reused exactly across tables, events, dashboards, and policy rules.
5. Scope fidelity: Invocation-scoped metrics MUST NOT be stored as conversation metrics, and conversation-scoped metrics MUST NOT be collapsed into invocation fields without an explicit derived metric rule.
6. Structural vs semantic separation: Structural validation and semantic validation MUST be treated as distinct stages with distinct outputs.

6. Canonical Event Envelope link

All events defined by this specification MUST use the following canonical envelope:

{
  "event_type": "string",
  "event_version": "v0.1",
  "event_id": "uuid",
  "occurred_at": "ISO-8601 timestamp",
  "producer": {
    "layer": "os",
    "component": "agent|workflow|service",
    "component_id": "string"
  },
  "trace": {
    "trace_id": "uuid",
    "span_id": "uuid",
    "parent_span_id": "uuid|null"
  },
  "subject": {
    "agent_id": "uuid|null",
    "workflow_id": "uuid|null",
    "invocation_id": "uuid|null",
    "conversation_id": "uuid|null",
    "tool_invocation_id": "uuid|null",
    "user_id": "string|null"
  },
  "payload": {}
}

Standardized event fields improve traceability, enable end-to-end monitoring in event-driven architectures, and support consistent metric aggregation across producers and consumers.

7. Canonical Metric Families link

7.1 Retrieval Quality link

Retrieval quality MUST be measured per invocation when retrieval is used. The canonical retrieval metrics are retrieval_precision_at_k, retrieval_recall_at_k, retrieval_f1_at_k, and retrieval_quality_score, where the default roll-up SHOULD be retrieval_f1_at_k unless a future version defines another aggregation rule.

The field retrieval_used MUST indicate whether retrieval participated in the invocation. When retrieval is not used, retrieval metric fields MUST be null rather than zero to avoid contaminating aggregates.

7.2 Groundedness and Hallucination link

Groundedness MUST be represented by grounded_fact_ratio and faithfulness_score, both normalized to the range 0 to 1. Hallucination risk MUST be represented by hallucination_probability, also normalized to 0 to 1, and MAY be derived as 1 - faithfulness_score when the evaluator implementation uses a supported-claims interpretation of faithfulness.

The boolean field hallucination_flag MUST represent a runtime or evaluation threshold crossing and MUST NOT replace the underlying scalar score.

7.3 Cost and Drift link

Execution cost MUST be stored per invocation using input_tokens, output_tokens, tool_tokens, total_tokens, model_unit_cost_usd, and total_cost_usd. Rolling cost comparison MUST use baseline_cost_window_days, baseline_cost_usd, and cost_drift_pct, with cost_drift_flag set when the configured threshold is exceeded.

Default policy guidance for v0.1 is a drift threshold of +30 percent relative to the baseline slice.

7.4 Semantic Output Validity link

Structured outputs MUST be evaluated at two layers:

• Structural validity, using a schema validator to confirm JSON shape and required fields.
• Semantic validity, using rule checks or evaluator models to confirm that field values are meaningful in context.

The canonical scalar output is semantic_validity_score, with semantic_validity_flag indicating threshold failure.

7.5 Response Sampling and Human Review link

User-facing responses MUST support unbiased quality review by random sampling. The canonical response-review mechanism MUST emit a review sample event when a response is selected and MUST persist review outcomes separately from the original invocation.

The default v0.1 policy target is a 10 percent random sample of user-facing finalized responses, with optional oversampling for risk conditions.

7.6 Multi-turn Coherence link

Conversation quality MUST support multi-turn evaluation via coherence_score and goal_alignment_score stored at conversation scope. These metrics SHOULD be computed over the full conversation or an explicitly declared evaluation window, not inferred from a single invocation.

7.7 Tool Reliability link

Tool reliability MUST be measured at raw-call scope and aggregated per tool_name and tool_version. The canonical aggregate metrics are tool_calls_total, tool_calls_succeeded, tool_calls_failed, tool_calls_retried, tool_success_rate, and p95_latency_ms.

7.8 Risk-aware Decision Alignment link

Risk-aware decision alignment MUST measure whether an agent’s recommendation, draft, or action remained within declared policy, authority, and risk boundaries for the invocation. The canonical metrics are risk_alignment_score, risk_policy_violations_count, blocked_actions_count, and unsafe_recommendation_flag. This metric family is invocation-scoped and SHOULD be promotion-blocking for agents operating in medium, high, or critical risk tiers.

7.9 Policy and Guardrail Effectiveness link

Policy and guardrail effectiveness MUST evaluate whether a guardrail catches genuinely unsafe or invalid outputs without excessively blocking valid ones. The canonical aggregate metrics are guardrail_triggers_total, guardrail_true_positive_rate, guardrail_false_positive_rate, guardrail_flag_rate, and guardrail_latency_ms. This metric family is guardrail-scoped rather than invocation-scoped and SHOULD govern whether a guardrail is eligible to move from monitoring mode into enforced runtime policy.

7.10 User Trust and Satisfaction Trend link

User trust and satisfaction trend MUST capture how end users experience the system over time rather than how a single response scored in isolation. The canonical metrics are user_trust_score, conversation_satisfaction_score, frustration_signals_count, trust_score_30d_avg, and trust_trend_30d. This metric family is conversation-scoped for raw values and agent-scoped for rolling aggregates, and SHOULD be used as a promotion-gate dimension for user-facing agents rather than as a per-call runtime block.

7.11 Memory Control Plane link

Memory operations MUST be measured with the same families that already exist wherever one applies. Recall results are retrieval results: retrieval_precision_at_k, retrieval_recall_at_k and the §7.2 groundedness metrics apply to tool:memory-recall unchanged, and MUST NOT be duplicated under memory-specific names.

Four families have no existing equivalent and are canonical here.

Recall mediation. recall_scope_applied (the resolved scope, as a string), recall_tiers_requested, recall_tiers_granted, recall_tiers_denied, recall_denial_rate (agent-scoped, rolling). A rising denial rate for a stable agent indicates a memory_profile that no longer matches what the agent is being asked to do, and SHOULD be treated as a configuration signal rather than an incident.

Consolidation lag. consolidation_lag_ms, measured from memory.record.formed to memory.record.consolidated for the same memory_id. Because consolidation runs in the background, lag is the metric that reveals whether the background path is keeping up; it MUST be tracked per tenant, since a single tenant’s ingest volume can starve the queue for others.

Fabric growth and staleness. memory_records_total and memory_records_by_tier (gauges), memory_growth_rate_daily, and memory_stale_ratio — the proportion of records whose validity_interval has closed but which remain un-superseded. A rising stale ratio means the fabric is accumulating assertions nothing has revisited, which degrades recall precision before it shows up as an error.

Promotion and erasure. promotion_attempts_total, promotion_rejection_rate, erasure_requests_total, erasure_completion_time_ms. Erasure completion time SHOULD be tracked against the business instance’s declared SLA, because it is a compliance commitment rather than a performance target.

Every metric in this family MUST carry correlation_id and, where the operation is tier-scoped, the tier. Recall spans MUST be exported to Langfuse nested under the requesting agent’s trace, and MUST record tiers requested, tiers denied, record count and the scope applied — a recall that cannot be attributed to a requesting agent is not observable in the sense §5 requires.

8. Persistence Model link

8.1 Tables: agent_invocations and agent_invocation_outcomes link

agent_invocations is inserted before execution and holds immutable identity and start-time dimensions. Its tenant-scoped key is (tenant_id, invocation_id); agent_id, workflow_id, conversation_id, user_id, and started_at are read from this table. Invocation status, completion time, and terminal metrics MUST be stored in exactly one append-only agent_invocation_outcomes row for the same (tenant_id, invocation_id). A missing outcome means the invocation is in progress or awaiting timeout recovery.

-- agent_invocations (start record)
tenant_id                    text not null,
invocation_id                uuid not null,
agent_id                     text not null,
workflow_id                  text null,
conversation_id              uuid null,
user_id                      text null,
started_at                   timestamptz not null,
correlation_id                text not null,
primary key (tenant_id, invocation_id)

-- agent_invocation_outcomes (terminal row)
tenant_id                    text not null,
invocation_id                uuid not null,
outcome_id                   uuid not null,
status                       text not null,
completed_at                 timestamptz not null,
latency_ms                   integer not null,
model_id                     text null,
input_tokens                 integer null,
output_tokens                integer null,
tool_tokens                  integer null,
total_tokens                 integer null,
model_unit_cost_usd          numeric(12,6) null,
total_cost_usd               numeric(12,6) null,
retrieval_used               boolean not null default false,
retrieval_k                  integer null,
retrieved_doc_count          integer null,
retrieval_precision_at_k     double precision null,
retrieval_recall_at_k        double precision null,
retrieval_f1_at_k            double precision null,
retrieval_quality_score      double precision null,
faithfulness_score           double precision null,
grounded_fact_ratio          double precision null,
hallucination_probability    double precision null,
hallucination_flag           boolean null,
semantic_validity_score      double precision null,
semantic_validity_flag       boolean null,
baseline_cost_window_days    integer null,
baseline_cost_usd            numeric(12,6) null,
cost_drift_pct               double precision null,
cost_drift_flag              boolean null,
primary key (tenant_id, invocation_id),
unique (tenant_id, outcome_id),
foreign key (tenant_id, invocation_id)
  references agent_invocations (tenant_id, invocation_id)

The outcome table also carries terminal result, confidence, error, escalation, and retry-summary fields defined by the Outcome Spec and Agent DB Schema. These invocation-level metric fields support per-call analysis; join them to their immutable start dimensions by the composite tenant key. Retry attempts themselves are append-only runtime events, not updates to an outcome.

8.2 Table: tool_invocations link

Tenant-owned observational aggregate. Prefer joining Agent DB agent_tool_call_records / agent_tool_call_results for audit; this table is the metrics-facing projection. Full ownership consolidation with Agent DB remains deferred (Layer 7 gap register).

tenant_id                    text not null,
id                           uuid not null,
invocation_id                uuid not null,
agent_id                     text not null,
workflow_id                  text null,
tool_name                    text not null,
tool_version                 text null,
status                       text not null,
error_type                   text null,
latency_ms                   integer null,
request_bytes                integer null,
response_bytes               integer null,
started_at                   timestamptz not null,
completed_at                 timestamptz null,
primary key (tenant_id, id)

This table captures raw tool outcomes so that reliability is measurable by tool, version, and calling agent rather than hidden in blended aggregates.

8.3 Table: evaluation_results link

Tenant-owned. Prefer Agent DB Group E (agent_eval_results) where the evaluation is registry-bound; this sketch remains for observational metrics that are not promotion-gate records. Ownership consolidation deferred.

tenant_id                    text not null,
id                           uuid not null,
subject_type                 text not null,
subject_id                   text not null,
metric_family                text not null,
metric_name                  text not null,
metric_value                 double precision null,
metric_label                 text null,
evaluator_agent_id           text null,
evaluator_version            text null,
explanation                  text null,
details_json                 jsonb null,
created_at                   timestamptz not null,
primary key (tenant_id, id)

This table MUST store evaluator outputs that do not belong as base invocation fields or that require richer detail, explanation, or versioned evaluator metadata.

8.4 Table: response_reviews link

Tenant-owned.

tenant_id                    text not null,
id                           uuid not null,
invocation_id                uuid not null,
conversation_id              uuid null,
sampling_reason              text not null,
review_status                text not null default 'pending',
reviewer_id                  uuid null,
human_quality_score          double precision null,
human_hallucination_flag     boolean null,
human_groundedness_score     double precision null,
human_comments               text null,
created_at                   timestamptz not null,
review_completed_at          timestamptz null,
primary key (tenant_id, id)

8.5 Table: conversation_metrics link

Tenant-owned.

tenant_id                    text not null,
conversation_id              uuid not null,
turn_count                   integer not null,
coherence_score              double precision null,
goal_alignment_score         double precision null,
user_trust_score             double precision null,
conversation_satisfaction_score double precision null,
frustration_signals_count    integer null,
evaluated_at                 timestamptz null,
evaluator_agent_id           text null,
primary key (tenant_id, conversation_id)

Conversation-scoped persistence is required because coherence and goal alignment are properties of multi-turn interaction, not isolated invocations.

8.6 Table: guardrail_metrics link

Tenant-owned.

tenant_id                    text not null,
id                           uuid not null,
guardrail_id                 text not null,
agent_id                     text null,
window_start_at              timestamptz not null,
window_end_at                timestamptz not null,
guardrail_triggers_total     integer not null,
guardrail_true_positive_rate double precision null,
guardrail_false_positive_rate double precision null,
guardrail_flag_rate          double precision null,
guardrail_latency_ms         integer null,
created_at                   timestamptz not null,
primary key (tenant_id, id)

8.7 Invocation Extensions for Risk Alignment link

The OS invocation table SHOULD also expose the following risk-alignment fields for agents whose authority can produce material side effects:

risk_alignment_score         double precision null,
risk_policy_violations_count integer null,
blocked_actions_count        integer null,
unsafe_recommendation_flag   boolean null

9. Canonical Events link

9.1 Retrieval Evaluation Event link

{
  "event_type": "os.invocation.retrieval_evaluated",
  "event_version": "v0.1",
  "event_id": "4f44f3db-9769-48fb-8f1a-a8284e2dff61",
  "occurred_at": "2026-05-13T09:22:14.222Z",
  "producer": {
    "layer": "os",
    "component": "agent",
    "component_id": "retrieval-evaluator-agent"
  },
  "trace": {
    "trace_id": "2f5ad26e-2658-4ab8-98ee-ef5609d2896f",
    "span_id": "83c0c2fb-f653-4eff-9f1b-9f126d2b8e25",
    "parent_span_id": "dc4b0bc1-177a-4062-a0f6-f5f9ea626097"
  },
  "subject": {
    "agent_id": "a-08",
    "workflow_id": "wf-07",
    "invocation_id": "inv-100245",
    "conversation_id": "conv-8871",
    "tool_invocation_id": null,
    "user_id": "usr-214"
  },
  "payload": {
    "retrieval_used": true,
    "retrieval_k": 8,
    "retrieved_doc_ids": ["doc_17", "doc_88", "doc_203", "doc_301"],
    "relevant_doc_ids": ["doc_17", "doc_203", "doc_411"],
    "metrics": {
      "precision_at_k": 0.50,
      "recall_at_k": 0.67,
      "f1_at_k": 0.57,
      "retrieval_quality_score": 0.57
    },
    "label_source": "offline_goldset"
  }
}

9.2 Groundedness Evaluation Event link

{
  "event_type": "os.invocation.groundedness_evaluated",
  "event_version": "v0.1",
  "event_id": "fbc55f05-6be8-4dfc-a3b8-741ce8281b2b",
  "occurred_at": "2026-05-13T09:22:15.012Z",
  "producer": {
    "layer": "os",
    "component": "agent",
    "component_id": "hallucination-evaluator-agent"
  },
  "trace": {
    "trace_id": "2f5ad26e-2658-4ab8-98ee-ef5609d2896f",
    "span_id": "cdb42a52-29a8-42a5-970e-754d1eddb6d5",
    "parent_span_id": "83c0c2fb-f653-4eff-9f1b-9f126d2b8e25"
  },
  "subject": {
    "agent_id": "a-08",
    "workflow_id": "wf-07",
    "invocation_id": "inv-100245",
    "conversation_id": "conv-8871",
    "tool_invocation_id": null,
    "user_id": "usr-214"
  },
  "payload": {
    "claim_count": 11,
    "grounded_claim_count": 9,
    "unsupported_claim_count": 2,
    "metrics": {
      "grounded_fact_ratio": 0.82,
      "faithfulness_score": 0.82,
      "hallucination_probability": 0.18,
      "hallucination_flag": false
    },
    "evidence_mode": "retrieved_context_plus_tool_outputs",
    "explanation": "2 claims were not supported by retrieved evidence."
  }
}

Groundedness and faithfulness-style scoring are widely used to evaluate whether model outputs are supported by evidence, which makes them a suitable basis for a canonical hallucination-risk metric family.

9.3 Cost Evaluation Event link

{
  "event_type": "os.invocation.cost_evaluated",
  "event_version": "v0.1",
  "event_id": "a3ab357b-8359-4218-8612-454fc6ea3433",
  "occurred_at": "2026-05-13T09:22:16.100Z",
  "producer": {
    "layer": "os",
    "component": "service",
    "component_id": "cost-drift-evaluator"
  },
  "trace": {
    "trace_id": "2f5ad26e-2658-4ab8-98ee-ef5609d2896f",
    "span_id": "a48ff5d7-3637-4bd6-8912-a3f6f20fc7b5",
    "parent_span_id": null
  },
  "subject": {
    "agent_id": "a-08",
    "workflow_id": "wf-07",
    "invocation_id": "inv-100245",
    "conversation_id": "conv-8871",
    "tool_invocation_id": null,
    "user_id": "usr-214"
  },
  "payload": {
    "model_id": "gpt-5.4-medium",
    "token_usage": {
      "input_tokens": 2140,
      "output_tokens": 811,
      "tool_tokens": 402,
      "total_tokens": 3353
    },
    "costs": {
      "model_unit_cost_usd": 0.000600,
      "total_cost_usd": 1.41,
      "baseline_window_days": 14,
      "baseline_cost_usd": 1.02,
      "cost_drift_pct": 38.24,
      "cost_drift_flag": true
    },
    "baseline_slice": {
      "agent_id": "a-08",
      "model_id": "gpt-5.4-medium",
      "workflow_id": "wf-07"
    }
  }
}

9.4 Semantic Validation Event link

{
  "event_type": "os.invocation.semantic_validation_completed",
  "event_version": "v0.1",
  "event_id": "36f3ee9f-0c03-48ae-a916-caf0df1f62e1",
  "occurred_at": "2026-05-13T09:22:17.040Z",
  "producer": {
    "layer": "os",
    "component": "agent",
    "component_id": "semantic-validator-agent"
  },
  "trace": {
    "trace_id": "2f5ad26e-2658-4ab8-98ee-ef5609d2896f",
    "span_id": "5cc33098-e460-426f-ab5c-0b9a5b7c9791",
    "parent_span_id": null
  },
  "subject": {
    "agent_id": "a-08",
    "workflow_id": "wf-07",
    "invocation_id": "inv-100245",
    "conversation_id": "conv-8871",
    "tool_invocation_id": null,
    "user_id": "usr-214"
  },
  "payload": {
    "schema_id": "trade_recommendation_v1",
    "structural_validation_passed": true,
    "semantic_checks": [
      {
        "name": "ticker_supported_by_broker",
        "passed": true
      },
      {
        "name": "risk_budget_within_account_limit",
        "passed": false,
        "severity": "high",
        "message": "Requested risk budget exceeds configured account maximum."
      },
      {
        "name": "time_horizon_matches_strategy_class",
        "passed": true
      }
    ],
    "metrics": {
      "semantic_validity_score": 0.67,
      "semantic_validity_flag": false
    }
  }
}

JSON Schema and related validators are effective for structural correctness, but semantic correctness requires additional rule logic or evaluator layers because valid shape does not imply valid meaning.

9.5 Response Sampling Event link

{
  "event_type": "os.response.sampled_for_review",
  "event_version": "v0.1",
  "event_id": "7e27a4fc-5864-403c-a00f-7c331fa475a5",
  "occurred_at": "2026-05-13T09:22:17.900Z",
  "producer": {
    "layer": "os",
    "component": "service",
    "component_id": "sampling-orchestrator"
  },
  "trace": {
    "trace_id": "2f5ad26e-2658-4ab8-98ee-ef5609d2896f",
    "span_id": "6c03f2ee-3653-410f-84f1-f71b49c613aa",
    "parent_span_id": null
  },
  "subject": {
    "agent_id": "a-08",
    "workflow_id": "wf-07",
    "invocation_id": "inv-100245",
    "conversation_id": "conv-8871",
    "tool_invocation_id": null,
    "user_id": "usr-214"
  },
  "payload": {
    "sampling_policy": "daily_random_10pct",
    "sampling_reason": "random_10pct",
    "review_queue": "daily-human-review",
    "priority": "normal",
    "snapshot": {
      "hallucination_probability": 0.18,
      "retrieval_quality_score": 0.57,
      "semantic_validity_score": 0.67
    }
  }
}

9.6 Conversation Coherence Event link

{
  "event_type": "os.conversation.coherence_evaluated",
  "event_version": "v0.1",
  "event_id": "c09505b1-c8b0-4195-856d-3f9dc9c497dd",
  "occurred_at": "2026-05-13T10:00:00.000Z",
  "producer": {
    "layer": "os",
    "component": "agent",
    "component_id": "conversation-evaluator-agent"
  },
  "trace": {
    "trace_id": "9be53652-0b09-4958-8bc5-865215a0358f",
    "span_id": "1cda87c4-e8c0-418f-a957-8576c527540c",
    "parent_span_id": null
  },
  "subject": {
    "agent_id": null,
    "workflow_id": null,
    "invocation_id": null,
    "conversation_id": "conv-8871",
    "tool_invocation_id": null,
    "user_id": "usr-214"
  },
  "payload": {
    "turn_count": 14,
    "evaluation_window": "full_conversation",
    "metrics": {
      "coherence_score": 0.84,
      "goal_alignment_score": 0.79
    },
    "signals_used": [
      "message_transcript",
      "tool_outcomes",
      "retrieval_quality_scores",
      "hallucination_probabilities"
    ],
    "notes": "Conversation remained consistent, but two turns drifted from the user's requested objective."
  }
}

9.7 Tool Invocation Event link

{
  "event_type": "os.tool.invocation_completed",
  "event_version": "v0.1",
  "event_id": "f5e10a90-6782-45d0-b374-ec9cf502ef53",
  "occurred_at": "2026-05-13T09:22:14.900Z",
  "producer": {
    "layer": "os",
    "component": "agent",
    "component_id": "a-08"
  },
  "trace": {
    "trace_id": "2f5ad26e-2658-4ab8-98ee-ef5609d2896f",
    "span_id": "ee7c5f89-d753-48b0-a7dd-aec6bc890441",
    "parent_span_id": "dc4b0bc1-177a-4062-a0f6-f5f9ea626097"
  },
  "subject": {
    "agent_id": "a-08",
    "workflow_id": "wf-07",
    "invocation_id": "inv-100245",
    "conversation_id": "conv-8871",
    "tool_invocation_id": "tool-55881",
    "user_id": "usr-214"
  },
  "payload": {
    "tool_name": "market_data_lookup",
    "tool_version": "v2.3.1",
    "status": "failed",
    "error_type": "timeout",
    "latency_ms": 3012,
    "retry_count": 1,
    "eventual_success": false
  }
}

9.8 Tool Reliability Aggregate Event link

{
  "event_type": "os.tool.reliability_aggregated",
  "event_version": "v0.1",
  "event_id": "b2418c6a-68fe-4281-a1f2-66438f6cb8c2",
  "occurred_at": "2026-05-13T10:00:00.000Z",
  "producer": {
    "layer": "os",
    "component": "service",
    "component_id": "tool-reliability-evaluator"
  },
  "trace": {
    "trace_id": "e9ca79bc-b27e-4f3f-946b-d26f0ee1310e",
    "span_id": "22556374-96a9-49da-a14f-6c4d06545e9b",
    "parent_span_id": null
  },
  "subject": {
    "agent_id": null,
    "workflow_id": null,
    "invocation_id": null,
    "conversation_id": null,
    "tool_invocation_id": null,
    "user_id": null
  },
  "payload": {
    "window_minutes": 60,
    "group_by": ["tool_name", "tool_version"],
    "tool_name": "market_data_lookup",
    "tool_version": "v2.3.1",
    "metrics": {
      "tool_calls_total": 1280,
      "tool_calls_succeeded": 1204,
      "tool_calls_failed": 76,
      "tool_calls_retried": 54,
      "tool_success_rate": 0.9406,
      "p95_latency_ms": 2110
    },
    "flagged": true,
    "flag_reason": "success_rate_below_threshold"
  }
}

10. Evaluation and Policy Rules link

Synedrix OS MUST support the following minimum policy behavior:

• A completed invocation MUST be eligible for retrieval evaluation when retrieval_used = true.
• A completed invocation SHOULD be eligible for groundedness evaluation whenever retrieved context, tool outputs, or cited evidence informed the response.
• Cost drift MUST be computed against a rolling baseline slice at minimum by (agent_id, model_id) and SHOULD include workflow_id when workflow-specific variance is material.
• Semantic validation MUST NOT be inferred from structural validation success alone.
• Every user-facing finalized response MUST be eligible for random sampling, with a default target of 10 percent.
• Every tool invocation MUST be recorded regardless of success or failure so that reliability statistics remain unbiased.
• Agents in medium, high, or critical risk tiers SHOULD be evaluated on risk_alignment_score, and promotion to active SHOULD be blocked when risk-policy violations are observed in blocking evaluation suites.
• Guardrails SHOULD graduate from monitor mode to enforce mode only when true-positive and false-positive performance is within policy tolerance, because unstable guardrails can degrade system quality even when their intent is correct.
• User trust and satisfaction metrics SHOULD influence promotion and lifecycle decisions for interface-facing agents, but SHOULD NOT be used as direct per-call runtime vetoes.

11. Domain Pack Consumption Rules link

Domain packs MAY define workflow reactions to canonical metric outputs, including threshold-based escalation, review routing, fallback behavior, and dashboard filtering. Domain packs MUST NOT redefine canonical OS metric families, MUST NOT fork OS event envelopes, and MUST NOT extend OS-owned tables directly from domain contexts.

For Synedrix Trading, these metrics MAY be consumed by n8n workflows as read-only policy signals, but they MUST NOT bypass trading invariants such as Agent 17 validation, Agent 11 veto, promotion gates, or any Architecture v6.0 non-negotiable rule.

12. Versioning and Compatibility link

This document is version 1.2. Supersedes v1.1. Additive changes do not invalidate prior metric family names.

Changes to canonical field names, envelope structure, or event type names require a spec version increment. Additive fields MAY be introduced in a minor version when backward compatibility is preserved. Any change that alters baseline-slice semantics, hallucination interpretation, or persistence scope MUST be treated as a breaking change.

13. Implementation Notes link

A practical rollout order for v0.1 is:

1. Add canonical fields and tables to the OS schema.
2. Emit raw tool and invocation events with trace correlation.
3. Add retrieval, groundedness, and cost evaluators.
4. Add semantic validation and conversation coherence evaluators.
5. Enable response sampling and daily human review routing.
6. Bind runtime policy and evaluation gates to the new canonical metrics.

This sequence matches standard observability practice by establishing normalized event data before deriving higher-order quality and risk metrics.

14. Platform Mapping and Storage Tiering link

This specification defines canonical fields and events; it does not by itself say which physical platform stores them. Per Synedrix OS Tech Stack, storage MUST follow platform ownership rather than being duplicated across systems:

Events (§9) MUST be emitted regardless of which platform ultimately persists the derived value; the event bus is the single normalized ingestion path, and platform-specific write paths are downstream consumers, not sources.

14a. QuestDB trend persistence link

Supabase holds audit-grade relational metric rows (§8, §14). QuestDB holds derived rolling trend series for dashboards and SLOs. Dashboards MUST read trends from QuestDB (or materialized views over it), not recompute them ad hoc from Supabase.

Base table link

Rules:

1. ts is the designated timestamp.
2. metric_name uses the canonical names from §7 (for example cost_drift_pct, tool_success_rate, trust_trend_30d, retrieval_f1_at_k, memory_growth_rate_daily).
3. metric_family is one of latency, cost, tool, quality, trust, memory, guardrail.
4. Sparse dimensions that are not first-class columns MAY appear in dims as JSON text.
5. Every sample that can be attributed to a tenant MUST carry tenant_id.

Materialized views link

Name rolling aggregates mv_<metric_name>_<window> (for example mv_trust_trend_30d, mv_tool_success_rate_60m). Each view MUST be defined with SAMPLE BY (or time-based GROUP BY) over metrics_samples and MUST NOT be the system of record for a single invocation.

See also QuestDB.

15. Dashboards, SLOs, and Alerting link

Canonical metrics MUST be observable operationally, not only queryable after the fact. The following default thresholds apply for v0.1 unless a domain pack declares a stricter local policy per §11:

Each alert MUST carry the same correlation identifiers as the triggering event (§6) so the on-call responder can pivot directly from the alert into the underlying trace in Langfuse or the underlying record in Supabase without re-deriving context. Dashboards SHOULD be organized per metric family (§7.1–7.10) rather than per agent, so cross-agent regressions in a single metric family remain visible even when no single agent crosses its own threshold.