Status

Draft canonical spec. Companion to Synedrix OS - Agent Spec. Supersedes: synedrix-os-agent-evaluation-spec-v0.1.md. Related artifacts: Synedrix OS - Agent Spec, Synedrix OS - Agent Runtime Policy Spec, Synedrix OS - Platform Architecture & Tech Stack, Synedrix OS - Workflow Contract Spec, Synedrix OS Observability Metrics Spec.

Purpose

This spec defines how a Synedrix OS agent is tested, scored, and approved across three quality dimensions: output quality, policy compliance, and operational health.

Synedrix OS separates LLM engineering observability (Langfuse), product and behavioural analytics (PostHog), and operational monitoring (Grafana/Prometheus) into distinct layers. The evaluation framework maps directly onto those layers rather than collapsing all quality signals into one score.

Evaluation is a first-class operating concern in Synedrix OS. Agents without declared evaluation suites must not graduate from draft to active status in the registry.

Three evaluation dimensions

1. Output quality (Langfuse)

Answers: Is the agent producing correct, coherent, and on-policy outputs?

Langfuse owns prompt versioning, LLM trace capture, token and cost attribution, evaluations, and quality scoring.

Types of output quality checks:

  • Factual correctness — does the output match ground truth for known inputs?
  • Schema compliance — does the output match the declared output_schema in the agent contract?
  • Confidence calibration — is stated confidence consistent with actual accuracy?
  • Reasoning coherence — does the reasoning support the conclusion?
  • Hallucination detection — are claims grounded in the provided context?
  • Policy instruction adherence — does the output follow the safety and governance overlays in the prompt policy?

2. Policy compliance (orchestration + audit layer)

Answers: Is the agent operating within its authority boundary and governance constraints?

Types of policy compliance checks:

  • Authority boundary checks — did the agent attempt actions outside its declared authority?
  • Approval gate compliance — did it correctly flag approval-required actions rather than proceeding autonomously?
  • Side-effect compliance — did it produce only declared side effects?
  • Escalation compliance — did it escalate correctly when conditions required it?
  • Tool permission compliance — did it invoke only tools within its declared tool_permissions?
  • Scope compliance — did it stay within its declared memory access scope?

3. Operational health (Grafana/Prometheus + Langfuse)

Answers: Is the agent performing reliably within cost and latency budgets?

  • Latency SLO compliance — is p50/p95/p99 latency within latency_slo_ms?
  • Token budget compliance — is average usage within declared budgets?
  • Error rate — proportion of invocations ending in failure or escalation.
  • Retry rate — high rates indicate model or tool instability.
  • Circuit breaker open rate — how often does the circuit breaker open?
  • Cost per invocation — within declared budget?
  • Fallback rate — how often does the agent degrade to fallback model or rules-only mode?

Observability metric families as evaluation inputs

The following metric families are defined in Synedrix OS Observability Metrics Spec and are available as inputs to evaluation suites and promotion gates. Each family is classified as blocking(suite failure prevents promotion) oradvisory (informs review but does not block promotion alone). The defaults below are OS-floor values; domain packs and business instances may tighten but not relax them.

Metric familySpec sectionScopeDefault gateBlocking?
Retrieval quality (retrieval_f1_at_k, retrieval_quality_score)§7.1InvocationRequired for retrieval-backed agentsAdvisory
Groundedness / hallucination (hallucination_probability, hallucination_flag, faithfulness_score)§7.2InvocationBlocking for interface and high-risk agentsBlocking
Cost and drift (cost_drift_flag, cost_drift_pct, total_cost_usd)§7.3InvocationAdvisory; triggers model routing reviewAdvisory
Semantic validity (semantic_validity_score, semantic_validity_flag)§7.4InvocationBlocking when structured output feeds downstream automationBlocking
Multi-turn coherence (coherence_score, goal_alignment_score)§7.6ConversationBlocking for interface agents onlyBlocking (interface agents)
Tool reliability (tool_success_rate, p95_latency_ms)§7.7Tool aggregateAdvisory; feeds anomaly detectionAdvisory
Risk alignment (risk_alignment_score, risk_policy_violations_count)§7.8InvocationBlocking for medium, high, and critical risk tiersBlocking
Guardrail effectiveness (guardrail_true_positive_rate, guardrail_false_positive_rate)§7.9Guardrail aggregateRequired before guardrail moves from monitor to enforce modeAdvisory
User trust and satisfaction (user_trust_score, trust_score_30d_avg, trust_trend_30d)§7.10Conversation / agentPromotion gate for interface agents; MUST NOT be used as per-call runtime vetoAdvisory

Promotion gate examples using observability metrics

High-risk or critical agent (risk_tier: high or critical) 

eval_suite:
  suite_id: eval-risk-alignment-blocking
  agent_id: <agent_id>
  dimensions: [policy-compliance, observability-metrics]
  blocking: true
  pass_thresholds:
    risk_alignment_score_min: 0.95
    risk_policy_violations_count_max: 0
    hallucination_probability_max: 0.15
    semantic_validity_flag_allowed: false
Illustrative example, not a production implementation. Review and test before use.

Interface agent (user-facing) 

eval_suite:
  suite_id: eval-interface-trust-gate
  agent_id: <agent_id>
  dimensions: [output-quality, observability-metrics]
  blocking: true          # blocks promotion; not a per-call runtime veto
  pass_thresholds:
    trust_score_30d_avg_min: 0.70
    coherence_score_min: 0.75
    hallucination_probability_max: 0.20
Illustrative example, not a production implementation. Review and test before use.

Evaluation suite schema 

eval_suite:
  suite_id:                    # e.g. eval-policy-guard-regression
  agent_id:
  description:
  dimensions:                  # output-quality | policy-compliance | operational-health | observability-metrics
  run_trigger:                 # on-deploy | scheduled | on-prompt-change | on-demand | on-model-change
  langfuse_dataset_ref:
  pass_thresholds:
    output_quality_min:        # 0.0–1.0
    policy_compliance_min:     # 0.0–1.0
    latency_p95_max_ms:
    error_rate_max:
    token_budget_compliance:
  blocking:                    # true = deployment blocked if suite fails
  notification_targets:
  regression_sensitivity:      # strict | standard | permissive
Illustrative example, not a production implementation. Review and test before use.

Evaluation triggers

TriggerDescriptionRecommended for
on-deployRun full suite before agent goes activeAll agents
on-prompt-changeRun output quality suite on any prompt layer version changeAll LLM-backed agents
scheduledPeriodic regression against live or shadow trafficProduction agents
on-demandManual trigger for debugging or incident reviewAll agents
on-model-changeTriggered when primary or fallback model version changesAll agents using managed models

Promotion gates

TransitionRequired gate
draft → activeAll blocking: true suites must pass
Prompt version bumpOutput quality suite must pass
Model version changeFull suite must pass
deprecated → retiredReplacement agent must be active with passing suites
suspended → activeFull suite plus incident review must pass

Promotion gates are enforced by the registry and must not be bypassable by domain packs or business instances.

Evaluation guide: interpreting scores and results

The sections above define the schema. This section is for whoever is looking at a suite’s actual output and needs to know what a number means, whether a “pass” can be trusted, and what to do about a failure.

How each score is produced

A pass_thresholds entry is not one uniform kind of measurement — three different production methods feed this framework, and they carry different weight:

  • Deterministic checks (exact-match, schema-validation, substring/regex-style assertions). No model judgment involved — the cheapest, most objective signal available, and the right tool for locking in a regression against a specific, already-diagnosed failure (a known-bad answer shape, a stale fact reappearing). A deterministic check either passed or it didn’t; there is no calibration question to ask about it.
  • LLM-as-judge checks (coherence_score, hallucination_probability, and similar 0.0–1.0 quality dimensions). A second model scores the agent’s output against the grounding it was given. The judge must never be the same model or checkpoint being evaluated — a model does not grade its own work, and doing so silently inflates every score it produces. Judge scores are a proxy for quality, not quality itself: treat a single run’s judge score as informative, not authoritative, until the judge itself has been spot-checked against a few outputs a human actually read. A judge at temperature: 0 is not guaranteed to score the same answer the same way twice, and this is not theoretical: observed directly on os-agt-009’s live on-the-fly scoring — two runs of a near-identical multi-turn conversation, both correctly declining an unretrieved fact with the same shape of answer, both with the same 10 retrieved sources passed to the judge, scored coherence_score: 0.80 / hallucination_probability: 0.20 on one run and coherence_score: 0.00 / hallucination_probability: 1.00 on the other. Quantized, batched inference backends can be genuinely non-deterministic even at temperature: 0 (floating-point batching effects), and the answer being judged is rarely temperature: 0 itself, so small wording differences between runs can tip a borderline judgment either way. Do not read one low judge score in isolation as proof of a regression — check whether it replicates before treating it as a finding, and treat a suite’s mean across several items as more trustworthy than any single item’s score. Closing this gap for real requires a judge-calibration pass (a small set of answers with known-correct human labels, checked against how consistently the judge agrees with them) — real, scoped work, not a side effect of building the suite itself.
  • Production ground truth (trust_score_30d_avg and similar metrics sourced from real end-user signals — e.g. thumbs up/down feedback captured as a score on the originating trace). This is the only one of the three that is not a proxy: it is real users reacting to real answers. It is also the slowest to accumulate and the most sensitive to sample size — see below.

Reading a score

  • coherence_score, trust_score_30d_avg, and similar dimensions: higher is better (0.0–1.0).
  • hallucination_probability and similar risk dimensions: lower is better (0.0–1.0) — this is a probability of a problem, not a quality score, and pass thresholds for it are expressed as a _max, not a _min. Do not read a high hallucination_probability as good the way a high coherence_score is good.
  • A _30d_avg-suffixed metric is a rolling window, not a single measurement. A “PASS” computed from a handful of events (early in an agent’s life, or right after a quiet period) is provisional, not durable — report the sample size (n) alongside the average itself, and re-check once real traffic has actually accumulated over the full window. A pass on n = 5 and a pass on n = 500 are not the same claim.

Reading the gate

blocking: true thresholds combine with AND, not an average — every threshold in the suite must be met simultaneously for the lifecycle transition to proceed (see “Promotion gates” above). A suite that is strong on coherence_score but fails hallucination_probability_max has not “mostly passed”; it has failed the gate.

A suite may reasonably run with blocking: false at first, while it is being stood up and its own thresholds are being sanity-checked against real output — but per this spec’s invariants, no agent may remain at active status indefinitely without at least one blocking: true suite actually enforced.

Routing a failure to the right fix

When a suite fails, the fix is not always where it looks like it is. Before spending effort, classify the failure by root cause and let that classification choose the fix — cheapest and most reversible first:

Root causeWhat it usually meansWhere to fix it
RetrievalThe agent never had the right context to answer from — a chunking, ranking, or pinning problem, not a generation problem.Fix retrieval/data: chunking, CANONICAL_FACTS-style pinning, re-ranking. Cheapest and most common historically — both real production incidents on record for os-agt-009 before ADR-OS-003 were retrieval problems, not model problems.
Model-behaviorThe agent had the right context but the generation model mishandled it in a known, reproducible way (formatting drift, repetition, leaked internal tokens, unreliable tool-call judgment).Fix at the decoding/serving layer first: structured/guided decoding, tool-choice forcing, sampling parameters. Try this before ever considering a weight change.
ReasoningThe agent had the right context and produced well-formed output, but the actual reasoning connecting context to answer was wrong — a genuine capability gap.The one category where a stronger/different model, or fine-tuning the agent’s own already-authored behavior, is actually the correct lever — see ADR-OS-003 §4’s decision-boundary rule on what fine-tuning may and may not be used for.
HallucinationUsually a retrieval problem wearing a generation-layer costume: the context needed to answer correctly was missing, thin, or contradictory, and the model filled the gap with plausible-sounding invention.Check retrieval first, same as the Retrieval row, before concluding the model itself is untrustworthy.
NoneThe item passed.No action.

Do not reach for the most expensive, hardest-to-reverse fix (fine-tuning, a model swap) before confirming — with evidence, not assumption — that the failure isn’t actually a retrieval or decoding-layer problem in disguise. This is the same evidence-gated ordering ADR-OS-003 adopts for os-agt-009, generalized here as the intended reading of any suite built against this spec.

Reference implementation

os-agt-009 (General Chat Agent)’s Stage 0 suite, built per ADR-OS-003, is the first concrete instance of this schema in the platform: a suite definition, a golden set of items tagged by the same root-cause categories as the table above, an independent judge, and a runner that also pulls trust_score_30d_avg from real production feedback rather than estimating it. Use it as the reference pattern when standing up a suite for a new agent. Run results are recorded as scores on each answer’s own trace; this repository is documentation-only and does not itself hold point-in-time run data.

Policy compliance scoring

Policy compliance is binary per check. Aggregate compliance rate:

compliance_rate = compliant_invocations / total_invocations
Illustrative example, not a production implementation. Review and test before use.

A compliance rate below policy_compliance_min must:

  1. Trigger an immediate alert to declared notification_targets.
  2. Emit an agent.evaluation.compliance-breach event to the audit layer.
  3. Flag the agent for review before next deployment.

High-risk agents (risk_tier: high or critical) must maintain a compliance rate of 1.0. Any breach is treated as a blocking incident.

Evaluation vs observability

ConcernLayerAnswers
Output quality evaluationLangfuseIs the model output correct and on-policy?
Policy compliance evaluationAudit + orchestrationIs the agent within its authority?
Operational healthGrafana/Prometheus + LangfuseIs the agent reliable and cost-effective?
Behavioural analyticsPostHogHow is the agent being used? What workflows invoke it most?

Langfuse answers model-level questions. PostHog answers product and workflow-level questions. Grafana/Prometheus answers infrastructure and SLO questions.

OS vs domain pack vs business instance split

ConcernSynedrix OSDomain PackBusiness Instance
Evaluation framework and suite schemaOwns.Declares domain-specific suites and datasets.Sets instance-level targets and notification targets.
Promotion gate enforcementOwns — not bypassable.Contributes required suites per agent.Cannot waive blocking suites.
Pass thresholdsDefines minimum floor values.May set stricter thresholds per agent class.May tighten; cannot lower below floor.
Compliance scoringOwns compliance rate calculation and alerting model.Declares applicable compliance checks per agent.[Sets alert routing and incident response owners.
agent_eval_suites              — suite definitions per agent
agent_eval_runs                — historical record of every suite execution
agent_eval_results             — per-check results within a run
agent_eval_promotion_gates     — gate outcomes per lifecycle transition
agent_eval_compliance_events   — immutable audit log of compliance breaches
Illustrative example, not a production implementation. Review and test before use.

Invariants

  • Every agent must have at least one blocking: true eval suite before transitioning to active.
  • Policy compliance checks for risk_tier: high or critical agents must be blocking and require 1.0 compliance.
  • Promotion gate enforcement is owned by the OS registry and cannot be bypassed.
  • Every compliance breach must emit an auditable event.
  • Evaluation suites must be versioned with immutable pass/fail history.

Versioning

Agent Specifications

Database and Tech Stack

Governance

Observability Metrics

  • Synedrix OS Observability Metrics Spec — Canonical metric definitions; defines the risk_alignment_score, semantic_validity_score, hallucination_flag, and user_trust_score families that promotion gates in this spec consume as blocking or advisory signals.

Synedrix OS · Agent Evaluation Spec · August 2026