Observability¶

Hookaido provides structured logging, Prometheus metrics, and OpenTelemetry tracing, all configurable via the observability block.

Quick Start¶

observability {
  access_log {
    enabled on
    output stderr
    format json
  }

  runtime_log {
    level info
    output stderr
    format json
  }

  metrics {
    listen ":9900"
    prefix "/metrics"
  }

  tracing {
    enabled on
    collector "https://otel.example.com/v1/traces"
  }
}

Logging¶

Hookaido produces two log streams, both structured JSON:

Access Log¶

Per-request logs for ingress, Pull API, and Admin API.

Shorthand:

observability {
  access_log on    # enable to stderr with JSON format
}

Block form:

observability {
  access_log {
    enabled on
    output stderr       # stdout, stderr, or file
    path /var/log/hookaido/access.log   # required when output=file
    format json
  }
}

Runtime Log¶

Application-level structured logs (startup, reload, errors, queue events).

Shorthand:

observability {
  runtime_log info    # level as shorthand: debug, info, warn, error, off
}

Block form:

observability {
  runtime_log {
    level info         # debug, info, warn, error, off
    output stderr      # stdout, stderr, or file
    path /var/log/hookaido/runtime.log
    format json
  }
}

Pull consumer lifecycle: an SSE stream logs pull_sse_connected when it is established and pull_sse_disconnected when it ends, both at INFO, carrying consumer_id, route, consumer_group, endpoint, remote_addr and token_ref (the configured secret reference, never the token). The teardown line also carries status_code, messages_sent and duration_seconds. The access log cannot substitute for these: a stream logs one http_request line when it opens and then stays open for hours, so it records neither who is still attached nor when anyone left. See Pull API — Who Is Attached.

Log Sinks¶

Sink	Description
`stdout`	Standard output
`stderr`	Standard error (default)
`file`	File output (requires `path`)

The --log-level CLI flag overrides the runtime log level from config.

Metrics¶

Prometheus-compatible metrics endpoint.

observability {
  metrics {
    listen ":9900"           # default: 127.0.0.1:9900
    prefix "/metrics"        # default: /metrics
    enabled on               # explicitly enable/disable
  }
}

Set enabled off to disable the metrics listener while keeping config in place.

Available Metrics¶

Process metrics:

Metric	Type	Description
`hookaido_up`	gauge	Always `1` while the process is serving; absence or a scrape failure is the signal
`hookaido_start_time_seconds`	gauge	Process start time as a Unix timestamp — subtract from `time()` for uptime, or alert on a change to catch restarts

Queue metrics:

Metric	Type	Description
`hookaido_queue_depth`	gauge	Current items by state (queued, leased, dead)
`hookaido_queue_total`	gauge	Current total items across all queue states
`hookaido_queue_oldest_queued_age_seconds`	gauge	Age of the oldest queued item in seconds
`hookaido_queue_ready_lag_seconds`	gauge	Ready lag of the earliest runnable queued item in seconds
`hookaido_queue_route_depth{route,consumer_group,state}`	gauge	Current items by route, consumer group and state (queued, leased, dead)
`hookaido_queue_route_oldest_queued_age_seconds{route,consumer_group}`	gauge	Age of the oldest queued item on that queue, in seconds
`hookaido_queue_route_ready_lag_seconds{route,consumer_group}`	gauge	Ready lag of the earliest runnable queued item on that queue, in seconds

The four unlabeled families above are instance-global and stay that way: a route label on them would put labeled and unlabeled series in one family, and sum(hookaido_queue_depth{state="queued"}) would then double-count. The hookaido_queue_route_* families carry the per-queue breakdown instead, and sum by (state) (hookaido_queue_route_depth) reproduces the aggregate.

Every configured route gets a series, including one whose queue is empty, so == 0 is expressible and a series that disappeared means the route was reconfigured away rather than "all clear". A route that is no longer configured but still holds items keeps reporting them. Cardinality is bounded by the configured routes and their consumer groups; the label set matches the hookaido_pull_* families, and consumer_group is empty for a route without consumer groups. Deliver routes that fan out to several targets are one series per route: depth sums the targets, age and lag report the worst of them.

Alerting on a consumer that went away. A pull consumer can stop draining without disconnecting — it sits in a read on a half-open connection while the server's keepalive writes fail — and ingress keeps answering 202 for that route the whole time. On a multi-route instance the instance-global gauges cannot show it: a low-volume route's backlog is numerically invisible next to a busy route's normal working set, and hookaido_queue_oldest_queued_age_seconds says an item is old without saying which route it belongs to. Joined on the route, the two sides say it outright:

# Items are queued on a route with no consumer attached to it.
  hookaido_queue_route_depth{state="queued"} > 0
and on (route, consumer_group)
  hookaido_pull_sse_connection_active == 0

# A backlog that is ageing, wherever it is.
max by (route, consumer_group) (hookaido_queue_route_oldest_queued_age_seconds) > 600

The first rule covers SSE consumers, which hold a connection. A consumer that polls POST {endpoint}/dequeue holds none between calls and is not counted by hookaido_pull_sse_connection_active; alert on the age rule, or on rate(hookaido_pull_acked_total[15m]) == 0 beside a non-zero depth, for those.

Ingress metrics:

Metric	Type	Description
`hookaido_ingress_accepted_total`	counter	Ingress requests accepted and enqueued
`hookaido_ingress_rejected_total`	counter	Ingress requests rejected (auth, rate-limit, etc)
`hookaido_ingress_rejected_by_reason_total{route,reason,status}`	counter	Ingress rejects by route, normalized reason + status (includes `memory_pressure` with status `503`). `route` is empty when the reject happened before a route resolved
`hookaido_ingress_auth_rejected_total{route,reason}`	counter	Auth rejects by route and classified cause (`no_valid_secret`, `signature_mismatch`, `timestamp_out_of_window`, `replay`, `malformed`, `credentials`, `forward_denied`, `unspecified`, `other`)
`hookaido_ingress_enqueued_total`	counter	Items enqueued via ingress (>accepted if fanout)
`hookaido_ingress_adaptive_backpressure_total{reason}`	counter	Ingress requests rejected by adaptive backpressure (by trigger reason)
`hookaido_ingress_adaptive_backpressure_applied_total`	counter	Total ingress requests rejected by adaptive backpressure

Every reject carries the route it happened on, so a throttled or overloaded route can be named rather than inferred. Two cases have no route to name and report route="": a request to an unmatched path (404), and one whose path matches but whose method does not (405) — several routes can match one path with different methods, so there is no single route to attribute it to. Prometheus does not distinguish an empty label value from an absent one, so those series are the same series they were before the label existed.

route is a label on this family rather than a family of its own, unlike the hookaido_queue_route_* gauges: each reject is counted under exactly one (route, reason, status) triple, so no aggregate is duplicated and sum by (reason) (...) still reproduces the instance-wide count. Only reason and status are zero-filled — routes come and go with the config, and pre-seeding every route against every reason would emit series for combinations that cannot occur (memory_pressure on a non-memory backend, say).

hookaido_ingress_adaptive_backpressure_total{reason} stays route-less by design: its reason is the trigger (queue pressure, ready lag), which is an instance-wide condition. Which routes got shed is already in the reject family under reason="adaptive_backpressure".

Delivery metrics:

Metric	Type	Description
`hookaido_delivery_attempts_total`	counter	Total push delivery attempts
`hookaido_delivery_acked_total`	counter	Deliveries acknowledged (2xx)
`hookaido_delivery_retry_total`	counter	Deliveries scheduled for retry
`hookaido_delivery_dead_total`	counter	Deliveries moved to DLQ
`hookaido_delivery_dead_by_reason_total{reason}`	counter	DLQ transitions by normalized reason (`max_retries`, `no_retry`, `policy_denied`, `unspecified`, `other`)

Pull metrics:

Metric	Type	Description
`hookaido_pull_dequeue_total`	counter	Pull dequeue requests by `route` and status label (`200`, `204`, `4xx`, `5xx`)
`hookaido_pull_acked_total`	counter	Successful Pull ack operations by `route`
`hookaido_pull_nacked_total`	counter	Successful Pull nack/mark-dead operations by `route`
`hookaido_pull_ack_conflict_total`	counter	Pull ack lease conflicts (`409`) by `route`
`hookaido_pull_nack_conflict_total`	counter	Pull nack lease conflicts (`409`) by `route`
`hookaido_pull_lease_active`	gauge	Active Pull leases currently tracked by `route`
`hookaido_pull_lease_expired_total`	counter	Lease expirations observed during Pull `ack`/`nack`/`extend` by `route`

Pull SSE metrics (see SSE streaming):

Metric	Type	Description
`hookaido_pull_sse_connections_total`	counter	SSE connections established, by `route`
`hookaido_pull_sse_messages_sent_total`	counter	Messages sent over SSE, by `route`
`hookaido_pull_sse_connection_active`	gauge	Currently active SSE connections, by `route`

All hookaido_pull_* series above also carry a consumer_group label. It is empty for a route without consumer groups, and Prometheus treats an empty label value as absent — so existing series and dashboards are unchanged, while a route that fans out to groups keeps them separable. Without it the connection gauge would read 2 for a route with two groups and one consumer each, which is the expected state, and an unexpected third consumer would be invisible.

hookaido_pull_sse_connection_active is deliberately unlabeled by consumer: a remote-address label would be unbounded cardinality for a diagnostic that is only needed occasionally. When the gauge is higher than you expect, name the consumers with GET /pull/consumers or from the pull_sse_connected / pull_sse_disconnected runtime log lines.

Secret and publish-policy metrics:

Metric	Type	Description
`hookaido_runtime_secret_gc_pruned_total`	counter	Expired runtime-secret versions pruned by the background sweeper, by pool name
`hookaido_runtime_secret_pool_versions{pool,state}`	gauge	Versions held per pool by validity state at scrape time (`valid`, `pending`, `expired`)
`hookaido_runtime_secret_pool_next_expiry_seconds{pool}`	gauge	Seconds until the pool's next version lapses (`+Inf` if none does, `0` if no valid version is left)
`hookaido_runtime_secret_pool_exhaustion_seconds{pool}`	gauge	Seconds until the pool holds no valid version at all (`+Inf` if a valid version is unbounded, `0` if it already holds none)
`hookaido_publish_rejected_managed_target_mismatch_total`	counter	Admin publish rejections with code `managed_target_mismatch`
`hookaido_publish_rejected_managed_resolver_missing_total`	counter	Admin publish rejections with code `managed_resolver_missing`

The hookaido_runtime_secret_pool_* gauges cover every registered pool, static secret blocks included — a static pool whose one version has lapsed rejects requests exactly as an empty runtime pool does. They are absent (not zero) in a process that declares no secrets at all.

Alert on exhaustion_seconds, not on next_expiry_seconds: the latter drops to near zero on every handover of a healthy overlapping rotation, while the former moves only when the pool is genuinely about to run out of credentials.

Store common metrics:

Metric	Type	Description
`hookaido_store_operation_seconds{backend,operation}`	histogram	Store operation duration by backend and operation
`hookaido_store_operation_total{backend,operation}`	counter	Store operation totals by backend and operation
`hookaido_store_errors_total{backend,operation,kind}`	counter	Store operation errors by backend, operation, and normalized kind

The common families are emitted by all first-party queue backends (sqlite, memory, postgres).

Store/SQLite compatibility metrics (sqlite backend):

Metric	Type	Description
`hookaido_store_sqlite_write_seconds`	histogram	SQLite write transaction duration (queue mutation paths)
`hookaido_store_sqlite_dequeue_seconds`	histogram	SQLite dequeue transaction duration
`hookaido_store_sqlite_checkpoint_seconds`	histogram	SQLite WAL checkpoint duration (periodic passive checkpoints)
`hookaido_store_sqlite_busy_total`	counter	SQLite busy/locked errors observed in instrumented paths
`hookaido_store_sqlite_retry_total`	counter	SQLite begin-transaction retry attempts after busy/locked errors
`hookaido_store_sqlite_tx_commit_total`	counter	Committed SQLite transactions in instrumented queue paths
`hookaido_store_sqlite_tx_rollback_total`	counter	Rolled-back SQLite transactions in instrumented queue paths
`hookaido_store_sqlite_checkpoint_total`	counter	Successful periodic SQLite WAL checkpoints
`hookaido_store_sqlite_checkpoint_errors_total`	counter	Failed periodic SQLite WAL checkpoints

Store/Memory metrics (memory backend):

Metric	Type	Description
`hookaido_store_memory_items{state}`	gauge	Current in-memory item count by state (`queued`, `leased`, `delivered`, `dead`)
`hookaido_store_memory_retained_bytes{state}`	gauge	Estimated retained bytes by state (`queued`, `leased`, `delivered`, `dead`)
`hookaido_store_memory_retained_bytes_total`	gauge	Estimated total retained bytes in memory store
`hookaido_store_memory_evictions_total{reason}`	counter	Memory-store evictions by reason (`drop_oldest`, retention evictions, etc.)

Backend Metric Expectations¶

Use backend-agnostic metric families as the default dashboard and alert base:

hookaido_store_operation_seconds{backend,operation}
hookaido_store_operation_total{backend,operation}
hookaido_store_errors_total{backend,operation,kind}
hookaido_delivery_dead_by_reason_total{reason}

Backend-specific coverage:

Backend	Required/common store metrics	Backend-specific metrics	Notes
`sqlite`	`hookaido_store_operation_*`, `hookaido_store_errors_total`	`hookaido_store_sqlite_*`	`hookaido_store_sqlite_*` is compatibility/debug surface for SQLite internals.
`memory`	`hookaido_store_operation_*`, `hookaido_store_errors_total`	`hookaido_store_memory_*`	`hookaido_store_sqlite_*` is intentionally absent.
`postgres`	`hookaido_store_operation_*`, `hookaido_store_errors_total`	none (store internals exposed via common families)	`hookaido_store_sqlite_` and `hookaido_store_memory_` are intentionally absent.

Migration guidance:

Prefer common store metric families for SLOs, saturation alerts, and cross-backend dashboards.
Keep hookaido_store_sqlite_* for SQLite-only deep diagnostics (for example WAL/checkpoint lock analysis).
Treat missing backend-specific series on other backends as "not emitted", not as zero or failure.

PromQL Examples (Backend-Aware)¶

Store p95 by backend and operation:

histogram_quantile(
  0.95,
  sum by (backend, operation, le) (
    rate(hookaido_store_operation_seconds_bucket[5m])
  )
)

Store error rate by backend, operation, and kind:

sum by (backend, operation, kind) (
  rate(hookaido_store_errors_total[5m])
)

Backend-specific store throughput comparison:

sum by (backend, operation) (
  rate(hookaido_store_operation_total[5m])
)

DLQ growth by dead reason:

sum by (reason) (
  increase(hookaido_delivery_dead_by_reason_total[15m])
)

A secret pool with nothing valid in it — every route naming it answers 401, while the process stays green on every other signal:

hookaido_runtime_secret_pool_versions{state="valid"} == 0

Warn before the cliff rather than after it (six hours of validity left):

hookaido_runtime_secret_pool_exhaustion_seconds < 21600

Auth rejects that are Hookaido's fault rather than the sender's, by route:

sum by (route) (
  increase(hookaido_ingress_auth_rejected_total{reason="no_valid_secret"}[15m])
) > 0

Which routes are being throttled or shed, ranked:

topk(5,
  sum by (route, reason) (
    rate(hookaido_ingress_rejected_by_reason_total{reason=~"rate_limit|queue_full|adaptive_backpressure|memory_pressure"}[5m])
  )
)

One route rejecting while the instance looks fine — the multi-route blind spot, for rejects rather than for backlog:

sum by (route) (rate(hookaido_ingress_rejected_by_reason_total[5m])) > 0

Senders pointed at a path that no longer exists (route="" is the unattributed bucket, so this is the honest way to ask):

increase(hookaido_ingress_rejected_by_reason_total{route="",reason="not_found"}[1h]) > 0

Alert example (backend-aware store error burst):

sum by (backend) (
  rate(hookaido_store_errors_total[5m])
) > 0

Publish metrics:

Metric	Type	Description
`hookaido_publish_accepted_total`	counter	Accepted publish mutations
`hookaido_publish_rejected_total`	counter	Rejected publish mutations
`hookaido_publish_rejected_validation_total`	counter	Rejections: validation errors
`hookaido_publish_rejected_policy_total`	counter	Rejections: policy violations
`hookaido_publish_rejected_conflict_total`	counter	Rejections: duplicate IDs
`hookaido_publish_rejected_queue_full_total`	counter	Rejections: queue at capacity
`hookaido_publish_rejected_store_total`	counter	Rejections: store errors
`hookaido_publish_scoped_accepted_total`	counter	Accepted scoped (managed) publish
`hookaido_publish_scoped_rejected_total`	counter	Rejected scoped (managed) publish

Tracing diagnostics:

Metric	Type	Description
`hookaido_tracing_enabled`	gauge	Whether tracing is configured
`hookaido_tracing_init_failures_total`	counter	Tracing initialization failures
`hookaido_tracing_export_errors_total`	counter	Tracing export errors

Compatibility/version metrics:

Metric	Type	Description
`hookaido_build_info{version=...}`	gauge	Process version label for dashboard/version gating
`hookaido_metrics_schema_info{schema=...}`	gauge	Metrics schema version label for compatibility guards

Tracing¶

OpenTelemetry OTLP/HTTP traces for request-level observability. HTTP servers (ingress, Pull API, Admin API) and the outbound push dispatcher client are instrumented.

Minimal Config¶

observability {
  tracing {
    enabled on
    collector "https://otel.example.com/v1/traces"
  }
}

Full Config¶

observability {
  tracing {
    enabled on
    collector "https://otel.example.com/v1/traces"
    url_path "/v1/traces"
    timeout "10s"
    compression gzip           # none or gzip
    insecure off               # allow plain HTTP (dev only)

    # TLS options
    tls {
      ca_file /path/to/ca.pem
      cert_file /path/to/cert.pem
      key_file /path/to/key.pem
      server_name "otel.example.com"
      insecure_skip_verify off
    }

    # Proxy
    proxy_url "http://proxy.internal:3128"

    # Retry on export failure
    retry {
      enabled on
      initial_interval "5s"
      max_interval "30s"
      max_elapsed_time "1m"
    }

    # Custom headers (e.g., for auth)
    header "Authorization" "Bearer otel-token"
    header "X-Custom-Header" "value"
  }
}

Directive	Default	Description
`enabled`	`off`	Enable/disable tracing
`collector`	—	OTLP/HTTP collector endpoint
`url_path`	`/v1/traces`	URL path on the collector
`timeout`	`10s`	Export timeout
`compression`	`none`	`none` or `gzip`
`insecure`	`off`	Allow HTTP (non-TLS) transport
`proxy_url`	—	HTTP proxy for exporter
`tls.ca_file`	—	CA certificate file for TLS
`tls.cert_file`	—	Client certificate file for mTLS
`tls.key_file`	—	Client key file for mTLS
`tls.server_name`	—	Override TLS server name
`tls.insecure_skip_verify`	`off`	Skip TLS certificate verification
`retry.enabled`	`on`	Retry failed exports
`retry.initial_interval`	`5s`	First retry delay
`retry.max_interval`	`30s`	Maximum retry delay
`retry.max_elapsed_time`	`1m`	Total retry time budget
`header`	—	Custom HTTP headers (repeatable)

Header entries must be valid HTTP header name/value pairs. Invalid entries fail config validation.

Health Diagnostics¶

The Admin API health endpoint (GET /healthz?details=1) aggregates observability data:

Queue state rollups with age/lag indicators
Backlog trend signals with operator action playbooks
Tracing counters (init failures, export errors)
Ingress adaptive-backpressure diagnostics (adaptive_backpressure_applied_total, adaptive_backpressure_by_reason) and rejection reason counters (rejected_by_reason, including memory_pressure)
Ingress auth-rejection diagnostics (auth_rejected_total, auth_rejected_by_reason) — the classified causes behind the auth bucket, folded across routes
Runtime-secret rollup (runtime_secrets): pools_without_valid_version plus its _names list, and one entry per pool with version counts by validity state and both expiry deadlines (absolute and as a countdown; null means no deadline applies). This is the piece a plain HTTP uptime checker can consume without a Prometheus scrape — a non-zero pools_without_valid_version means at least one route is rejecting every webhook it receives.
Delivery diagnostics include dead-letter reason breakdown (dead_by_reason) for DLQ growth attribution
Memory-store diagnostics (when backend is memory): items_by_state, retained bytes, eviction counters, and memory_pressure status/limits/reject counters
Top route/target backlog buckets
Queue diagnostics are cached (short TTL) and served stale-while-refresh under heavy load to keep control-plane endpoints responsive.

Operational guidance for control-plane responsiveness:

Keep SLO probes on GET /healthz (without details) for the fastest liveness path.
Use GET /healthz?details=1 and GET /metrics for diagnostics/monitoring; under queue saturation these endpoints prioritize bounded latency over strictly real-time queue snapshots.

Saturation Notes¶

Queue saturation analysis showed one hot path in the ingest/admission write flow: with queue_limits.max_depth enabled, each enqueue previously executed COUNT(*) over active queue states (queued + leased) inside a write transaction.

At high occupancy, that repeated count increased write transaction time and lock contention (hookaido_store_sqlite_write_seconds, hookaido_store_sqlite_busy_total, hookaido_store_sqlite_retry_total).

Hookaido now maintains O(1) active-depth counters (queue_counters) via SQLite triggers and uses them for max_depth admission checks.

For memory backend deployments, Hookaido also applies a retained-footprint pressure guard and emits memory_pressure ingress reject reasons before hard process failure risk.

To validate improvements in your environment, compare before/after load runs using: - p95/p99 ingress latency and 503 rate - hookaido_store_sqlite_write_seconds histogram shape - hookaido_store_sqlite_busy_total and hookaido_store_sqlite_retry_total growth rate

See Admin API for details.

Adaptive Backpressure Tuning¶

Use the dedicated production runbook in Adaptive Backpressure Tuning.

Key principle: - defaults.adaptive_backpressure should react before hard queue_limits.max_depth pressure, not after.

Use these series together: - hookaido_ingress_adaptive_backpressure_total{reason} - hookaido_ingress_rejected_by_reason_total{route,reason,status} - ingress latency p95/p99 from HTTP telemetry

hookaido_ingress_rejected_by_reason_total{reason="auth"} counts every auth reject as one bucket, which is the right shape for a reject census and the wrong one for diagnosis: an empty secret pool, a sender signing with the wrong key and a clock-skewed timestamp are three unrelated problems with three different owners. hookaido_ingress_auth_rejected_total{route,reason} separates them and names the route. The two reconcile — sum(hookaido_ingress_auth_rejected_total) equals the auth bucket — so existing rules on the coarse family keep their meaning.

Dashboard Compatibility Notes¶

When dashboards span mixed Hookaido versions (for example v1.2.x and v1.3.x), treat missing metrics as "not emitted" rather than zero:

Gate rules and panels by hookaido_metrics_schema_info{schema="1.7.0"} == 1 (or hookaido_build_info version labels).
Since 1.7.0, hookaido_ingress_rejected_by_reason_total carries a route label. Matchers are unaffected ({reason="auth"} still selects every route) and sum by (reason) (...) is unchanged, but a selector that previously returned exactly one series per reason/status now returns one per route as well. Panels that plotted it raw gain a line per route; aggregate them with sum by (reason) to restore the old shape.
In PromQL, prefer compatibility-safe expressions (for example metric OR on() vector(0)) where appropriate.
Document minimum supported Hookaido version per dashboard bundle to avoid false "all good" signals from absent series.

Audit Logging¶

All Admin API and MCP mutations emit structured JSONL audit events (to stderr or configured runtime log):

{
  "timestamp": "2026-02-09T10:00:00Z",
  "principal": "ops@example.test",
  "role": "operate",
  "tool": "messages_publish",
  "input_hash": "sha256:abc...",
  "result": "ok",
  "duration_ms": 42,
  "metadata": { ... }
}

Audit metadata varies by operation:

Config mutations: config_mutation (operation, mode, outcome)
Runtime control: runtime_control (operation, outcome)
ID-based mutations: id_mutation (operation, IDs requested/unique/changed)
Filter mutations: filter_mutation (operation, matched/changed, preview flag)
Publish: admin_proxy_publish (rollback counters, if Admin-proxy mode)

← Documentation Index