ArcadeDB Cloud Observability: OpenTelemetry Tracing, Structured Logging, and Kubernetes Health Probes

Running a database in Kubernetes means more than exposing a port and hoping for the best. Operators need to know: is the process alive? Is it ready to take traffic? Where did this request slow down, and why? ArcadeDB’s Cloud Observability Architecture closes that gap with four independently shippable pillars: Kubernetes health probes, deeper metrics with OTLP export, structured logging with correlation IDs, and distributed tracing via OpenTelemetry.

All of it is opt-in. If you don’t touch a single configuration key, your server behaves exactly as it does today.

The Gap We’re Closing

ArcadeDB already has a solid single-node observability story: Micrometer metrics with an in-memory registry, an optional /prometheus endpoint, JVM and executor-pool binders, a rich engine Profiler (cache hit ratio, pages, WAL, transaction counts, MVCC conflicts), a manual query profiler, and an /api/v1/ready endpoint.

What was missing for cloud-native, Kubernetes-first deployments:

1. Metrics depth - query, command, and transaction latency lived only in the manual profiler, with no always-on RED (Rate, Errors, Duration) timers or percentile histograms.
2. Distributed tracing - no spans, no trace-context propagation, no way to follow a request across the query engine, a transaction, and a Raft replication hop.
3. Structured logging and correlation - logs were text-only, with no JSON output and no way to tie a log line back to the trace or request that produced it.
4. Kubernetes health probes - only a readiness endpoint, no standard liveness check.

One Instrumentation Pass, Two Signals

The design leans on a property of Micrometer’s Observation API: instrument a code path once, and it can emit both a metric (a timer) and a span, depending on what’s registered. ArcadeDB already ships Micrometer, so the core server now wraps its hot paths (HTTP request handling, query and command execution, transaction commit, Raft replication) in Observation calls.

With no tracer registered, an Observation is just a metrics-only timer, exactly what happens today. Install the optional tracing plugin, and the same instrumentation points start emitting spans too, with zero additional code paths to maintain.

Pillar 1: Metrics Depth

New always-on RED timers, with percentile histograms and SLO buckets, surfaced in both /prometheus and, optionally, OTLP:

• arcadedb.http.requests - tagged by method, path template (never the raw URI, to avoid cardinality blowup), status, and database.
• arcadedb.query.duration - tagged by language, database, and query type.
• arcadedb.tx.duration - tagged by database and outcome (commit or rollback).

A new EngineMetricsBinder, modeled on the existing PoolMetrics, bridges the engine Profiler’s cache hit ratio, page and WAL statistics, and MVCC conflict counts, plus database-level and sparse-vector numbers, into Micrometer gauges tagged by database. Set arcadedb.serverMetrics.otlp.enabled=true and these same series also flow to an OTLP collector, no changes to the existing Prometheus scrape path.

Pillar 2: Distributed Tracing

Tracing lives in a brand-new, entirely optional tracing module, mirroring how the existing metrics module is packaged: a separate Maven module, provided scope on the server, loaded via the standard ServerPlugin SPI. The OpenTelemetry SDK, micrometer-tracing-bridge-otel, and the OTLP exporter live only inside this module. The core server’s compile classpath never sees them.

Set arcadedb.serverMetrics.tracing.enabled=true and the plugin registers a bridged tracer into Micrometer’s global ObservationRegistry. The same Observations from Pillar 1 start producing spans, nested correctly from HTTP down through query execution and transaction commit. Inbound HTTP requests continue an upstream trace via the standard W3C traceparent header, and outbound Raft RPCs propagate context so a write traces from leader to follower.

Leave the plugin jar off the classpath, or the flag unset, and it’s a true no-op: zero span overhead, nothing changes.

Pillar 3: Structured Logging and Correlation IDs

Metrics tell you something is slow. Traces tell you where. Logs tell you why. Pillar 3 ties the three together with a correlation ID that flows through all of them.

At the start of each request, ArcadeDB now populates a diagnostic context with the active trace and span IDs (when tracing is on) and a generated request ID (so correlation works even with tracing off), scoped per request and cleared in a finally block to avoid leaking across the server’s worker thread pool.

An opt-in JsonLogFormatter (arcadedb.server.logFormat=json) emits one JSON object per line: timestamp, level, logger, thread, message, trace ID, span ID, database, request ID, and exception. It’s built on ArcadeDB’s existing JSONObject, so there’s no new JSON dependency. The default text formatter is untouched byte-for-byte; if you want trace correlation without switching to JSON, arcadedb.server.logIncludeTrace=true appends a [traceId=...] tag to text-mode log lines instead.

Pillar 4: Kubernetes Health Probes

The smallest, highest-value pillar, and the one that ships first: a new GET /api/v1/health liveness endpoint. It’s cheap by design, no database I/O, no auth required, returning 200 whenever the process and HTTP layer are up. That matters for Kubernetes specifically: liveness must never depend on database readiness, or an orchestrator will kill a node that’s simply still warming up.

The existing /api/v1/ready endpoint is unchanged by default. It gains one optional behavior: set arcadedb.server.readinessRequiresHA=true and, when HA is active, readiness reflects whether the node has joined the Raft group and caught up, giving you a proper readinessProbe for clustered deployments.

livenessProbe:
  httpGet:
    path: /api/v1/health
    port: 2480
  initialDelaySeconds: 10
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /api/v1/ready
    port: 2480
  initialDelaySeconds: 5
  periodSeconds: 5

Configuration Reference

Every new key defaults to off or to today’s behavior:

Backward Compatibility by Design

Every pillar was built under one non-negotiable constraint: with no new configuration, an upgraded deployment behaves byte-for-byte identically to today. Existing endpoints and response shapes (/prometheus, /api/v1/server, /api/v1/ready) are never modified, only added to. No new mandatory dependencies land on the core server; the OpenTelemetry SDK and its heavier dependencies live exclusively in the optional, SPI-loaded tracing module. No existing configuration key is renamed.

That means you can upgrade today and adopt observability at your own pace, one pillar, one flag, at a time.

Pairing It with Grafana

If you’re already using the ArcadeDB Grafana plugin for BI dashboards over SQL, Cypher, or Gremlin, the new OTLP metrics and traces slot into the same observability stack: point an OpenTelemetry Collector at ArcadeDB, fan out to Prometheus and Tempo (or your vendor of choice), and build Grafana dashboards that go from a metric spike to a trace to the exact log line, without leaving your Kubernetes cluster.