BenoStreamDB Monitoring & Operations

BenoStreamDB is designed to be highly observable and easy to operate in serverless, ephemeral environments as well as traditional long-running daemon setups.

Observability by deployment mode

The engine records metrics in one place (the core registry) and every host exposes them, so a scrape sees the engine’s ingest/query/compaction/index metrics regardless of how the engine is deployed.

Deployment mode	Metrics endpoint	Health
Embedded library (Python / Rust)	http://<BSDB_METRICS_BIND>:<BSDB_METRICS_PORT>/metrics (default 127.0.0.1:9090), enabled by the observability feature	/health, /readyz
Flight SQL server (benostreamdb-flight)	http://<BSDB_METRICS_BIND>:<BSDB_METRICS_PORT>/metrics (default 127.0.0.1:9090)	/health, /readyz
Search gateway (bsdb-search)	http://<BSDB_SEARCH_BIND>:<BSDB_SEARCH_PORT>/metrics (default 127.0.0.1:9200)	/_health, /healthz, /livez, /readyz
Gateway / Iceberg REST bins	GET /metrics (gateway), GET /v1/metrics (iceberg_rest)	GET /health, GET /v1/health
Spark / Trino connectors	BenoStreamJNIBridge.gatherMetrics() / BenoStreamDBJNIBridge.gatherMetrics() — bridge into the host’s metrics system	host-provided

The search gateway’s /metrics serves both its own bsdb_search_* registry and the core engine’s bsdb_* metrics, so one scrape covers the whole process.

Metrics configuration

Variable	Description	Default
BSDB_METRICS_BIND	Bind address for the embedded/Flight metrics HTTP listener.	127.0.0.1
BSDB_METRICS_PORT	Port for the embedded/Flight metrics HTTP listener.	9090

The listener defaults to localhost. Set BSDB_METRICS_BIND=0.0.0.0 only when a scraper runs on another host, and front it with auth/TLS.

⚠️ Security: none of these endpoints authenticate. Run every BenoStreamDB server on a trusted internal network, behind a gateway/reverse proxy that terminates TLS and enforces authentication. Do not expose them to the public internet. See SECURITY.md.

Metric catalog

Engine metrics (bsdb_*)

Recorded with the prometheus crate in src/telemetry/metrics.rs:

Metric	Type	Labels	Meaning
bsdb_ingest_rows_total	counter	—	Rows ingested.
bsdb_query_latency_seconds	histogram	—	Query latency.
bsdb_search_latency_seconds	histogram	—	Vector/keyword search latency.
bsdb_commit_duration_seconds	histogram	—	Manifest commit duration.
bsdb_compaction_duration_seconds	histogram	—	Compaction duration.
bsdb_index_build_duration_seconds	histogram	—	HNSW-IVF build duration.
bsdb_active_files	gauge	—	Active Parquet files.
bsdb_active_segments	gauge	—	Active segments.
bsdb_cache_hits_total / _misses_total	counter	cache_name	Cache hit/miss by cache.
bsdb_io_bytes_read_total / _written_total	counter	—	Object-store I/O bytes.
bsdb_manifest_conflicts_total	counter	—	OCC commit conflicts.
bsdb_ingest_rss_bytes	gauge	—	Process RSS during ingest.
bsdb_ingest_backpressure_pauses_total	counter	—	Ingest pauses on the RAM high-water mark.
bsdb_ingest_backpressure_pause_seconds	histogram	—	Duration of each pause.
bsdb_index_build_gate_wait_seconds	histogram	—	Wait for an index-build permit.
bsdb_free_disk_bytes	gauge	—	Free disk for local tables.
bsdb_merged_deletes_phase_seconds	histogram	phase	Merge-on-read delete phases.
bsdb_merged_deletes_cache_total	counter	result	Merged-deletes cache hit/miss.
bsdb_delete_file_cache_total	counter	result	Parsed-delete cache hit/miss.
bsdb_merged_deletes_files_total	counter	kind	Delete files merged by kind.
bsdb_merged_deletes_calls_total	counter	—	load_merged_deletes calls.
bsdb_read_phase_seconds	histogram	phase	Parquet read phases.
bsdb_parquet_meta_cache_total	counter	result	Parquet-metadata cache hit/miss.

Recorded with the metrics facade (query planner, compaction, manifest commit):

Metric	Type	Meaning
bsdb_manifest_commit_skipped_removals_total	counter	Commits that skipped removals.
bsdb_manifest_commit_retries_total	counter	OCC commit retries.
bsdb_manifest_commit_rebases_total	counter	OCC commit rebases.
bsdb.query.planning_duration	histogram	Query planning duration.
bsdb.query.segment_pruning_duration	histogram	Segment pruning duration.
bsdb.query.segment_search_duration	histogram	Per-segment search duration.
bsdb.pruned.vector_bbox	counter	Segments pruned by vector bounding box.
bsdb_data_files_compacted	counter	Data files compacted.
bsdb_compaction_bytes_written	counter	Bytes written by compaction.

Search gateway metrics (bsdb_search_*)

Metric	Type	Labels	Meaning
bsdb_search_active_requests	gauge	—	In-flight HTTP requests.
bsdb_search_http_requests_total	counter	method, route	Requests by route class.
bsdb_search_http_request_duration_seconds	histogram	method, route	Request latency.
bsdb_search_query_seconds	histogram	op	Search latency by op (match/knn/hybrid/filter).
bsdb_search_docs_indexed_total	counter	result	Documents indexed.
bsdb_search_bulk_items_total	counter	status	Bulk items by outcome.
bsdb_search_refresh_seconds	histogram	—	_refresh duration.
bsdb_search_index_cache_hits_total / _misses_total	counter	—	Index table-cache hit/miss.
bsdb_search_index_fetch_bytes_total	counter	kind	Index bytes fetched from object storage.

Telemetry & Tracing

BenoStreamDB leverages a push-based OpenTelemetry (OTLP) pipeline to export distributed traces. Because it often runs in serverless functions (like AWS Lambda) where instances may be frozen between invocations, the Table API explicitly flushes traces on teardown (via the Drop trait) to ensure no data is lost.

Configuration

Tracing is disabled by default. To enable tracing, set the JAEGER_ENABLED environment variable. When enabled, traces are automatically pushed to an OTLP-compatible endpoint.

Environment Variable	Description	Default
JAEGER_ENABLED	Set to true to enable OpenTelemetry exporting.	false
OTEL_EXPORTER_OTLP_ENDPOINT	The destination URL for OTLP traces.	http://localhost:4317 (OTLP/gRPC default)
RUST_LOG	The log level filter (e.g., info, benostreamdb=debug).	info

Core Instrumentation

We instrument key paths to provide visibility into latency and bottlenecks:

• Write Path: write_async, commit_async

• Read Path: read_async, stream_all, vector_search_index, vector_search_flat

• Manifest Orchestration: Optimistic concurrency loops, schema updates, and conflict resolution in the ManifestManager.

Stateless Operational CLI (bsdb)

The bsdb binary is a standalone CLI tool that performs administrative actions directly against the object storage tier without requiring a long-running database server to be active.

Usage

# Start an interactive SQL REPL
bsdb repl

# Execute a single SQL query
bsdb query --query "SELECT * FROM my_table LIMIT 10"

# Register a table in the session
bsdb register --name my_table --uri s3://my-bucket/my-table

Table Management

You can perform routine maintenance using the bsdb table subcommand:

# Inspect table metadata and statistics
bsdb table inspect --uri s3://my-bucket/my-table

# Compact small data files to optimize read performance
bsdb table compact --uri s3://my-bucket/my-table

# Vacuum (delete) data files that are no longer referenced and older than N days
bsdb table vacuum --uri s3://my-bucket/my-table --older-than-days 7

Note: The CLI is stateless; it interacts directly with the storage URI.