Configuration Guide¶

BenoStreamDB is designed to be highly configurable through environment variables, runtime options, and a centralized configuration file.

Environment Variables¶

Core Engine & Memory Management¶

Variable	Description	Default
`BSDB_CACHE_GB`	Memory limit for vector (HNSW-IVF), inverted, and byte caches in GB.	`1`
`BSDB_BLOCK_CACHE_GB`	Memory limit for decoded RecordBatch Hot Row Cache in GB.	`1`
`BSDB_DISK_CACHE_DIR`	Local disk directory used for caching remote index files.	`/tmp/hdb_cache`
`BSDB_CACHE_DIR`	Local directory for the layered-index cache when reading external (REST-catalog) tables.	`$TMPDIR/benostream_cache`
`BSDB_STORAGE_URI`	Default table/storage URI for the gateway and Iceberg REST binaries.	`file:///tmp`
`BSDB_CONFIG`	Path to a centralized `benostream.toml` configuration file.	None
`BSDB_MAX_CONCURRENCY`	Maximum concurrent segment reader threads per query.	Auto (min(cores, 64))
`RAYON_NUM_THREADS`	Global Rayon worker pool thread count for parallel index construction.	Auto (50% of CPU cores)
`BSDB_DATAFUSION_MEMORY_GB`	DataFusion query memory limit in GB; SQL sorts/joins spill to disk past it.	Half the effective memory budget
`BSDB_TARGET_PARTITIONS`	DataFusion target partition count for query execution.	Effective CPU budget (respects cgroup limits)
`BSDB_INDEX_BUILD_CONCURRENCY`	Maximum concurrent background index builds. Non-positive values are ignored.	One build per ~8 GB RAM, capped at CPU count, min 1
`BSDB_MIN_FREE_DISK_GB`	Refuse a local flush that would drop free disk below this many GB.	`1.0`

Concurrency Tuning: `BSDB_MAX_CONCURRENCY` vs `RAYON_NUM_THREADS`¶

BSDB_MAX_CONCURRENCY and RAYON_NUM_THREADS operate on different execution layers and should generally not be set to the same value:

RAYON_NUM_THREADS (CPU-Bound Data Parallelism):
- Governs Rayon’s worker threadpool for compute-intensive tasks: AVX-512/NEON SIMD vector distance calculations, HNSW graph construction and deserialization, and Product Quantization (PQ) training.
- Rule: Keep $\le$ physical CPU cores (typically cores / 2 or cores - 1). Setting it higher causes OS thread context switching and CPU cache thrashing.
BSDB_MAX_CONCURRENCY (I/O-Bound Async Stream Concurrency):
- Governs Tokio async task concurrency for opening and streaming Parquet segments from local disk or remote object storage (s3://, gs://).
- Non-blocking I/O tasks spend most of their time awaiting network packets or disk blocks without consuming CPU.
- Rule: Can be $2\times$ to $4\times$ CPU cores (e.g. 8–32) for fast parallel segment fetching, bounded only by RAM memory buffer headroom.

Sizing & Tuning Matrix¶

Deployment Profile	Machine Specs	`RAYON_NUM_THREADS`	`BSDB_MAX_CONCURRENCY`	Notes
Benchmark / Container	4 Cores, 4 GB RAM	`2` – `4`	`4` – `8`	Stable profile used in 100k & 1M OpenSearch benchmark.
Production Server	16 Cores, 32 GB RAM	`8` – `14`	`16` – `32`	High throughput for mixed scalar + vector queries.
Cloud Object Store (S3/GCS)	8 Cores, 32 GB RAM	`6`	`32` – `48`	High I/O concurrency hides cloud object storage latency.

Security¶

Authentication (search gateway + Flight SQL)¶

Stateless auth — no user database. Identity is delegated to a shared secret or an external IdP’s signing key. The same credentials protect the OpenSearch/ES API, the Qdrant API, and the Flight SQL gRPC service.

Variable	Description	Default
`BSDB_API_KEY`	Shared secret. HTTP: `Authorization: Bearer <key>`, `Authorization: ApiKey <key>`, or `api-key: <key>` (Qdrant). Flight: `authorization: Bearer <key>`.	unset
`BSDB_JWT_SECRET`	HS256 shared secret for JWT verification.	unset
`BSDB_JWT_PUBLIC_KEY`	RS256 public key (PEM) for JWT verification — the “delegate to your IdP” path.	unset
`BSDB_JWT_AUDIENCE`	Required `aud` claim (if set).	unset
`BSDB_JWT_ISSUER`	Required `iss` claim (if set).	unset
`BSDB_AUTH_REQUIRED`	Fail closed (`401`/`UNAUTHENTICATED`) when no credential is configured.	`false`
`BSDB_METRICS_PUBLIC`	Exempt `/metrics` from auth (for an internal Prometheus scraper).	`false`

Keep the secret in a secret manager (Vault, AWS/GCP/Azure Secrets Manager) or a 0600 file mounted at /run/secrets/ (Docker/K8s) or /etc/benostreamdb/secrets/, loaded via systemd LoadCredential= / EnvironmentFile= (mode 0600). Never commit it or place it in a world-readable env file.

SSRF guard¶

Variable	Description	Default
`BSDB_SSRF_GUARD`	Enable the SSRF guard. When set to `1`/`true`/`yes`, external URIs and custom object-store endpoints (`AWS_ENDPOINT_URL`, `AZURE_STORAGE_ENDPOINT`, `AZURE_ENDPOINT`, `GOOGLE_STORAGE_ENDPOINT`, `GOOGLE_ENDPOINT`, `GCS_ENDPOINT`) that resolve to loopback, private, or link-local addresses are rejected. Off by default so local catalogs (`http://localhost:8181`) keep working; enable it in production.	off

Embeddings¶

Embedding provenance is a core table property (benostream.embedding.*, see the main README’s “Embedding Provenance” section). These variables configure the embedding generator used by the MCP server and the Python helpers.

Variable	Description	Default
`BSDB_VALIDATE_EMBEDDINGS`	`strict` (also `1`/`true`/`on`) scans every `List`/stringified vector row against the table’s declared dimension. By default only `FixedSizeList` columns are checked, at schema level; model identity is never inferred from values.	off (schema checks only)
`BSDB_EMBEDDING_PROVIDER`	`local` (in-process Candle), `http`, or `none`.	`local` unless `BSDB_EMBEDDING_URL` is set
`BSDB_EMBEDDING_MODEL`	Hugging Face BERT-family model id that publishes `model.safetensors`.	`BAAI/bge-small-en-v1.5`
`BSDB_EMBEDDING_CACHE_DIR`	Private model cache; reused after the first download.	`<data-dir>/models`
`BSDB_EMBEDDING_URL`	Ollama / OpenAI-compatible / TEI embedding endpoint.	None
`BSDB_EMBEDDING_API_KEY`	Bearer token for the HTTP endpoint.	None
`BSDB_EMBEDDING_BATCH_SIZE`	Candle inference batch size.	`16`
`BSDB_EMBEDDING_MAX_LENGTH`	Tokenizer truncation length.	`512`
`BSDB_EMBEDDING_DIM`	Optional dimension override for MRL truncation. Known models supply it from the id map, and loaded models supply it from `config.json` (`hidden_size`).	Model default
`BSDB_EMBEDDING_QUERY_PREFIX`	Override the query instruction prefix. Empty string disables it.	Model default
`BSDB_EMBEDDING_DOCUMENT_PREFIX`	Override the document instruction prefix. Empty string disables it.	Model default

The Candle loader runs on CPU in the published build. The legacy model_path, device, quantized, intra_threads, pooling, and show_download_progress JSON keys are accepted for backward compatibility but ignored.

Write-Ahead Log (WAL) & Ingestion¶

Variable	Description	Default
`BSDB_WAL_DIR`	Directory for the Write-Ahead Log (WAL) used for streaming recovery.	`{table_uri}/_wal`
`BSDB_WAL_DURABILITY`	WAL flush durability policy (`always`, `adaptive`, `periodic`).	`adaptive`
`BSDB_WAL_COMPACT_MB`	File size threshold in MB before triggering WAL log segment compaction.	`1024` (1 GB)
`BSDB_WAL_SYNC_BATCH_SIZE`	Appended operations batch size before triggering a WAL sync flush.	`10`
`BSDB_WAL_SYNC_INTERVAL_MS`	Maximum elapsed milliseconds between background WAL sync flushes.	`100`
`BSDB_STREAMING_FLUSH_INTERVAL_SECS`	Background timer interval in seconds to automatically flush write buffers to Iceberg snapshots.	None
`BSDB_MAX_INGEST_RAM_GB`	Ingest RAM high-water mark in GB; writers pause above it (with a fail-open grace) until background tasks reclaim memory.	80% of available memory
`BSDB_INGEST_MEMORY_BUDGET_GB`	Heap-trim budget during bulk ingest: RSS above it triggers `malloc_trim` at unit boundaries.	80% of available memory
`BSDB_INGEST_BACKPRESSURE_GRACE_SECS`	Fail-open grace before a stalled writer proceeds anyway. Non-positive or unparseable values fall back to the default.	`60`
`BSDB_MANIFEST_MAX_FILES_BEFORE_CONSOLIDATION`	Manifest files accumulated by the append-only fast path before a full consolidation commit.	`32`
`BSDB_VACUUM_MIN_FILE_AGE_SECS`	Minimum age before vacuum deletes an unreferenced file. `0` disables the grace period.	`60`
`BSDB_SEARCH_WAL_DURABILITY`	Compatibility alias for `BSDB_WAL_DURABILITY`, read by the search gateway’s table opener.	`adaptive`

Vector Index & GPU¶

Variable	Description	Default
`BSDB_GPU_DEVICE`	Device or comma-separated pool for vector index builds and distance kernels: `auto`, `cpu`, `cuda[:N]`, `mps`, or `cuda:0,cuda:1`. A pool assigns one device per worker thread round-robin; a single HNSW search still runs on one device. `BENOSTREAM_GPU_DEVICE` is accepted as a fallback.	unset (`auto`)
`BSDB_HNSW_M`	HNSW per-node neighbour count.	`16`
`BSDB_HNSW_EF_CONSTRUCTION`	HNSW build beam width.	`max(2 × M, 40)`
`BSDB_HNSW_N_LISTS`	IVF list count.	Heuristic from vector count
`BSDB_KMEANS_SEED`	PRNG seed for k-means centroid initialization during IVF index building. Set to an integer to override the deterministic default seed (`0x00C0_FFEE_1337`).	`0x00C0_FFEE_1337` (deterministic)
`BSDB_HNSW_SEED`	PRNG seed for HNSW layer level assignment during graph building.	unset (thread-local randomness)
`BSDB_NVRTC_PATH`	NVRTC library file or directory used to compile GPU kernels.	Auto-discovery (CUDA home, Python site-packages, `LD_LIBRARY_PATH`)
`BSDB_DRIFT_MEMORY_MB`	In-memory budget for graph-algorithm UDAF accumulators before they spill to the CSR index.	Engine memory budget

Catalog¶

Variable	Description	Default
`BSDB_CATALOG_TYPE`	Catalog implementation: `rest`, `glue`, `hive`, `jdbc`, `nessie`, or `unity`.	Inferred from the URI scheme or `BSDB_CONFIG`
`BSDB_CATALOG_URI` / `BSDB_CATALOG_URL`	Catalog endpoint.	None
`BSDB_CATALOG_NAME`	Warehouse / catalog name to register.	None
`BSDB_CATALOG_NAMESPACE`	Default namespace for unqualified tables.	None
`BSDB_CATALOG_CREDENTIAL`	Vendor credential (`client_id:client_secret`, or the provider’s equivalent).	None
`BSDB_CATALOG_TOKEN`	Bearer token for catalog requests.	None

See Catalog Usage and benostream.toml below for the full catalog configuration surface.

MCP Server (`benostreamdb-mcp`)¶

Variable	Description	Default
`BSDB_WAREHOUSE`	Local warehouse path exposed to the agent.	System temp dir
`BSDB_READWRITE_SCHEMAS`	Comma-separated or JSON array of schemas that allow writes.	`["scratch"]`
`BSDB_READONLY_SCHEMAS`	Comma-separated or JSON array of schemas locked read-only.	`["default", "public"]`
`BSDB_DEFAULT_WRITE_SCHEMA`	Schema for unqualified table creation.	`scratch`
`BSDB_EPHEMERAL`	Use a temporary warehouse for this session (`1`/`true`).	`false`
`BSDB_MCP_CONFIG`	Full JSON configuration string.	None
`BSDB_EMBEDDING_*`	Embedding generator settings (see Embeddings).	See above

See contrib/benostreamdb-mcp/README.md for the full CLI/JSON mapping.

Search Gateway (`benostreamdb-search`)¶

The OpenSearch/Qdrant gateway is a separate component with its own configuration surface (BSDB_SEARCH_*, BSDB_QDRANT_BIND, BSDB_QDRANT_PORT). See the search gateway README for the full table.

Server binaries¶

The Flight SQL server (benostreamdb-flight) has its own variables:

Variable	Description	Default
`BSDB_FLIGHT_BIND`	gRPC bind address.	`127.0.0.1`
`BSDB_FLIGHT_PORT`	gRPC port.	`50051`
`BSDB_FLIGHT_MAX_ROWS`	Row cap applied to Arrow Flight results.	`100000`
`BSDB_FLIGHT_READ_ONLY`	Refuse DDL, DML, and ingest. Set to `1`/`true`.	off
`BSDB_WAREHOUSE`	Warehouse root for the server.	`./data`
`BSDB_CATALOG_NAME`	Catalog name to expose.	From `benostream.toml`

See the Flight SQL README.

Cloud Storage & Telemetry¶

Variable	Description	Default
`AWS_ENDPOINT_URL`	Custom S3 endpoint URL (used for RustFS, LocalStack, Ceph).	AWS default
`JAEGER_ENABLED`	Enable distributed OpenTelemetry tracing via Jaeger / OTLP.	`false`
`BSDB_METRICS_BIND`	Bind address for the embedded/Flight metrics HTTP listener.	`127.0.0.1`
`BSDB_METRICS_PORT`	Port for the embedded/Flight metrics HTTP listener.	`9090`
`BSDB_METRICS_PUBLIC`	Exempt `/metrics` from auth (for an internal Prometheus scraper).	`false`
`BSDB_OTLP_ENDPOINT`	OTLP/HTTP metrics endpoint (e.g. `http://localhost:4318`). The pusher appends `/v1/metrics` and is disabled when unset.	None
`BSDB_OTLP_HEADERS`	Comma-separated `key=value` headers for the OTLP push (for example an auth token).	None
`BSDB_OTLP_INTERVAL_SECS`	OTLP push period in seconds.	`15`
`BSDB_SERVICE_NAME`	`service.name` resource attribute on exported metrics.	`benostreamdb`

Development, Test & Benchmark Variables¶

These are not engine runtime settings. They are read by scripts, tests, or packaging workflows in this repository.

Variable	Used by	Description
`BSDB_ALLOW_DEBUG`	`scripts/bench_csr_subgraph.py`	Allow the CSR benchmark to run against a debug build (`1`).
`BSDB_MAX_DEGREE`	`scripts/bench_csr_subgraph.py`	Degree cap for the generated subgraph.
`BSDB_DATA`	`scripts/prepare_demo.py`, `scripts/build_three_tier.py`, `scripts/build_demo_dataset.py`	Input directory for downloaded Wikipedia/demo dumps.
`BSDB_INDEX`	`benchmarks/competitors/run_bench.sh`	Index type selected by the competitor benchmark harness (`hnsw`, `hnsw_tq8`, `hnsw_pq`, …).
`BSDB_TABLE_DIR`	`scripts/build_three_tier.py`	Output table directory.
`BSDB_THREE_TIER_OUT`	`scripts/build_three_tier.py`	Root output directory for the three-tier build.
`BSDB_EMBED_SHARD_ROWS`	`scripts/prepare_demo.py`	Rows per embedding shard.
`BSDB_LOAD_CHUNK_ROWS`	`scripts/prepare_demo.py`	Rows per load chunk.
`BSDB_DEMO_DB`	`examples/web_ui/`	Demo database path for the Streamlit/Dash UIs.
`BSDB_SKIP_GPU_PROBE`	`python/benostreamdb/__init__.py`	Internal guard that stops the GPU-probe child from recursing.
`BSDB_SOAK_RSS_CEILING_GB`	`tests/soak.rs`	RSS ceiling for the soak test (default 8 GiB).
`BSDB_TEST_S3_BUCKET`	S3 multi-writer tests	Bucket used by the concurrency tests.
`BSDB_WORKLOAD_SEED`	randomized workload tests	LCG seed (default `12345`).
`BSDB_WORKLOAD_STEPS`	randomized workload tests	Number of randomized steps (default `400` for Rust, `100` for the Flight Python test).
`BSDB_RELEASE_VERSION`	release workflow / dbt adapter	Forced version string for packaging.
`BSDB_VERSION`	`docker-compose.production.yml`	Image tag for the search/Flight containers (`latest` by default).

See Monitoring & Operations for the full metric catalog and the per-deployment-mode endpoints.

The benostream.toml File¶

You can use a TOML file to manage complex configurations, especially for catalogs and multi-cloud storage.

BenoStreamDB looks for this file in the following order:

Environment variable BSDB_CONFIG
./benostream.toml (current directory)
~/.benostream/config.toml

Example Configuration¶

[storage]
type = "s3"
bucket = "my-data-lake"
region = "us-east-1"
endpoint = "http://rustfs:9000"

[cache]
memory_limit_gb = 8
block_cache_gb = 8
disk_cache_enabled = true
disk_cache_path = "/mnt/fast-ssd/hdb_cache"

[wal]
durability = "adaptive"

[catalog]
type = "rest"
url = "https://polaris.example.com/api/catalog/v1"
warehouse = "main_warehouse"
credential = "CLIENT_ID:CLIENT_SECRET"
scope = "PRINCIPAL_ROLE:ALL"
token_refresh_interval_secs = 300

Storage Credentials¶

BenoStreamDB uses the standard object-store crate, which automatically picks up credentials from:

AWS: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, or IAM Roles.
GCP: GOOGLE_APPLICATION_CREDENTIALS (JSON key file path).
Azure: AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_KEY.

Query Configuration (QueryConfig)¶

Query-level options are set via the QueryConfig struct (Rust). The Python binding exposes the same SQL execution surface through Table.sql() and Session.sql().

Option	Description	Default
`query_timeout_secs`	Maximum duration before a query is cancelled. Set to `0` for no timeout.	`0` (no timeout)
`max_result_rows`	Hard cap on rows returned. Prevents unbounded result sets from exhausting memory.	`10,000,000`
`max_concurrency`	Maximum parallel segment readers. Capped at 64 to prevent oversubscription.	Auto-detected (`available_parallelism()`, max 64)

Python Example¶

import benostreamdb as bsdb

table = bsdb.Table("s3://bucket/my-table")

# The table is registered as 't' for the duration of the query
results = table.sql(
    "SELECT * FROM t WHERE embedding <-> '[0.1, 0.2]'::vector"
)

Query timeouts, result-row caps, and reader concurrency are QueryConfig fields on the Rust surface; they are not keyword arguments on the Python Table.sql() method.

Rust Example¶

use benostreamdb::{Table, QueryConfig};

let table = Table::new("s3://bucket/my-table")?;
let config = QueryConfig::default()
    .query_timeout_secs(30)
    .max_result_rows(100_000);

let batches = table.query_with_config("SELECT * FROM documents", config).await?;

Vector index configuration¶

(merged from the former CONFIGURATION.md)

This guide covers configuration parameters for tuning vector search performance in BenoStreamDB.

Configuration Parameters¶

HNSW Parameters¶

`ef_search`¶

Controls the search beam width for HNSW index queries.

Type: Integer
Default: 64
Range: 1 - 1000
Effect: Higher values increase accuracy but reduce speed

Usage (optimizer hint):

SELECT /*+ INDEX_HINT(ef_search=128) */
       id, embedding <-> '[0.1, 0.2, 0.3]'::vector AS distance
FROM documents
ORDER BY distance
LIMIT 10;

Python API (per-query vector filter):

import benostreamdb as bsdb

table = bsdb.Table("s3://bucket/documents")

results = table.to_pandas(
    vector_filter={
        "column": "embedding",
        "query": [0.1, 0.2, 0.3],
        "k": 10,
        "ef_search": 128,
    }
)

Performance Impact:

ef_search = 32: Fast, ~90% recall
ef_search = 64: Balanced, ~95% recall (default)
ef_search = 128: Accurate, ~98% recall
ef_search = 256: Very accurate, ~99.5% recall

IVF Parameters¶

`probes`¶

Controls the number of IVF clusters to search.

Type: Integer
Default: 10
Range: 1 - 1000
Effect: Higher values increase accuracy but reduce speed

Usage (optimizer hint):

SELECT /*+ INDEX_HINT(probes=20) */
       id, embedding <-> '[0.1, 0.2, 0.3]'::vector AS distance
FROM documents
ORDER BY distance
LIMIT 10;

Python API (per-query vector filter):

import benostreamdb as bsdb

table = bsdb.Table("s3://bucket/documents")

results = table.to_pandas(
    vector_filter={
        "column": "embedding",
        "query": [0.1, 0.2, 0.3],
        "k": 10,
        "probes": 20,
    }
)

Performance Impact:

probes = 5: Fast, ~85% recall
probes = 10: Balanced, ~92% recall (default)
probes = 20: Accurate, ~97% recall
probes = 50: Very accurate, ~99% recall

Index Control¶

`use_index`¶

Controls whether to use vector indexes or force sequential scan.

Type: Boolean
Default: true
Effect: When false, forces sequential scan (useful for testing)

Usage (optimizer hint):

-- Force a sequential scan (for testing)
SELECT /*+ INDEX_HINT(use_index=false) */
       id, embedding <-> '[0.1, 0.2, 0.3]'::vector AS distance
FROM documents
ORDER BY distance
LIMIT 10;

Tuning Guidelines¶

Accuracy vs Speed Tradeoff¶

Use Case	ef_search	probes	Expected Recall	Relative Speed
Real-time search	32	5	~88%	4x faster
Balanced	64	10	~94%	1x (baseline)
High accuracy	128	20	~98%	0.5x slower
Maximum accuracy	256	50	~99.5%	0.2x slower

Dataset Size Recommendations¶

Small datasets (< 100K vectors):

SELECT /*+ INDEX_HINT(ef_search=128, probes=20) */
       id, embedding <-> '[0.1, 0.2, 0.3]'::vector AS distance
FROM documents
ORDER BY distance
LIMIT 10;

Medium datasets (100K - 10M vectors):

SELECT /*+ INDEX_HINT(ef_search=64, probes=10) */ ...   -- defaults

Large datasets (> 10M vectors):

SELECT /*+ INDEX_HINT(ef_search=32, probes=5) */ ...

Dimensionality Recommendations¶

Low dimensions (< 128):

SELECT /*+ INDEX_HINT(ef_search=64, probes=10) */ ...

Medium dimensions (128 - 768):

SELECT /*+ INDEX_HINT(ef_search=64, probes=10) */ ...

High dimensions (> 768):

SELECT /*+ INDEX_HINT(ef_search=128, probes=20) */ ...

Benchmarking¶

Measuring Recall¶

import benostreamdb as bsdb
import numpy as np

table = bsdb.Table("s3://bucket/documents")
query_vector = [0.1, 0.2, 0.3]

# High-recall reference run (wide beam)
reference = table.to_pandas(
    vector_filter={
        "column": "embedding",
        "query": query_vector,
        "k": 100,
        "ef_search": 512,
    }
)
reference_ids = set(reference["id"])

# Run under test (default beam)
results = table.to_pandas(
    vector_filter={
        "column": "embedding",
        "query": query_vector,
        "k": 100,
        "ef_search": 64,
    }
)
result_ids = set(results["id"])

# Calculate recall
recall = len(reference_ids & result_ids) / len(reference_ids)
print(f"Recall@100: {recall:.2%}")

Measuring Latency¶

import time
import numpy as np
import benostreamdb as bsdb

table = bsdb.Table("s3://bucket/documents")
vector_filter = {
    "column": "embedding",
    "query": [0.1, 0.2, 0.3],
    "k": 10,
    "ef_search": 64,
}

# Warmup
for _ in range(10):
    table.to_pandas(vector_filter=vector_filter)

# Measure
latencies = []
for _ in range(100):
    start = time.time()
    table.to_pandas(vector_filter=vector_filter)
    latencies.append(time.time() - start)

print(f"P50: {np.percentile(latencies, 50)*1000:.2f}ms")
print(f"P95: {np.percentile(latencies, 95)*1000:.2f}ms")
print(f"P99: {np.percentile(latencies, 99)*1000:.2f}ms")

Advanced Configuration¶

Per-Query Configuration¶

Per-query vector-search hints are supported via SQL comments:

-- Supported: INDEX_HINT(ef_search=..., probes=...)
SELECT /*+ INDEX_HINT(ef_search=128, probes=20) */
    id, embedding <-> '[0.1, 0.2, 0.3]'::vector AS distance
FROM documents
ORDER BY distance
LIMIT 10;

Index build parameters are set during index creation using the fluent add_index method:

import benostreamdb as bsdb

table = bsdb.Table("s3://bucket/my-table")

# TurboQuant 8-bit quantization (Recommended Default)
# 4x compression, near-lossless accuracy
table.add_index("embedding", "hnsw_tq8")

# TurboQuant 4-bit quantization
# 8x compression, maximum efficiency
table.add_index("embedding", "hnsw_tq4")

# Custom HNSW parameters
table.add_index(
    "embedding",
    {
        "type": "hnsw",
        "complexity": 16, # Max connections per node (formerly 'm')
        "quality": 200,    # Construction beam width (formerly 'ef_construction')
    }
)

# Product Quantization (PQ)
table.add_index(
    "embedding",
    {
        "type": "hnsw_pq",
        "compression": 32 # PQ subspaces (formerly 'subspaces')
    }
)

Troubleshooting¶

Slow Queries¶

Symptom: Queries taking longer than expected

Solutions:

Reduce ef_search or probes for faster (but less accurate) results
Check if index exists: SHOW INDEXES FROM table_name
Verify index is being used: Check query plan
Consider using binary vectors for 32x speedup

Low Recall¶

Symptom: Missing relevant results

Solutions:

Increase ef_search or probes
Verify index quality: Rebuild if necessary
Check vector normalization (for cosine distance)
Ensure query vector has same preprocessing as indexed vectors

Out of Memory¶

Symptom: OOM errors during queries

Solutions:

Reduce ef_search to lower memory usage
Use binary vectors for 32x memory reduction
Use sparse vectors for high-dimensional sparse data
Increase available memory

Best Practices¶

Start with defaults: Use default parameters (ef_search=64, probes=10) initially
Measure first: Benchmark recall and latency before tuning
Tune incrementally: Adjust one parameter at a time
Monitor production: Track recall and latency metrics in production
Document settings: Record configuration choices for reproducibility

Configuration Reference¶

Hint key	Type	Default	Range	Description
`ef_search`	Integer	64	1-1000	HNSW search beam width
`probes`	Integer	10	1-1000	IVF clusters to search
`use_index`	Boolean	true	true/false	Enable/disable index usage

Configuration Guide¶

Environment Variables¶

Core Engine & Memory Management¶

Concurrency Tuning: BSDB_MAX_CONCURRENCY vs RAYON_NUM_THREADS¶

Sizing & Tuning Matrix¶

Security¶

Authentication (search gateway + Flight SQL)¶

SSRF guard¶

Embeddings¶

Write-Ahead Log (WAL) & Ingestion¶

Vector Index & GPU¶

Catalog¶

MCP Server (benostreamdb-mcp)¶

Search Gateway (benostreamdb-search)¶

Server binaries¶

Cloud Storage & Telemetry¶

Development, Test & Benchmark Variables¶

The benostream.toml File¶

Example Configuration¶

Storage Credentials¶

Query Configuration (QueryConfig)¶

Python Example¶

Rust Example¶

Vector index configuration¶

Configuration Parameters¶

HNSW Parameters¶

ef_search¶

IVF Parameters¶

probes¶

Index Control¶

use_index¶

Tuning Guidelines¶

Accuracy vs Speed Tradeoff¶

Dataset Size Recommendations¶

Dimensionality Recommendations¶

Benchmarking¶

Measuring Recall¶

Measuring Latency¶

Advanced Configuration¶

Per-Query Configuration¶

Troubleshooting¶

Slow Queries¶

Low Recall¶

Out of Memory¶

Best Practices¶

Configuration Reference¶

Concurrency Tuning: `BSDB_MAX_CONCURRENCY` vs `RAYON_NUM_THREADS`¶

MCP Server (`benostreamdb-mcp`)¶

Search Gateway (`benostreamdb-search`)¶

`ef_search`¶

`probes`¶

`use_index`¶