# BenoStreamDB Roadmap & Architecture Status

**Vision**: Make Apache Iceberg executable as a high-performance database with rebuildable secondary index overlays (CSR graph, HNSW vector, BM25 text, Roaring bitmaps).

---

## 🎯 Architecture & Status at a Glance

| Status | Capability Area | Key Components / Interfaces |
| :--- | :--- | :--- |
| ✅ **Developer Preview** | **Core Lakehouse Engine** | Apache Iceberg V2/V3 format, Parquet storage, DataFusion ANSI SQL, MVCC lock-free commits, WAL, Ingest RAM backpressure. |
| ✅ **Developer Preview** | **Concurrency & Durability** | Unified atomic write buffer (`PendingWrite`), Table-wide maintenance barriers (`maintenance_lock`), Fenced vacuum/orphan GC. |
| ✅ **Developer Preview** | **Vector Search Overlays** | Zero-copy Arrow IPC HNSW & IVF-PQ, TurboQuant™ (TQ8; TQ4 retiring), SIMD intrinsics, GPU acceleration (`benostream-gpu-ann`: CUDA, Metal, Vulkan). |
| ✅ **Developer Preview** | **Text & Scalar Overlays** | Inverted BM25 Okapi indexes, Roaring Bitmap scalar indexes, Composite multi-column filters, Statistics pruning. |
| ✅ **Developer Preview** | **Core Graph Primitives** | Zero-copy Memory-Mapped CSR Graph Index (`MmapCsrGraph`), Core Graph Traversal operator (`Table::graph_neighborhood`), Graph SQL UDFs. |
| ✅ **Developer Preview** | **JSON Path Functions & Index** | PostgreSQL `json_*` UDFs (`json_extract_path`, `json_contains`, `json_path_query`, …) over `Utf8` columns, plus a `json_path` `(path, value) -> row_ids` Puffin overlay with planner pushdown. |
| ✅ **Developer Preview** | **Lakehouse Connectors** | Native Apache Spark connector (`spark-benostreamdb`), Trino connector (`trino-benostreamdb`), Official dbt adapter (`dbt-benostreamdb`). Full engine function-surface parity across Python/dbt/Spark/Trino, guarded by `tests/python/test_function_parity.py`. |
| ✅ **Developer Preview** | **Ecosystem Interfaces** | Optional Arrow Flight SQL server (`server/flight_sql`), Contrib Search Gateway (`contrib/benostreamdb-search`: OpenSearch 7.10 & Qdrant REST), stateless auth (API key + JWT, shared module). |
| ✅ **Developer Preview** | **AI-Native & Search Extensions** | MCP server (`contrib/benostreamdb-mcp`), SQL graph traversal (`FROM graph_neighbors(...)`), `Table::subscribe()` change feed + predicate-filtered subscriptions. |
---

## 🏛️ Architectural Foundations (Completed)

BenoStreamDB implements an **indexed, compute-disaggregated lakehouse architecture**:
1. **The Overlay Invariant**: Parquet data files in Apache Iceberg are the authoritative source of truth. All secondary indexes (CSR graph, HNSW vector, BM25 text, Roaring bitmaps) are **advisory, rebuildable sidecar overlays**. If an index is lost or deleted, queries fall back to Parquet scans while the overlay rebuilds.
2. **Serverless-First, Server-Optional**: The core database is an in-process, embeddable library (Python/Rust/JVM) operating directly against object storage (S3/GCS/Azure/Local). Arrow Flight SQL provides an optional single-process network access layer for remote clients and BI tools.
3. **Lakehouse Independence**: All core connectors (`dbt-benostreamdb`, `spark-benostreamdb`, `trino-benostreamdb`) and optional servers (`server/flight_sql`) operate independently of contrib compatibility packages.

### Key Completed Milestones
- **Core Concurrency Hardening**:
  - Unified `pending_writes: Arc<RwLock<Vec<PendingWrite>>>` ensuring atomic WAL-to-buffer pairing.
  - Table-level `maintenance_lock` establishing an exclusive barrier between concurrent writes and destructive operations (`truncate`, `vacuum`, `compaction`).
  - Strict manifest load error propagation across all mutation paths.
  - Regression verified under parallel stress testing (`tests/test_h1_h2_concurrency_regression.rs`).
- **Core Graph Engine**:
  - Memory-mapped Compressed Sparse Row (`.graph_v2.csr.{offsets,edges,dict}`) engine.
  - Core `Table::graph_neighborhood()` traversal operator with fast-path CSR execution, relation filtering, and unindexed edge fallback.
  - Graph-scoped vector and BM25 queries callable directly from Python, SQL, and search APIs.
- **Native Ingest Orchestrator (`Table::ingest_async`)**:
  - Cluster-free bulk ingestion with memory-budgeted chunking (`malloc_trim` heap policy), lock-free OCC manifest commits, and resume idempotency.
- **Multi-Catalog Resolution**:
  - Seamless integration with Apache Polaris, Nessie, AWS Glue, Hive Metastore, and file-based Iceberg catalogs.
- **PostgreSQL `json` Path Functions**:
  - `json_extract_path` / `json_extract_path_text` (variadic path), `json_contains` (recursive `@>`), `json_exists`, `json_typeof`, and `json_path_exists` / `json_path_query` (jsonpath subset: `$.a.b[0]`, `[*]`). Implemented in `src/core/sql/udf/json.rs`.
- **Cross-connector function parity**:
  - The engine's full custom function surface (vector scalar UDFs, vector
    aggregates, JSON, graph UDAFs) is exposed consistently across Python, dbt,
    Spark, and Trino, and guarded by `tests/python/test_function_parity.py`
    (source of truth: `udf::registered_function_names()`). The Spark DSv2
    `FunctionCatalog` exposes 30 functions, and a pass-through reader
    (`option("query", "<engine SQL>")`) makes the vector aggregates and graph
    UDAFs — which have no DSv2 scalar equivalent — reachable from Spark. The
    engine's ad-hoc `Table::sql()` context now registers the full surface via the
    shared `udf::register_all_custom_udfs` helper.
- **Ingest back-pressure liveness**:
  - The ingest RAM high-water mark is now baseline-relative (it bounds the
    engine's *growth* since the write began) with a bounded fail-open grace
    (`BSDB_INGEST_BACKPRESSURE_GRACE_SECS`, default 60s), so a writer can no
    longer block forever when process RSS is already at/above the cap for reasons
    the write cannot relieve.
- **MCP Server (`contrib/benostreamdb-mcp`)**:
  - Dedicated MCP server exposing BenoStreamDB tables, schemas, SQL queries, and
    hybrid vector/graph retrieval as native tools for Claude Desktop, Cursor, and
    Antigravity. Windows `.msi` installer via `cargo-wix`.
  - **0.13.0:** local, no-LLM embeddings (pure-Rust Candle BERT, BGE-small by
    default) with Ollama/OpenAI/TEI HTTP providers, `list_embedding_models`, and
    embedding provenance shared with the core engine.
- **JSON-Path Inverted Index**:
  - `IndexAlgorithm::JsonPath { paths }` builds a `(path, value) -> row_ids`
    Parquet overlay in `src/core/index/json_path.rs`, packed into the segment's
    Puffin bundle under the `json_path` category. The planner rewrites
    `json_contains` / `json_exists` / `json_path_exists` /
    `json_extract_path_text(...) = 'value'` into index lookups
    (`HybridReader::query_json_path_first`), falling back to a full scan for
    unindexed paths. Verified by a differential oracle against a full scan
    (`tests/test_json_path_index.rs`).
- **SQL Graph Traversal Operator Extensions**:
  - DataFusion table functions expose graph walks directly in `FROM` clauses
    (`graph_neighbors`, `graph_shortest_path`, `graph_all_shortest_paths`,
    `graph_subgraph`, `graph_connecting_paths`), following the in-memory /
    out-of-core `GraphMode` pattern. Implemented in
    `src/core/sql/graph_udf/graph_table_functions.rs`; reachable from Python,
    dbt, Spark (pass-through), Trino, and Flight SQL.
- **Declarative Edge Table DDL**:
  - First-class table metadata (via `Manifest.properties`): `table_type`
    (`node` | `edge` | `table`), `src_col`, `dst_col`, `relation_col`,
    `weight_col`, `id_col`, `label_col`. `create_edge_table` /
    `create_node_table` stamp the convention; `Session.list_graph_tables()`
    enumerates them. Declaring `table_type = 'edge'` auto-configures the forward
    and reverse CSR graph indexes (`Table::ensure_edge_indexes_async`).
- **`Table::subscribe()` + predicate-filtered subscriptions**:
  - Per-table `tokio::sync::broadcast` channel emitting committed `RecordBatch`es
    plus a `Commit` marker; exposed in Python, as the `subscribe_events` SQL table
    function, a Flight streaming ticket, and an MCP tool.
    `Table::subscribe_filtered("age > 30")` re-applies the predicate above each
    batch.
- **End-to-end benchmark suite**:
  - Vector ANN (SIFT/HNSW/TurboQuant), graph analytics, SQL OLAP, lexical &
    hybrid (BEIR), production concurrency/maintenance/recovery, and the
    shared-envelope Docker competitor matrix (FAISS, hnswlib, LanceDB, pgvector,
    OpenSearch, Neo4j+GDS, cuGraph).
- **Workflow tutorials**:
  - One runnable tutorial per workflow: (1) AI/semantic retrieval via MCP,
    (2) JSON/document retrieval with a JSON-path index, (3) graph traversal via
    `graph_neighbors(...)`. Python versions are Jupyter notebooks
    (`examples/tutorials/*.ipynb`, verified to execute end-to-end); the
    SQL/connector forms (Spark/Trino/dbt/Flight/MCP/Rust) are markdown pages
    under `docs/tutorials/`.

---

## 📋 Active Roadmap Themes

> **Tracking.** Actionable open items are tracked as GitHub issues labeled
> [`roadmap`](https://github.com/benolabsai/benostreamdb/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap).
> The themes below retain the design rationale and priority; the issue list is the
> source of truth for status. `status: committed` marks current-cycle work,
> `status: exploratory` marks ideas awaiting user or benchmark validation, and
> `status: deferred` marks work that is intentionally unscheduled.

> **Release line.** v0.13.0 shipped embedding provenance (core
> `benostream.embedding.*` table properties, the write/commit vector-space guard,
> and local BGE-small embeddings in the MCP server). **v0.14.0 is the next
> planned release.**

Themes are ordered by priority.

### Theme 1: Deprecate & Remove TQ4 Quantization Engine Option — 🔴 **Highest Priority**
*Retire 4-bit TurboQuant (`hnsw_tq4` / `tq4`) across the engine, APIs, and Puffin sidecars.*

> **Rationale & Benchmark Findings.** Comprehensive benchmarking demonstrated that 4-bit TurboQuant (`tq4`) performs significantly worse in recall compared to full-precision `hnsw` and 8-bit TurboQuant (`tq8`), without delivering any compelling added value or Pareto efficiency. `tq8` already provides ~4× compression with resilient recall, and product quantization (`pq`) serves extreme compression use cases far more effectively.
>
> While TQ4 has been removed from benchmark suites, competitor matrixes, and reports for the upcoming release, its full engine code removal is scheduled as the highest priority item for the next release (respecting the current feature freeze).

- [ ] **Remove `IndexAlgorithm::HnswTq4` variant and parser** — 🔴 **Highest Priority**:
  - Deprecate and remove `IndexAlgorithm::HnswTq4`, `IndexAlgorithm::hnsw_tq4()`, and parser mappings (`"hnsw_tq4"`, `"tq4"`).
- [ ] **Remove TQ4 quantization pipeline from HNSW/IVF and Segment writers** — 🔴 **Highest Priority**:
  - Clean up `QuantizerImpl::TurboQuant` 4-bit path in `src/core/index/hnsw_ivf.rs`, `PUFFIN_BLOB_VECTOR_HNSW_TQ4` in `src/core/puffin.rs`, and segment index writers.
- [ ] **Clean connector and language bindings** — 🔴 **Highest Priority**:
  - Remove Python `table.quantize(..., "tq4")`, PyO3 helper parsing, MCP tool prompts, Spark/dbt catalog descriptions, and surface parity tests.
- [ ] **Clean documentation and test suites** — 🔴 **Highest Priority**:
  - Remove `differential_ann_tq4_recall` and `explain_reports_tq4_index_access_path` tests, update vector API documentation.

### Theme 2: Multi-Precision Vector Indexes
*Let one vector column serve both exact and compressed search.* — 🔴 **High**

> **Current state.** A vector column holds a **single** index: adding a second
> replaces the first (see `Table::add_index` in `src/core/table/index_config.rs`),
> and the default is now full-precision `hnsw`. Quantized variants are opt-in:
> `hnsw_tq8` (TurboQuant 8-bit, ≈4× smaller) and `hnsw_pq` — trading recall for
> a smaller index and cheaper distance computation. The reader already ranks
> multiple index files for a column (`vector_search_index_raw`), but with
> float-first precedence and no per-query override.

- [ ] **Multiple vector indexes per column + query-time selection** — 🟠 **High**:
  - Allow a column to carry a float `hnsw` (exact / best recall) **and** a
    TurboQuant `hnsw_tq8`/`hnsw_pq` index, chosen per query, e.g.
    `table.vector_search(..., index='tq8')` — exact precision for
    recall-critical reads, the compressed index for latency-sensitive scans.
  - Requires: (1) `add_index` to **append** a vector variant instead of replacing
    the whole family; (2) a query-time selector threaded through
    `vector_search_index`/`vector_search_index_raw`/`vector_search_scored` and
    the Python/SQL surfaces; (3) defining precedence when no selector is given
    (full precision first).

### Theme 3: Overlay Indexes on Read-Only Delta & Hudi (Native Rust Translator)
*Attach the overlay indexes to existing Delta / Hudi tables — writing only indexes, never data.* — 🔴 **High**

> **Current state.** Delta and Hudi tables are Parquet under the hood, and the
> overlay index engine already treats any Parquet file as data. The authoritative
> data files and the advisory index overlays are stored completely separately
> (`s3://my-index-bucket/overlays/`), so the upstream table does not even need to
> know BenoStreamDB exists. The existing
> `Table.register_external(overlay_uri, iceberg_metadata_uri)` path already
> attaches an overlay to an external, read-only Iceberg table without write
> permission on the upstream storage or catalog.
>
> **Approach: a read-side `TableFormat` trait, not a metadata materializer.**
> XTable/UniForm solve a *bidirectional* metadata-sync problem (Delta↔Iceberg↔Hudi
> with write-back); we need one direction, read-only, in-process. So we do **not**
> materialize Iceberg metadata files. Instead, generalize the read path behind a
> `TableFormat` trait that produces the existing `Manifest`/`ManifestEntry`/
> `DeleteFile` structures directly: `IcebergFormat` is today's code, and
> `DeltaFormat`/`HudiFormat` parse the native logs. Nothing is written upstream
> and no Avro manifests are generated — the overlay bucket holds only our
> sidecars. Both existing translators are JVM-based (**Apache XTable** is Java;
> **Delta UniForm** is Scala/Java), so a native Rust implementation avoids a JVM
> runtime dependency entirely.

- [ ] **Extract a read-side `TableFormat` trait** — 🟠 **High**:
  - `load_snapshot()` → data files + delete files + schema + partition spec +
    snapshot id; `read_deletes()`; `partition_transform()`. `IcebergFormat` is a
    lift-and-shift of today's read path (pure refactor, no behaviour change).
- [ ] **`DeltaFormat` (read-only)** — 🟠 **High**:
  - Parse `_delta_log/NNN.json` (newline-delimited `add`/`remove`/`metaData`/
    `protocol` actions) + `.checkpoint.parquet`; merge-on-read via **deletion
    vectors** (Delta 3.x, RoaringBitmap `.bin` — we already have Roaring). The
    quick win: a bounded, well-documented parser.
- [ ] **`HudiFormat` (read-only)** — 🟢 **Deferred**:
  - Parse the `.hoodie/` timeline (commit/deltacommit instants) + base Parquet +
    MOR **log files** (Avro `HoodieLogBlock`). COW first, MOR later; Hudi 1.0's
    timeline format change is a moving target. **The long pole.**
- [ ] **Incremental-on-staleness builds (tail the log, don't `LIST`)** — 🟠 **High**:
  - The upstream writer is external (Databricks/Flink), so we cannot hook its
    stream. A background crawler **tails the latest `_delta_log` entry / timeline
    instant** and compares it to the last-seen version — no full `LIST`, no
    per-file `stat`; steady-state cost is one small `GET` per poll. On change, it
    incrementally builds indexes for the **new files only** (HNSW append-insert +
    BM25 posting merge). Poll interval adapts to the observed change rate
    (backoff + jitter), with an explicit `refresh()` for query-time staleness.
- [ ] **Out-of-band append correctness (the Overlay Invariant)** — 🟠 **High**:
  - Until the background build catches up, the planner searches the existing
    HNSW/BM25 index for the old data and falls back to a Parquet scan for the
    newly appended files — 100% snapshot correctness at all times.
- [ ] **Out-of-core sidecars (mmap)** — 🟢 **Deferred**:
  - Extend the existing `MmapCsrGraph` pattern to the HNSW/BM25 sidecars: mmap
    the graph/postings, keep only the hot top-level (centroids, entry points)
    resident, and page in on demand via `BSDB_DISK_CACHE_DIR`. This is the only
    way a sidecar can exceed host memory.

### Theme 4: Hardware-Accelerated CPU Distance Kernels via SimSIMD (ARM NEON, AVX-512, VNNI) — 🔴 **High**
*Deliver hand-tuned SIMD vector distance execution on ARM64 and modern x86 without manual assembly maintenance.*

> **Current state & Performance Gap.** Vector distance calculations in `src/core/index/distance.rs` dominate HNSW graph traversal and point queries. Today, x86_64 uses manual 256-bit AVX2/FMA intrinsics, but:
> 1. **ARM64 (Apple Silicon, AWS Graviton 3/4, Windows ARM / Snapdragon X Elite)** currently lacks explicit SIMD intrinsics in `distance.rs` and falls back to scalar loop unrolling (`dot_product_portable`), missing out on 1.5×–2.5× throughput gains from hardware vector registers.
> 2. **AVX-512** on modern AMD Zen 4/5 and Intel Xeon/Core processors is unutilized, leaving 512-bit registers (16 floats/instruction) and 32 ZMM registers untapped.
> 3. **Quantized Vector Math**: CPU lacks hardware VNNI (`int8` dot products) and half-precision (`f16`/`bf16`) SIMD instructions, forcing software conversions.
>
> **Approach: Integrate Unum's `simsimd` crate.** `simsimd` provides production-hardened, zero-dependency C/assembly kernels with automatic runtime CPU dispatching across AVX2, AVX-512, AVX-512-FP16, ARM NEON, and ARM SVE. Integrating `simsimd` accelerates CPU-bound HNSW traversals across all platforms while eliminating ~400 lines of fragile `unsafe` architecture-specific intrinsics.

- [ ] **Integrate `simsimd` dependency into `benostreamdb` and `benostream-gpu-ann`** — 🟠 **High**:
  - Add `simsimd` (Apache-2.0, pure Rust bindings with compiled SIMD kernels) to workspace dependencies.
- [ ] **Wire runtime dispatch in `src/core/index/distance.rs`** — 🟠 **High**:
  - Replace manual `dot_product_avx2`, `l2_distance_squared_avx2`, and `dot_product_portable` with `simsimd::spatial` / `simsimd::binary` dispatches.
  - Automatically route L2, Inner Product (Dot), Cosine, Hamming, and Jaccard distances through hardware-detected kernels.
- [ ] **Native Quantized CPU Distance Acceleration (`i8` / `f16` / `bf16`)** — 🟠 **High**:
  - Expose hardware-accelerated integer dot products (`VNNI` / `ARM SDOT`) and `f16` distance kernels for TurboQuant / quantized vector scans on CPU.
- [ ] **Multi-Architecture Benchmarking** — 🟡 **Medium**:
  - Measure search latency improvements across Apple Silicon M-series (NEON), AWS Graviton 4 (SVE), AMD Zen 4/5 (AVX-512), and Windows on ARM.

### Theme 5: GPU Scaling (single-query multi-GPU + cross-call batching) — 🟠 **Medium**

> **Shipped in 0.12.0.** The multi-GPU device pool
> (`gpu::set_gpu_device_pool`, wired into Trino/Spark) round-robins devices
> across **worker threads**, and each kernel call is already batched (one
> H2D → launch → D2H over the whole vector set, with a 50k floor:
> `GPU_DISPATCH_THRESHOLD`).
>
> **Remaining.** The round-robin only spreads work across **concurrent**
> dispatches — a single large query still runs on one device
> (`HnswIvfIndex::search` makes one synchronous `compute_distance` call), and
> there is no overlap of transfer with compute across calls.

- [ ] **Single-query multi-GPU sharding** — shard the candidate set (e.g.
  `HnswIvfIndex::search`'s centroid / distance batch) across the devices in the
  pool, launch concurrently on each, and merge top-k. This is what turns the
  pool from a concurrency-only feature into a genuine per-query speedup.
- [ ] **Cross-call batching + streams** — overlap H2D / compute / D2H across
  successive batches with CUDA streams instead of one synchronous round trip
  per call; pipeline rather than block.
- [ ] **Avoid redundant transfers** — keep vectors/centroids device-resident and
  cache the uploaded query. Today each call re-uploads the query and copies the
  whole batch to host (`htod_copy(vectors.to_vec())`).
- [ ] **Adaptive dispatch threshold** — `GPU_DISPATCH_THRESHOLD` (50k) is a
  coarse floor; make it adaptive to the measured launch/transfer overhead so
  mid-size batches can still benefit without paying H2D/D2H for nothing.

### Theme 6: AI Agent Tools & Client Ecosystem
*Native interfaces for AI frameworks and autonomous agent runtimes.* — 🟠 **Medium**

- [ ] **LangChain & LlamaIndex Partner Integrations** — 🟢 **Deferred**:
  - `langchain-benostreamdb`: LangChain-compatible `VectorStore` and `Retriever` implementations with metadata filtering.
  - `llama-index-vector-stores-benostreamdb`: LlamaIndex vector store index adapter.
- [ ] **Broker Event Adapters** — 🟢 **Deferred**:
  - Pluggable Kafka, RabbitMQ, and NATS consumers for automated event-driven
    lakehouse ingestion.
  - **Placement: a new `contrib/benostreamdb-ingest` crate, not the core.** The
    publish/subscribe seam already lives in the core — the
    [`WorkCoordinator` trait](../src/core/table/coordinator.rs:89) (with
    `WorkUnit` / `Lease`), its only implementation
    [`ObjectStoreCoordinator`](../src/core/table/coordinator.rs:112), and the
    claim → read → write → commit loop in
    [`ingest_with_coordinator`](../src/core/table/ingest.rs:304). A broker
    adapter is therefore just another `impl WorkCoordinator`, so **no new core
    API is required**:
    ```
    contrib/benostreamdb-ingest/
      Cargo.toml            # [[lib]] + [[bin]] bsdb-ingest; features = kafka/nats/rabbitmq
      src/kafka.rs          # KafkaWorkCoordinator    : impl WorkCoordinator
      src/nats.rs           # NatsWorkCoordinator     : impl WorkCoordinator
      src/rabbitmq.rs       # RabbitMqWorkCoordinator : impl WorkCoordinator
      src/main.rs           # bsdb-ingest daemon → Table::ingest_with_coordinator(..)
      tests/
    ```
  - **Why `contrib/` and not core:** the broker clients are heavy and
    platform-sensitive (`rdkafka`/`librdkafka` is a C dependency, plus `lapin`
    and `async-nats`); pulling them into the default engine build would regress
    build time, binary size, and cross-platform reach. This mirrors the existing
    boundary — core = engine + traits, `contrib/` = deployment surfaces
    (`bsdb-search`, `bsdb-mcp`) — with `bsdb-ingest` as the third column.
  - **Semantics to settle at implementation:** a broker *consumer* is a source of
    work, whereas `WorkCoordinator` is the queue/lease authority. Either
    implement the trait directly against the broker's consumer-group offsets
    (broker is the queue), or keep `ObjectStoreCoordinator` as the lease
    authority and add a broker → queue enqueue bridge.
  - **Config:** follow the `BSDB_` / `BSDB_SEARCH_` convention →
    `BSDB_INGEST_*` (brokers, topics, group id, lease TTL, coordinator backend
    selection: `object-store` | `kafka` | `nats`).
  - **Deploy / docs:** a `deploy/` compose entry (Kafka/NATS alongside the
    RustFS/Nessie stack) and `docs/integrations/INGEST.md` + its
    `docs/source/guides/` copy, consistent with SEARCH_GATEWAY and MCP.
- [ ] **Named embedding profiles and per-provider credentials** — 🟠 **Medium**:
  - Add `embedding.profiles` (name → provider/model/URL/dimension, with
    `api_key_env` references) and an optional `benostream.embedding.profile`
    table property.
  - Resolve secrets at runtime from the environment or a secret manager; never
    store API keys in table metadata. This lets one MCP process serve multiple
    HTTP providers without a raw key dictionary, while preserving the
    table-scoped model/dimension contract.
  - Keep the current single-config behavior as the default profile for backward
    compatibility.
- [ ] **Per-session MCP scratch namespace + TTL** — derive the agent scratch
  schema from the authenticated subject (JWT `sub` / API-key id, sanitized to
  `scratch_<subject>`) and give ephemeral tables a TTL + background GC, so two
  agents never share a namespace. Today `scratch` is a single shared schema.
- [ ] **Policy tooling & observability** — a `bsdb policy show|set|lint` CLI
  (lint flags schemas present in storage but absent from the policy), an
  effective-permissions read endpoint, and a
  `benostream_policy_denials_total{schema,action}` metric so over-restriction is
  visible instead of looking like missing data.

### Theme 7: Graph Overlays
*Elevating knowledge graphs into a first-class lakehouse data modeling primitive.* — 🟠 **Medium**

> **Partially delivered.** The programmatic edge-table API has shipped:
> `Table.create_edge_table(...)`, `Table.from_networkx(...)`, and
> `Table.to_networkx(...)` create and round-trip standard edge tables with
> automatic endpoint/embedding sidecar indexes. The graph UDAFs
> (`graph_neighbors`, `graph_pagerank`, `graph_shortest_path`, …) are now
> reachable from every connector surface — including Spark, via the pass-through
> reader. Typed global URNs below remain open.

- [ ] **Typed Global Entity URNs (`table:id`)** — 🟢 **Deferred**:
  - Extend the CSR graph dictionary to support composite entity identifiers (e.g. `users:101`, `orders:5002`) so a single CSR graph overlay can seamlessly traverse across heterogeneous Iceberg tables.

### Theme 8: Semi-Structured & JSON
*First-class access to JSON documents without forcing a rigid column schema.* — 🟢 **Deferred**

> **Current state.** JSON is stored as a `Utf8` string column (text, not a
> decomposed binary form, so the functions use PostgreSQL's `json_*` names, not
> `jsonb_*`). The search gateway
> (`contrib/benostreamdb-search/src/infer.rs`) already infers an Arrow schema
> from sample documents (ES 7.10 dynamic mapping).

- [ ] **Iceberg v3 `variant` Mapping** — 🟢 **Deferred**:
  - Map semi-structured columns to Iceberg v3 `variant` (the spec-native
    semi-structured type) so JSON round-trips through catalogs instead of being
    a bespoke engine-only type; Arrow-side representation is `Struct`/`Utf8`
    with path pushdown. This is a storage-fidelity improvement — it still needs
    the JSON-path index above to make containment fast.

### Theme 9: Reactive Lakehouse Streaming
*Bringing change data capture (CDC) and reactive streaming to the lakehouse.* — 🟢 **Deferred**

- [ ] **Arrow Flight SQL Streaming Subscriptions** — 🟢 **Deferred**:
  - Implement Flight `DoExchange` / `DoGet` streaming endpoints allowing remote clients (Python, DuckDB, web dashboards) to subscribe to table diffs in real time without polling. Build this once somebody actually needs remote subscriptions.

### Theme 10: Domain-Specific Text Analyzers for BM25
*Specialized tokenization pipelines for enterprise code, legal, and multilingual knowledge bases.* — 🟢 **Deferred**

- [ ] **Configurable Table Analyzer Configurations**:
  - Allow table schemas to declare custom text analyzers in metadata (`analyzer = 'code_camelcase'`, `stopwords = 'en'`).
- [ ] **Built-in Tokenizer Suite**:
  - **Code Tokenizer**: Splits on CamelCase, snake_case, and symbol delimiters for source code search.
  - **Multilingual Stemming**: Integration of Snowball stemmers for English, Spanish, French, and German.
  - **CJK Segmentation**: Character/bi-gram tokenization for Chinese, Japanese, and Korean corpora.

---

## Benchmarks as Product Work

Benchmarks and demonstrations are part of the product, not marketing polish.
The Launch Candidate requires **excellent end-to-end benchmarks** and
**documentation/tutorials** that show the three workflows concretely. Both have
shipped — see **Key Completed Milestones**.