Welcome to BenoStreamDB¶
BenoStreamDB is an index-overlay engine for the lakehouse built in Rust. It layers reconstructible, persistent secondary indexes—HNSW vector search, BM25 Okapi full-text, Roaring Bitmaps, JSON path filters, and CSR graphs—directly onto Apache Iceberg tables in object storage.
Tip
Looking for the fastest way to get started? Jump straight to the 2-Minute Quickstart to run your first vector search and SQL query in under 2 minutes.
Key Architecture Highlights¶
Serverless by Default: Run directly against object storage via an embedded library (Python, Rust, JVM) with zero database server overhead.
Advisory Index Overlays: Build rebuildable secondary indexes alongside standard Parquet data files. The underlying data is never rewritten or locked into a proprietary format.
Vector Search & pgvector SQL: Query vector indexes directly using standard pgvector SQL operators (
<->,<=>,<#>) in DataFusion.Graph Analytics & Graph RAG: Query CSR graph index overlays using native SQL table functions directly over Iceberg edge tables.
Ecosystem Connectors: Native connectors for Apache Spark, Trino, and dbt, plus optional Arrow Flight SQL and REST search gateways.
Grounded Hardware Support: Dynamic batch GPU acceleration via NVIDIA CUDA (runtime NVRTC), Apple Metal, and Vulkan/WGPU, with SIMD-optimized CPU execution as the fast, low-latency default.
Getting Started
- 2-Minute Quickstart
- ⚡ 1. Install (15 seconds)
- 🐍 2. Embedded Python: Indexes & Vector Search (45 seconds)
- 🔀 3. Hybrid Search: BM25 + Vector Fusion (20 seconds)
- 📊 4. Query with SQL & pgvector Syntax (30 seconds)
- 🏎️ 5. 4x Index Compression with TurboQuant™ (15 seconds)
- 🐳 Docker Gateway: OpenSearch & Qdrant APIs
- 🧭 Next Steps
- Installation
- 🔄 Fluent Query API
- Python Fluent API
- Rust Fluent API
- Benefits
- Python Vector Distance API with GPU Acceleration
- 📊 Real-World Testing Plan
- Phase 2: Nessie Integration (Next)
- Phase 3: Production Hardening ✅ COMPLETE
- 🏗️ Architecture
- 🔌 Connectors
- 🔨 Building Connectors
- 🧪 Development
- 📈 Roadmap
- 🤝 Contributing
- 📄 License
- 🙏 Acknowledgments
- Quickstart
- 1. The
bsdb-searchREST server - 2. The Python client
- Next steps
- Architecture: The Indexed Lakehouse
- ADR-001: Tightened Overlay Index Artifact Contract & Compound Puffin Bundles
- 1. Executive Summary
- 2. Problem Statement & Motivation
- 3. The Tightened Index Artifact Specification
- 4. Compound Index Bundles (Apache Iceberg Puffin Specification)
- 5. Detached Overlay Catalogs for External Iceberg Tables
- 6. Merge-On-Read (MOR) Lifecycle for Secondary Indexes
- 7. Migration & Compatibility Plan
- 8. Alternatives Considered
- 9. Observability & Telemetry
- 10. Security Considerations
- Appendix A: Implementation Cross-Reference Map
Connectors & Integrations
Table Formats & Catalogs
- Iceberg Compatibility Matrix
- Iceberg V2/V3 API Documentation
- Multi-Catalog Usage Guide
- Supported Catalogs
- 1. Hive Metastore (Detailed Example)
- 2. Project Nessie
- 3. AWS Glue Catalog
- 4. Iceberg REST Catalog (Snowflake / Apache Polaris & Lakekeeper)
- Next Steps
- Snowflake Polaris
- Architecture Overview
- Prerequisites
- Step 1: Configure BenoStreamDB to Sync with Polaris
- Step 2: Configure Storage Access in Snowflake (External Volume)
- Step 3: Configure Polaris Catalog Integration in Snowflake
- Step 4: Create the Table in Snowflake
- Step 5: Ingest via BenoStreamDB and Query in Snowflake
- Key Benefits
Query & Search
- pgvector SQL Support Guide
- Python Vector Distance API Reference
- Overview
- Supported Distance Metrics
- GPU Backend Support
- Installation
- Quick Start
- API Reference
- Performance Optimization
- Integration with SQL
- Troubleshooting
- Examples
- See Also
- API reference
- Python API
- Spark Connector
- Trino Connector
- Configuration
- REST Search APIs (
benostreamdb-search) - Error Handling
- Performance Tips
- See Also
- Graph RAG & Iceberg Edge Tables Guide
- 1. Standard Edge Table Schema Convention
- 2. Automated Sidecar Indexing
- 3. NetworkX Interoperability
- 4. End-to-End Hybrid Graph RAG Workflow
- 5. Iceberg Partitioning & Optimization Best Practices
- 6. End-to-End Graph RAG Search & Community Summarization
- 7. Analytics Engineering with dbt (
dbt-benostreamdb)
- GPU Setup Guide for BenoStreamDB
Operations & Hardening
Tutorials
Benchmarks & Evaluation
- Benchmarking: how to reproduce every number
- BenoStreamDB Benchmarking Plan
- 1. Goals and non-goals
- 2. Datasets (external, standard, citable)
- 3. Competitor set
- 4. Harnesses
- 5. Resource-envelope rules
- 6. Metrics and output schema
- 7. Production workload classes
- 8. Methodology and reproducibility
- 9. Reporting format
- 10. Automation and CI
- 11. Known findings and current status
- 12. Roadmap
- BenoStreamDB — Release Readiness Assessment
API Reference