Arrow Flight SQL Gateway (`benostreamdb-flight`)¶

benostreamdb-flight provides a high-performance Apache Arrow Flight and Flight SQL network interface over BenoStreamDB.

Architecture & Deployment Model¶

BenoStreamDB is serverless-first: applications with native language support (Python, Rust, JVM via Spark/Trino) embed the engine directly in-process without needing a database server.

For network clients, polyglot applications (Go, C++, Node.js), and BI tools that communicate via JDBC/ODBC/ADBC, the optional Flight SQL server exposes the engine over standard gRPC.

Architecture Note: Arrow Flight SQL currently provides single-process server access. Distributed coordinator scheduling or multi-node clustered execution is not implied by Flight SQL support.

Client Application (JDBC / ODBC / ADBC / Go / C++ / BI tools)
                    │
                    ▼
         Arrow Flight SQL (gRPC / HTTP/2)
                    │
                    ▼
         BenoStreamFlightSqlService
                    │
                    ▼
            BenoStreamSession
                    │
                    ▼
       DataFusion Query Engine + BenoStreamDB Core

⚠️ Security & Authentication¶

Stateless Authentication: The server supports stateless authentication via static API keys (BSDB_API_KEY) and JWT tokens (HS256 shared secret or RS256 IdP public keys). Tokens are transmitted via the gRPC authorization: Bearer <token> metadata header.
No Native TLS: The server binary does not terminate TLS natively. When exposed across network boundaries, deploy behind a reverse proxy (e.g., Envoy, NGINX, AWS ALB) that handles TLS termination and certificates.
Binding: Defaults to 127.0.0.1:50051. For hardened environments, set BSDB_AUTH_REQUIRED=true to fail closed if unauthenticated requests arrive.

Configuration¶

Configure the server via environment variables:

Variable	Description	Default
`BSDB_FLIGHT_BIND`	Bind address for gRPC listener	`127.0.0.1`
`BSDB_FLIGHT_PORT`	Port for Flight SQL listener	`50051`
`BSDB_FLIGHT_MAX_ROWS`	Row cap applied to Arrow Flight result sets	`100000`
`BSDB_FLIGHT_READ_ONLY`	Refuse DDL, DML, and ingest. Set to `1`/`true`	Unset (writes enabled)
`BSDB_WAREHOUSE`	Default storage warehouse for created tables	`./data`
`BSDB_CONFIG`	Path to catalog configuration TOML	Unset
`BSDB_API_KEY`	Shared secret API key for client auth	Unset
`BSDB_JWT_SECRET`	Secret for HMAC-SHA256 JWT tokens	Unset
`BSDB_AUTH_REQUIRED`	Fail closed on unauthenticated requests	`false`
`BSDB_METRICS_BIND`	HTTP metrics listener address	`127.0.0.1`
`BSDB_METRICS_PORT`	HTTP Prometheus metrics port	`9090`
`BSDB_GPU_DEVICE`	GPU device or comma-separated pool (`cuda:0,cuda:1`); pool assigns devices per worker thread	unset (`auto`)

Running the Server¶

Build and start the server:

cargo run --release -p benostreamdb-flight

Or run via Docker:

docker run -p 50051:50051 -p 9090:9090 \
  -e BSDB_API_KEY="your-secret-key" \
  -e BSDB_WAREHOUSE="s3://my-bucket/warehouse" \
  benostreamdb-flight:latest

Client Usage Examples¶

1. Python via ADBC (Arrow Database Connectivity)¶

import pyarrow.flight as flight
from adbc_driver_flightsql import dbapi

# Connect with Bearer authentication
conn = dbapi.connect(
    uri="grpc://127.0.0.1:50051",
    db_kwargs={
        "adbc.flight.sql.rpc.call_header.authorization": "Bearer your-secret-key"
    }
)

with conn.cursor() as cur:
    cur.execute("SELECT id, title FROM documents ORDER BY id LIMIT 10")
    results = cur.fetchallarrow()
    print(results)

2. DuckDB¶

INSTALL flight;
LOAD flight;

-- Query BenoStreamDB via Flight SQL
SELECT * FROM flight_sql_query('grpc://127.0.0.1:50051', 'SELECT * FROM documents');

3. BI & SQL Tools (DBeaver, Tableau, Superset)¶

Use the standard Apache Arrow Flight SQL JDBC driver. Configure the connection URL as:

jdbc:arrow-flight-sql://127.0.0.1:50051?useEncryption=false

Arrow Flight SQL Gateway (benostreamdb-flight)¶