GPU Setup Guide for BenoStreamDB¶

This guide provides detailed instructions for setting up GPU acceleration for vector operations in BenoStreamDB.

Overview¶

BenoStreamDB supports GPU acceleration for vector distance computations across multiple hardware backends:

  • NVIDIA CUDA - For NVIDIA GPUs (GeForce, Quadro, Tesla, RTX 20/30/40/50 series, Blackwell; tested with CUDA 11, 12, and 13)

  • Apple Metal (MPS) - For Apple Silicon Macs (M1–M5, verified)

  • Intel XPU / Integrated GPU - For Intel integrated (Iris Xe) and discrete GPUs via WGPU (tested and verified)

  • AMD ROCm / Vulkan - For AMD Radeon and Instinct GPUs (architecturally supported via WGPU; physical hardware testers welcome!)

GPU acceleration offloads compute-heavy batch vector distance kernels to hardware accelerators.

Important Workload Guidance:

  • Single-Query Searches (Batch = 1): For individual point queries and sequential HNSW graph traversals, CPU execution (with AVX2 / AVX-512 / ARM NEON SIMD) is generally faster than GPU. Fixed host-to-device PCIe transfer latencies and kernel launch overheads dominate single-vector calculations on datasets under 100k vectors.

  • Batch-Parallel Workloads: GPU acceleration is primarily advantageous for dense batch operations (e.g. evaluating large query batches simultaneously against high-dimensional vector matrices, or brute-force flat scoring) where massive parallelism outweighs PCIe transfer latency.

  • Future Full-GPU Workloads: End-to-end resident GPU vector search (keeping index graphs and candidate queues on-device) is on the roadmap to enable GPU query speedups for live lakehouse tables.

Installation¶

Unified Binary (PyPI)¶

BenoStreamDB provides a unified binary package with support for major compute backends. Runtime detection automatically activates available hardware drivers, safely falling back to CPU if drivers are unavailable.

pip install benostreamdb

Hardware Requirements:

  • NVIDIA: Requires NVIDIA driver (libcuda.so on Linux, nvcuda.dll on Windows).

  • AMD: Requires ROCm/Vulkan drivers.

  • Intel: Requires Level Zero/Vulkan drivers.

  • Apple: Requires macOS 12.3+ (Built-in Metal).

import benostreamdb as bsdb

# Auto-detect and use best available compute backend
device = bsdb.Device("auto")
print(f"Using backend: {device.backend}")

# Pick a specific backend (Torch-aligned strings)
device = bsdb.Device("cuda")    # NVIDIA or AMD ROCm (Torch standard)
device = bsdb.Device("xpu")     # Intel XPU (Torch standard)
device = bsdb.Device("mps")     # Apple Silicon Metal
device = bsdb.Device("cpu")     # CPU fallback (always available)

# Check availability
print(bsdb.Device.is_available("cuda"))  # True if NVIDIA or AMD ROCm present
print(bsdb.Device.is_available("xpu"))   # True if Intel hardware present

NVIDIA CUDA Setup¶

Requirements¶

  • GPU: NVIDIA GPU with compute capability 6.0 or higher

    • Pascal (GTX 10 series) or newer

    • Recommended: RTX 20/30/40/50 series, A100, H100, B100/B200

  • Driver: NVIDIA driver 450.80.02 or later (550+ recommended for RTX 50 series / Blackwell)

  • CUDA Runtime & Toolkits: Supports CUDA 11.x, 12.x, and 13.x (CUDA 13 verified and tested locally). Runtime discovery links against libcuda.so (Linux) or nvcuda.dll (Windows) provided by the driver.

Note on NVRTC JIT Compilation: BenoStreamDB compiles CUDA kernels at runtime via NVRTC (Just-In-Time). The CUDA Toolkit and nvcc compiler are not required for building or running. BenoStreamDB features custom, version-agnostic NVRTC discovery that automatically resolves libnvrtc across CUDA 11, CUDA 12, and CUDA 13 (including pip wheels like nvidia-cuda-nvrtc-cu12 or nvidia-cuda-nvrtc-cu13).

Supported GPUs¶

Series

Compute Capability

Supported

GTX 10 series (Pascal)

6.0-6.1

✅ Yes

GTX 16 series (Turing)

7.5

✅ Yes

RTX 20 series (Turing)

7.5

✅ Yes

RTX 30 series (Ampere)

8.6

✅ Yes

RTX 40 series (Ada)

8.9

✅ Yes

RTX 50 series (Blackwell)

10.0 / 12.0

✅ Yes

Tesla V100

7.0

✅ Yes

A100

8.0

✅ Yes

H100

9.0

✅ Yes

B100 / B200 (Blackwell)

10.0

✅ Yes

Installation on Linux (Ubuntu/Debian)¶

# Install NVIDIA driver (provides libcuda.so at runtime)
sudo apt-get update
sudo apt-get install nvidia-driver-535

# Verify driver installation
nvidia-smi

No CUDA Toolkit installation required. The NVIDIA driver supplies libcuda.so at runtime; kernels are JIT-compiled via NVRTC.

Building from Source with CUDA Support¶

CUDA is an optional compile-time feature. Enable it when building:

# Rust library
cargo build --release --features cuda

# Python bindings
maturin develop --features cuda

Without --features cuda, the CUDA backend will be unavailable at runtime (other backends like ROCm, Metal, and XPU remain functional).

Installation on Windows¶

Recommended: WSL2 (Windows Subsystem for Linux) Windows users should use WSL2 to run BenoStreamDB with GPU support:

  1. Install WSL2 and Ubuntu (e.g., wsl --install -d Ubuntu-22.04)

  2. Install NVIDIA Windows Driver (this provides the necessary kernel-mode interface for WSL2)

  3. Within the WSL2 Ubuntu environment, follow the Linux installation instructions above.

  4. Verify from within WSL:

    nvidia-smi

Native Windows (outside WSL2): We build and package native Windows binaries with full architectural support for CUDA via dynamic linking to nvcuda.dll and runtime NVRTC JIT compilation. However, because our automated CI test suite currently runs on Linux, native Windows GPU execution is currently untested in CI.

If you are running BenoStreamDB natively on Windows with an NVIDIA GPU, we would love your help: please test your workload and report your results or open an issue to let us know how it runs on your machine. For production workflows today, WSL2 remains our tested and recommended environment.

Verification¶

import benostreamdb as bsdb
import numpy as np

# Create CUDA context
ctx = bsdb.ComputeContext("cuda")
print(f"CUDA backend initialized: {ctx.backend}")

# Test GPU computation
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(10000, 768).astype(np.float32)
distances = bsdb.l2_batch(query, database, device=ctx)
print(f"Computed {len(distances)} distances on GPU")

# Check performance stats
stats = ctx.get_stats()
print(f"GPU time: {stats['total_gpu_time_ms']}ms")

AMD ROCm Setup¶

Requirements¶

  • GPU: AMD Radeon RX 5000 series or newer (RDNA 1, 2, 3), or Instinct MI series.

  • OS: Linux (Primary support for compute workloads).

  • Backend: BenoStreamDB uses WGPU/Vulkan for AMD compute, ensuring compatibility across a wide range of Linux distributions.

Hardware Testing Status (Help Wanted): BenoStreamDB targets AMD GPUs via its cross-platform WGPU / Vulkan compute pipeline. However, our internal testing environment has validated WGPU primarily on Intel integrated graphics; we currently lack physical AMD discrete GPUs (RX 6000/7000 series or Instinct) in our regular CI test harness. While the WGSL distance shaders compile and run against standard Vulkan compute targets, AMD execution is currently untested on real hardware.

If you have an AMD GPU with ROCm or Vulkan on Linux, we would love your help! Please run the verification snippet below and report your benchmark results or open an issue.

Target GPUs¶

Series

Architecture

Testing Status

RX 5000 series

RDNA 1

🔬 Architecture supported (Untested on hardware — Help Wanted)

RX 6000 series

RDNA 2

🔬 Architecture supported (Untested on hardware — Help Wanted)

RX 7000 series

RDNA 3

🔬 Architecture supported (Untested on hardware — Help Wanted)

Radeon VII

GCN 5.1

🔬 Architecture supported (Untested on hardware — Help Wanted)

MI100

CDNA 1

🔬 Architecture supported (Untested on hardware — Help Wanted)

MI200 series

CDNA 2

🔬 Architecture supported (Untested on hardware — Help Wanted)

Installation on Ubuntu¶

# Download and install AMD GPU driver installer
wget https://repo.radeon.com/amdgpu-install/latest/ubuntu/noble/amdgpu-install_6.0.2-1_all.deb
sudo apt install ./amdgpu-install_6.0.2-1_all.deb

# Install ROCm
sudo amdgpu-install --usecase=rocm

# Add user to video and render groups
sudo usermod -a -G video,render $USER

# Reboot to apply changes
sudo reboot

# Verify installation
rocm-smi
rocminfo

Installation on RHEL/CentOS¶

# Add ROCm repository
sudo tee /etc/yum.repos.d/rocm.repo <<EOF
[ROCm]
name=ROCm
baseurl=https://repo.radeon.com/rocm/rhel8/rpm
enabled=1
gpgcheck=1
gpgkey=https://repo.radeon.com/rocm/rocm.gpg.key
EOF

# Install ROCm
sudo yum install rocm-hip-sdk

# Add user to video and render groups
sudo usermod -a -G video,render $USER

# Reboot
sudo reboot

Verification¶

import benostreamdb as bsdb

# Create ROCm context
ctx = bsdb.ComputeContext("rocm")
print(f"ROCm backend initialized: {ctx.backend}")

# Test computation
import numpy as np
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(10000, 768).astype(np.float32)
distances = bsdb.l2_batch(query, database, device=ctx)
print(f"Computed {len(distances)} distances on AMD GPU")

Apple Metal (MPS) Setup¶

Requirements¶

  • Hardware: Apple Silicon Mac (M1, M2, M3, M4, M5, or newer)

  • OS: macOS 12.3 (Monterey) or later

  • No additional installation required - Metal is built into macOS

Supported Devices¶

Device

Chip

Supported

MacBook Air (2020+)

M1/M2/M3/M4/M5

✅ Yes

MacBook Pro (2020+)

M1/M2/M3/M4/M5 Pro/Max/Ultra

✅ Yes

Mac mini (2020+)

M1/M2/M2 Pro

✅ Yes

Mac Studio

M1/M2 Max/Ultra

✅ Yes

iMac (2021+)

M1/M3/M4/M5

✅ Yes

Mac Pro (2023+)

M2 Ultra

✅ Yes

Verification¶

import benostreamdb as bsdb

# Auto-detect should find Metal on Apple Silicon
ctx = bsdb.ComputeContext.auto_detect()
print(f"Backend: {ctx.backend}")  # Should show "mps"

# Or explicitly create Metal context
ctx = bsdb.ComputeContext("mps")
print(f"Metal backend initialized")

# Test computation
import numpy as np
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(10000, 768).astype(np.float32)
distances = bsdb.l2_batch(query, database, device=ctx)
print(f"Computed {len(distances)} distances on Apple GPU")

Performance Notes¶

  • Apple Silicon uses unified memory (shared between CPU and GPU)

  • No explicit memory transfer overhead

  • M1 Max/Ultra and M2 Max/Ultra have more GPU cores for better performance

  • Recommended for databases up to available system memory

Intel XPU Setup¶

Requirements¶

  • GPU: Intel Iris Xe (tested and verified), Arc A-Series (Alchemist), Arc B-Series (Battlemage), or Data Center GPU Max.

  • Testing Status: Tested and validated on Intel integrated graphics (Iris Xe) via WGPU/Vulkan.

  • OS: Linux (Native) or Windows via WSL2.

  • Drivers: Requires Level Zero and Vulkan user-mode drivers.

Installation on Linux (Ubuntu/Debian)¶

For Intel hardware, you must install the compute and media runtimes to enable WGPU acceleration.

# Add Intel graphics repository (Noble 24.04 instructions)
wget -qO - https://repositories.intel.com/graphics/intel-graphics.key | sudo gpg --dearmor --yes -o /usr/share/keyrings/intel-graphics.gpg
echo "deb [arch=amd64,i386 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/graphics/ubuntu noble main" | sudo tee /etc/apt/sources.list.d/intel-graphics.list
sudo apt update

# Install Level Zero and Media runtimes
sudo apt install intel-level-zero-gpu intel-media-va-driver-non-free

# Verify installation
vulkaninfo | grep "vendorID = 0x8086"

Verification (Linux)¶

# Verify Vulkan/WGPU installation
vulkaninfo | grep vendor
# Or check adapter listing in BenoStreamDB
python -c "import benostreamdb as bsdb; print(bsdb.Device.list_available_backends())"

Installation on Windows (via WSL2)¶

Windows users should ensure they have the latest Intel Graphics drivers installed on the host. These provide Vulkan support to WSL2, enabling BenoStreamDB to detect and use the GPU via WGPU.

Verification¶

import benostreamdb as bsdb

# Create XPU (Intel) context
device = bsdb.Device("xpu")
print(f"Intel backend initialized: {device.backend}")

# Test computation
import numpy as np
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(10000, 768).astype(np.float32)
distances = bsdb.l2_batch(query, database, device=device)
print(f"Success: Computed on {device.backend}")

Multi-GPU Systems¶

For systems with multiple GPUs, you can pin one device or install a pool.

Pin a specific device:

import benostreamdb as bsdb

# List available backends
ctx = bsdb.ComputeContext.auto_detect()
print(f"Available backends: {ctx.list_available_backends()}")

# Use specific GPU device
ctx = bsdb.ComputeContext("cuda", index=0)  # First GPU
ctx = bsdb.ComputeContext("cuda", index=1)  # Second GPU

# Check which device is being used
print(f"Using device: {ctx.device_id}")

Install a process-wide pool so worker threads spread across devices:

# Each worker thread that first asks for a context gets cuda:0 or cuda:1
bsdb.set_gpu_device_pool("cuda:0,cuda:1")
print(bsdb.gpu_device_pool())   # ['cuda:0', 'cuda:1']

# Clear the pool and fall back to the process-wide device
bsdb.set_gpu_device_pool("")

For Flight SQL, MCP, and other server processes, use the same list:

BSDB_GPU_DEVICE="cuda:0,cuda:1" benostreamdb-flight

The pool is concurrency scaling, not single-query sharding: each worker thread is assigned one device, but one HNSW search call still runs on a single GPU. Use separate processes or threads pinned to different devices when you need both GPUs busy with independent searches.

Troubleshooting¶

GPU Not Detected¶

Symptom: auto_detect() returns CPU backend

Solutions:

  1. Verify GPU drivers are installed:

    • NVIDIA: nvidia-smi

    • AMD: rocm-smi or vulkaninfo

    • Intel: vulkaninfo (Check for vendor 0x8086)

    • Apple: Check System Settings → Hardware

  2. Check backend availability:

    ctx = bsdb.ComputeContext.auto_detect()
print(ctx.list_available_backends())

  3. Try creating backend explicitly:

    try:
    ctx = bsdb.ComputeContext("cuda")
except RuntimeError as e:
    print(f"CUDA not available: {e}")

Out of Memory Errors¶

Symptom: RuntimeError: GPU out of memory

Solutions:

  1. Process database in chunks:

    chunk_size = 10000
all_distances = []
for i in range(0, len(database), chunk_size):
    chunk = database[i:i+chunk_size]
    distances = bsdb.l2_batch(query, chunk, device=ctx)
    all_distances.append(distances)
all_distances = np.concatenate(all_distances)

  2. Use smaller data types (float32 instead of float64)

  3. Use sparse vectors for sparse data

  4. Use binary vectors for binary features

Slow Performance¶

Symptom: GPU is slower than CPU

Possible causes:

  1. Small batch size: GPU overhead dominates for < 1,000 vectors

    • Solution: Use CPU for small batches, GPU for large batches

  2. Data type mismatch: Using float64 instead of float32

    • Solution: Convert to float32: data.astype(np.float32)

  3. Memory transfer overhead: Creating new context each time

    • Solution: Reuse GPU context across multiple operations

  4. Wrong backend: Using Intel XPU path on NVIDIA GPU

    • Solution: Use CUDA for NVIDIA, ROCm for AMD

Driver Version Mismatch¶

Symptom: RuntimeError: CUDA driver version is insufficient

Solution:

# Check current driver version
nvidia-smi

# Update NVIDIA driver (Ubuntu)
sudo apt-get update
sudo apt-get install --only-upgrade nvidia-driver-535

# Or install latest driver
sudo ubuntu-drivers autoinstall

Performance & Workload Characteristics¶

In empirical competitor benchmarks, GPU speedup is workload-dependent:

  • Point Queries & HNSW Search: Single-query ANN lookups (sequential graph walks) are dominated by pointer traversals and latency rather than arithmetic throughput. Under single-stream query workloads (batch = 1), optimized CPU SIMD execution (AVX2/AVX-512) delivers lower p50 latency and higher QPS than GPU due to PCIe transfer and kernel dispatch overheads.

  • Dense Batch Matrix Operations: Standalone distance kernels (l2_batch, cosine_batch) running large batches ($N \ge 100,000$ vectors, batch size $\ge 1,000$) effectively saturate GPU warps and achieve high throughput.

  • Index Construction: Background IVF clustering and k-means centroid assignments benefit from GPU acceleration during offline index generation.

  • Full GPU Workloads (Roadmap): True query speedups for lakehouse vector search require resident GPU index caching and batched search queues, which are actively tracked on the roadmap.

See the Benchmarking Guide for reproducible competitor comparison numbers under identical container envelopes.

Best Practices¶

  1. Reuse Device across multiple operations

  2. Use float32 instead of float64 for better GPU performance

  3. Batch operations when possible (process multiple queries together)

  4. Profile your workload

  5. Choose appropriate backend for your hardware

  6. Monitor GPU memory usage for large databases

  7. Use sparse/binary vectors when applicable to reduce memory

See Also¶