# GPU Setup Guide for BenoStreamDB

This guide provides detailed instructions for setting up GPU acceleration for vector operations in BenoStreamDB.

## Overview

BenoStreamDB supports GPU acceleration for vector distance computations across multiple hardware backends:

- **NVIDIA CUDA** - For NVIDIA GPUs (GeForce, Quadro, Tesla, RTX 20/30/40/50 series, Blackwell; tested with CUDA 11, 12, and 13)
- **Apple Metal (MPS)** - For Apple Silicon Macs (M1–M5, verified)
- **Intel XPU / Integrated GPU** - For Intel integrated (Iris Xe) and discrete GPUs via WGPU (tested and verified)
- **AMD ROCm / Vulkan** - For AMD Radeon and Instinct GPUs (architecturally supported via WGPU; physical hardware testers welcome!)

GPU acceleration offloads compute-heavy batch vector distance kernels to hardware accelerators.

> **Important Workload Guidance:** 
> - **Single-Query Searches (Batch = 1):** For individual point queries and sequential HNSW graph traversals, **CPU execution (with AVX2 / AVX-512 / ARM NEON SIMD) is generally faster than GPU**. Fixed host-to-device PCIe transfer latencies and kernel launch overheads dominate single-vector calculations on datasets under 100k vectors.
> - **Batch-Parallel Workloads:** GPU acceleration is primarily advantageous for **dense batch operations** (e.g. evaluating large query batches simultaneously against high-dimensional vector matrices, or brute-force flat scoring) where massive parallelism outweighs PCIe transfer latency.
> - **Future Full-GPU Workloads:** End-to-end resident GPU vector search (keeping index graphs and candidate queues on-device) is on the roadmap to enable GPU query speedups for live lakehouse tables.

## Installation

### Unified Binary (PyPI)

BenoStreamDB provides a unified binary package with support for major compute backends. Runtime detection automatically activates available hardware drivers, safely falling back to CPU if drivers are unavailable.

```bash
pip install benostreamdb
```

> **Hardware Requirements:** 
> - **NVIDIA**: Requires NVIDIA driver (`libcuda.so` on Linux, `nvcuda.dll` on Windows).
> - **AMD**: Requires ROCm/Vulkan drivers.
> - **Intel**: Requires Level Zero/Vulkan drivers.
> - **Apple**: Requires macOS 12.3+ (Built-in Metal).

```python
import benostreamdb as bsdb

# Auto-detect and use best available compute backend
device = bsdb.Device("auto")
print(f"Using backend: {device.backend}")

# Pick a specific backend (Torch-aligned strings)
device = bsdb.Device("cuda")    # NVIDIA or AMD ROCm (Torch standard)
device = bsdb.Device("xpu")     # Intel XPU (Torch standard)
device = bsdb.Device("mps")     # Apple Silicon Metal
device = bsdb.Device("cpu")     # CPU fallback (always available)

# Check availability
print(bsdb.Device.is_available("cuda"))  # True if NVIDIA or AMD ROCm present
print(bsdb.Device.is_available("xpu"))   # True if Intel hardware present
```

## NVIDIA CUDA Setup

### Requirements

- **GPU**: NVIDIA GPU with compute capability 6.0 or higher
  - Pascal (GTX 10 series) or newer
  - Recommended: RTX 20/30/40/50 series, A100, H100, B100/B200
- **Driver**: NVIDIA driver 450.80.02 or later (550+ recommended for RTX 50 series / Blackwell)
- **CUDA Runtime & Toolkits**: Supports CUDA 11.x, 12.x, and 13.x (CUDA 13 verified and tested locally). Runtime discovery links against `libcuda.so` (Linux) or `nvcuda.dll` (Windows) provided by the driver.

> **Note on NVRTC JIT Compilation:** BenoStreamDB compiles CUDA kernels at **runtime** via NVRTC (Just-In-Time). The CUDA Toolkit and `nvcc` compiler are **not** required for building or running. BenoStreamDB features custom, version-agnostic NVRTC discovery that automatically resolves `libnvrtc` across CUDA 11, CUDA 12, and CUDA 13 (including pip wheels like `nvidia-cuda-nvrtc-cu12` or `nvidia-cuda-nvrtc-cu13`).

### Supported GPUs

| Series | Compute Capability | Supported |
|--------|-------------------|-----------|
| GTX 10 series (Pascal) | 6.0-6.1 | ✅ Yes |
| GTX 16 series (Turing) | 7.5 | ✅ Yes |
| RTX 20 series (Turing) | 7.5 | ✅ Yes |
| RTX 30 series (Ampere) | 8.6 | ✅ Yes |
| RTX 40 series (Ada) | 8.9 | ✅ Yes |
| RTX 50 series (Blackwell) | 10.0 / 12.0 | ✅ Yes |
| Tesla V100 | 7.0 | ✅ Yes |
| A100 | 8.0 | ✅ Yes |
| H100 | 9.0 | ✅ Yes |
| B100 / B200 (Blackwell) | 10.0 | ✅ Yes |

### Installation on Linux (Ubuntu/Debian)

```bash
# Install NVIDIA driver (provides libcuda.so at runtime)
sudo apt-get update
sudo apt-get install nvidia-driver-535

# Verify driver installation
nvidia-smi
```

> **No CUDA Toolkit installation required.** The NVIDIA driver supplies `libcuda.so` at runtime; kernels are JIT-compiled via NVRTC.

### Building from Source with CUDA Support

CUDA is an optional compile-time feature. Enable it when building:

```bash
# Rust library
cargo build --release --features cuda

# Python bindings
maturin develop --features cuda
```

Without `--features cuda`, the CUDA backend will be unavailable at runtime (other backends like ROCm, Metal, and XPU remain functional).

### Installation on Windows

**Recommended: WSL2 (Windows Subsystem for Linux)**
Windows users should use **WSL2** to run BenoStreamDB with GPU support:

1. Install WSL2 and Ubuntu (e.g., `wsl --install -d Ubuntu-22.04`)
2. Install NVIDIA Windows Driver (this provides the necessary kernel-mode interface for WSL2)
3. Within the WSL2 Ubuntu environment, follow the **Linux installation** instructions above.
4. Verify from within WSL:
   ```bash
   nvidia-smi
   ```

**Native Windows (outside WSL2):**
We build and package native Windows binaries with full architectural support for CUDA via dynamic linking to `nvcuda.dll` and runtime NVRTC JIT compilation. However, because our automated CI test suite currently runs on Linux, native Windows GPU execution is currently **untested in CI**. 

If you are running BenoStreamDB natively on Windows with an NVIDIA GPU, we would love your help: please test your workload and [report your results or open an issue](https://github.com/benolabsai/benostreamdb/issues) to let us know how it runs on your machine. For production workflows today, WSL2 remains our tested and recommended environment.

### Verification

```python
import benostreamdb as bsdb
import numpy as np

# Create CUDA context
ctx = bsdb.ComputeContext("cuda")
print(f"CUDA backend initialized: {ctx.backend}")

# Test GPU computation
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(10000, 768).astype(np.float32)
distances = bsdb.l2_batch(query, database, device=ctx)
print(f"Computed {len(distances)} distances on GPU")

# Check performance stats
stats = ctx.get_stats()
print(f"GPU time: {stats['total_gpu_time_ms']}ms")
```

## AMD ROCm Setup

### Requirements

- **GPU**: AMD Radeon RX 5000 series or newer (RDNA 1, 2, 3), or Instinct MI series.
- **OS**: Linux (Primary support for compute workloads).
- **Backend**: BenoStreamDB uses **WGPU/Vulkan** for AMD compute, ensuring compatibility across a wide range of Linux distributions.

> **Hardware Testing Status (Help Wanted):** BenoStreamDB targets AMD GPUs via its cross-platform **WGPU / Vulkan** compute pipeline. However, our internal testing environment has validated WGPU primarily on Intel integrated graphics; we currently lack physical AMD discrete GPUs (RX 6000/7000 series or Instinct) in our regular CI test harness. While the WGSL distance shaders compile and run against standard Vulkan compute targets, AMD execution is currently **untested on real hardware**.
>
> If you have an AMD GPU with ROCm or Vulkan on Linux, we would love your help! Please run the verification snippet below and [report your benchmark results or open an issue](https://github.com/benolabsai/benostreamdb/issues).

### Target GPUs

| Series | Architecture | Testing Status |
|--------|-------------|----------------|
| RX 5000 series | RDNA 1 | 🔬 Architecture supported (Untested on hardware — Help Wanted) |
| RX 6000 series | RDNA 2 | 🔬 Architecture supported (Untested on hardware — Help Wanted) |
| RX 7000 series | RDNA 3 | 🔬 Architecture supported (Untested on hardware — Help Wanted) |
| Radeon VII | GCN 5.1 | 🔬 Architecture supported (Untested on hardware — Help Wanted) |
| MI100 | CDNA 1 | 🔬 Architecture supported (Untested on hardware — Help Wanted) |
| MI200 series | CDNA 2 | 🔬 Architecture supported (Untested on hardware — Help Wanted) |

### Installation on Ubuntu

```bash
# Download and install AMD GPU driver installer
wget https://repo.radeon.com/amdgpu-install/latest/ubuntu/noble/amdgpu-install_6.0.2-1_all.deb
sudo apt install ./amdgpu-install_6.0.2-1_all.deb

# Install ROCm
sudo amdgpu-install --usecase=rocm

# Add user to video and render groups
sudo usermod -a -G video,render $USER

# Reboot to apply changes
sudo reboot

# Verify installation
rocm-smi
rocminfo
```

### Installation on RHEL/CentOS

```bash
# Add ROCm repository
sudo tee /etc/yum.repos.d/rocm.repo <<EOF
[ROCm]
name=ROCm
baseurl=https://repo.radeon.com/rocm/rhel8/rpm
enabled=1
gpgcheck=1
gpgkey=https://repo.radeon.com/rocm/rocm.gpg.key
EOF

# Install ROCm
sudo yum install rocm-hip-sdk

# Add user to video and render groups
sudo usermod -a -G video,render $USER

# Reboot
sudo reboot
```

### Verification

```python
import benostreamdb as bsdb

# Create ROCm context
ctx = bsdb.ComputeContext("rocm")
print(f"ROCm backend initialized: {ctx.backend}")

# Test computation
import numpy as np
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(10000, 768).astype(np.float32)
distances = bsdb.l2_batch(query, database, device=ctx)
print(f"Computed {len(distances)} distances on AMD GPU")
```

## Apple Metal (MPS) Setup

### Requirements

- **Hardware**: Apple Silicon Mac (M1, M2, M3, M4, M5, or newer)
- **OS**: macOS 12.3 (Monterey) or later
- **No additional installation required** - Metal is built into macOS

### Supported Devices

| Device | Chip | Supported |
|--------|------|-----------|
| MacBook Air (2020+) | M1/M2/M3/M4/M5 | ✅ Yes |
| MacBook Pro (2020+) | M1/M2/M3/M4/M5 Pro/Max/Ultra | ✅ Yes |
| Mac mini (2020+) | M1/M2/M2 Pro | ✅ Yes |
| Mac Studio | M1/M2 Max/Ultra | ✅ Yes |
| iMac (2021+) | M1/M3/M4/M5 | ✅ Yes |
| Mac Pro (2023+) | M2 Ultra | ✅ Yes |

### Verification

```python
import benostreamdb as bsdb

# Auto-detect should find Metal on Apple Silicon
ctx = bsdb.ComputeContext.auto_detect()
print(f"Backend: {ctx.backend}")  # Should show "mps"

# Or explicitly create Metal context
ctx = bsdb.ComputeContext("mps")
print(f"Metal backend initialized")

# Test computation
import numpy as np
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(10000, 768).astype(np.float32)
distances = bsdb.l2_batch(query, database, device=ctx)
print(f"Computed {len(distances)} distances on Apple GPU")
```

### Performance Notes

- Apple Silicon uses unified memory (shared between CPU and GPU)
- No explicit memory transfer overhead
- M1 Max/Ultra and M2 Max/Ultra have more GPU cores for better performance
- Recommended for databases up to available system memory

## Intel XPU Setup

### Requirements

- **GPU**: Intel Iris Xe (tested and verified), Arc A-Series (Alchemist), Arc B-Series (Battlemage), or Data Center GPU Max.
- **Testing Status**: Tested and validated on Intel integrated graphics (Iris Xe) via WGPU/Vulkan.
- **OS**: Linux (Native) or Windows via WSL2.
- **Drivers**: Requires Level Zero and Vulkan user-mode drivers.

### Installation on Linux (Ubuntu/Debian)

For Intel hardware, you must install the compute and media runtimes to enable WGPU acceleration.

```bash
# Add Intel graphics repository (Noble 24.04 instructions)
wget -qO - https://repositories.intel.com/graphics/intel-graphics.key | sudo gpg --dearmor --yes -o /usr/share/keyrings/intel-graphics.gpg
echo "deb [arch=amd64,i386 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/graphics/ubuntu noble main" | sudo tee /etc/apt/sources.list.d/intel-graphics.list
sudo apt update

# Install Level Zero and Media runtimes
sudo apt install intel-level-zero-gpu intel-media-va-driver-non-free

# Verify installation
vulkaninfo | grep "vendorID = 0x8086"
```

### Verification (Linux)

```bash
# Verify Vulkan/WGPU installation
vulkaninfo | grep vendor
# Or check adapter listing in BenoStreamDB
python -c "import benostreamdb as bsdb; print(bsdb.Device.list_available_backends())"
```

### Installation on Windows (via WSL2)

Windows users should ensure they have the latest Intel Graphics drivers installed on the host. These provide Vulkan support to WSL2, enabling BenoStreamDB to detect and use the GPU via WGPU.

### Verification

```python
import benostreamdb as bsdb

# Create XPU (Intel) context
device = bsdb.Device("xpu")
print(f"Intel backend initialized: {device.backend}")

# Test computation
import numpy as np
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(10000, 768).astype(np.float32)
distances = bsdb.l2_batch(query, database, device=device)
print(f"Success: Computed on {device.backend}")
```

## Multi-GPU Systems

For systems with multiple GPUs, you can pin one device or install a pool.

Pin a specific device:

```python
import benostreamdb as bsdb

# List available backends
ctx = bsdb.ComputeContext.auto_detect()
print(f"Available backends: {ctx.list_available_backends()}")

# Use specific GPU device
ctx = bsdb.ComputeContext("cuda", index=0)  # First GPU
ctx = bsdb.ComputeContext("cuda", index=1)  # Second GPU

# Check which device is being used
print(f"Using device: {ctx.device_id}")
```

Install a process-wide pool so worker threads spread across devices:

```python
# Each worker thread that first asks for a context gets cuda:0 or cuda:1
bsdb.set_gpu_device_pool("cuda:0,cuda:1")
print(bsdb.gpu_device_pool())   # ['cuda:0', 'cuda:1']

# Clear the pool and fall back to the process-wide device
bsdb.set_gpu_device_pool("")
```

For Flight SQL, MCP, and other server processes, use the same list:

```bash
BSDB_GPU_DEVICE="cuda:0,cuda:1" benostreamdb-flight
```

The pool is **concurrency scaling**, not single-query sharding: each worker thread is assigned one
device, but one HNSW `search` call still runs on a single GPU. Use separate processes or threads
pinned to different devices when you need both GPUs busy with independent searches.

## Troubleshooting

### GPU Not Detected

**Symptom:** `auto_detect()` returns CPU backend

**Solutions:**
1. Verify GPU drivers are installed:
   - NVIDIA: `nvidia-smi`
   - AMD: `rocm-smi` or `vulkaninfo`
   - Intel: `vulkaninfo` (Check for vendor `0x8086`)
   - Apple: Check System Settings → Hardware

2. Check backend availability:
   ```python
   ctx = bsdb.ComputeContext.auto_detect()
   print(ctx.list_available_backends())
   ```

3. Try creating backend explicitly:
   ```python
   try:
       ctx = bsdb.ComputeContext("cuda")
   except RuntimeError as e:
       print(f"CUDA not available: {e}")
   ```

### Out of Memory Errors

**Symptom:** `RuntimeError: GPU out of memory`

**Solutions:**
1. Process database in chunks:
   ```python
   chunk_size = 10000
   all_distances = []
   for i in range(0, len(database), chunk_size):
       chunk = database[i:i+chunk_size]
       distances = bsdb.l2_batch(query, chunk, device=ctx)
       all_distances.append(distances)
   all_distances = np.concatenate(all_distances)
   ```

2. Use smaller data types (float32 instead of float64)
3. Use sparse vectors for sparse data
4. Use binary vectors for binary features

### Slow Performance

**Symptom:** GPU is slower than CPU

**Possible causes:**
1. **Small batch size**: GPU overhead dominates for < 1,000 vectors
   - Solution: Use CPU for small batches, GPU for large batches

2. **Data type mismatch**: Using float64 instead of float32
   - Solution: Convert to float32: `data.astype(np.float32)`

3. **Memory transfer overhead**: Creating new context each time
   - Solution: Reuse GPU context across multiple operations

4. **Wrong backend**: Using Intel XPU path on NVIDIA GPU
   - Solution: Use CUDA for NVIDIA, ROCm for AMD

### Driver Version Mismatch

**Symptom:** `RuntimeError: CUDA driver version is insufficient`

**Solution:**
```bash
# Check current driver version
nvidia-smi

# Update NVIDIA driver (Ubuntu)
sudo apt-get update
sudo apt-get install --only-upgrade nvidia-driver-535

# Or install latest driver
sudo ubuntu-drivers autoinstall
```

## Performance & Workload Characteristics

In empirical competitor benchmarks, GPU speedup is **workload-dependent**:

- **Point Queries & HNSW Search:** Single-query ANN lookups (sequential graph walks) are dominated by pointer traversals and latency rather than arithmetic throughput. Under single-stream query workloads (batch = 1), optimized CPU SIMD execution (AVX2/AVX-512) delivers lower p50 latency and higher QPS than GPU due to PCIe transfer and kernel dispatch overheads.
- **Dense Batch Matrix Operations:** Standalone distance kernels (`l2_batch`, `cosine_batch`) running large batches ($N \ge 100,000$ vectors, batch size $\ge 1,000$) effectively saturate GPU warps and achieve high throughput.
- **Index Construction:** Background IVF clustering and k-means centroid assignments benefit from GPU acceleration during offline index generation.
- **Full GPU Workloads (Roadmap):** True query speedups for lakehouse vector search require resident GPU index caching and batched search queues, which are actively tracked on the roadmap.

See the [Benchmarking Guide](BENCHMARKING.md) for reproducible competitor comparison numbers under identical container envelopes.

## Best Practices

1. **Reuse Device** across multiple operations
2. **Use float32** instead of float64 for better GPU performance
3. **Batch operations** when possible (process multiple queries together)
4. **Profile your workload**
5. **Choose appropriate backend** for your hardware
6. **Monitor GPU memory** usage for large databases
7. **Use sparse/binary vectors** when applicable to reduce memory

## See Also

- [Python Vector API Documentation](PYTHON_VECTOR_API.md) - Complete API reference
- [Vector Configuration Guide](CONFIGURATION.md) - Index tuning
- [Benchmarking Guide](BENCHMARKING.md) - Performance testing