Bayesian BM25

Core Theory

The Problem with Raw BM25 Scores

Standard BM25 produces unbounded scores that lack consistent meaning across queries. A score of 8.5 might be highly relevant for one query but mediocre for another, making threshold-based filtering and multi-signal fusion unreliable. Bayesian BM25 solves this by transforming scores into calibrated probabilities in $[0, 1]$.

Sigmoid Likelihood

The likelihood of relevance given a BM25 score follows a sigmoid model parameterized by steepness $\alpha$ and midpoint $\beta$:

This maps any real-valued BM25 score $s$ into $(0, 1)$, capturing the intuition that relevance probability increases with score but saturates at the extremes.

Composite Prior

A document's prior relevance probability combines two information-theoretic signals: term frequency (higher TF indicates richer topical content) and document length ratio (documents near the corpus average length are more likely to be relevant):

where $\pi_{tf} = \sigma(\log(1 + tf))$ and $\pi_{len} = 0.5 + 0.5 \cdot (1 - |r - 1|)$ with $r$ being the document length ratio. The composite prior is multiplicative in probability space, which becomes additive in log-odds space.

Bayesian Posterior

The final posterior combines likelihood, prior, and a corpus-level base rate $b_r$ using Bayes' theorem. In log-odds space, the three terms decompose additively:

This three-term decomposition separates the score-dependent signal (likelihood) from document-specific context (prior) and corpus-level prevalence (base rate), enabling each component to be estimated independently. The base rate alone reduces Expected Calibration Error by 68–77% without requiring any relevance labels.

Log-Odds Conjunction

Naive probability multiplication ($P = \prod P_i$) suffers from conjunction shrinkage: combining multiple agreeing high-probability signals produces a low fused probability. The log-odds conjunction resolves this by averaging in logit space with confidence scaling:

The $n^{\alpha}$ factor (with $\alpha = 0.5$ giving $\sqrt{n}$ scaling) compensates for the averaging effect so that $n$ perfectly agreeing signals return the consensus probability, not a diluted version. With per-signal weights $w_i$, this becomes the Log-OP formulation $\sigma\!\bigl(n^{\alpha} \sum w_i \cdot \text{logit}(P_i)\bigr)$, where $\alpha$ and weights compose multiplicatively.

Pipeline

graph LR
    S["Raw BM25 Score"] --> SIG["Sigmoid Likelihood
L = sigma(alpha * (s - beta))"]
    SIG --> POST["Bayesian Posterior
logit(P) = logit(L) + logit(pi) + logit(br)"]
    TF["Term Frequency"] --> PRIOR["Composite Prior
pi = pi_tf * pi_len"]
    DL["Doc Length Ratio"] --> PRIOR
    PRIOR --> POST
    BR["Base Rate"] --> POST
    POST --> PROB["Calibrated
Probability"]
    PROB --> FUSE["Fusion
Log-Odds / AND / OR"]
    VEC["Vector Score"] --> COS["cosine_to_probability()"]
    VEC --> VPT["VectorProbabilityTransform
P = sigmoid(log(f_R/f_G) + logit(base))"]
    ANN["ANN Index
(IVF / HNSW)"] -.->|"local sample"| VPT
    COS --> FUSE
    VPT --> FUSE
    FUSE --> RANK["Final
Ranking"]
  

Multi-Signal Fusion

All fusion operations work in probability space with well-defined semantics. Because every signal is a calibrated probability, they compose freely.

Learnable Weights

LearnableLogOddsWeights learns per-signal reliability from labeled data via a Hebbian gradient that is backprop-free. Starting from the Naive Bayes uniform initialization ($w_i = 1/n$), the gradient $\nabla_{z_j} = n^{\alpha}(p - y) \cdot w_j(x_j - \bar{x}_w)$ adjusts weights based on pre-synaptic activity times post-synaptic error.

Attention-Based Fusion

AttentionLogOddsWeights replaces static weights with query-dependent attention. A linear projection from query features to softmax attention weights allows the fusion to adapt per query: some queries benefit more from lexical signals, others from semantic similarity. Optional per-signal logit normalization (normalize=True) equalizes signal scales before the weighted sum.

Sparse Signal Gating

For high-dimensional signal spaces, ReLU gating ($\max(0, \text{logit})$, MAP estimation), Swish gating ($\text{logit} \cdot \sigma(\text{logit})$, Bayes estimation), and GELU gating ($\text{logit} \cdot \sigma(1.702 \cdot \text{logit})$, Gaussian noise model) suppress noisy negative-logit signals before aggregation (Paper 2, Theorems 6.5.3/6.7.4/6.8.1). Softplus gating ($\log(1 + e^{\beta \cdot \text{logit}}) / \beta$, Remark 6.5.4) is a smooth ReLU that never zeroes out evidence, making it suitable for small datasets where discarding any signal is costly. The generalized Swish gate $\text{logit} \cdot \sigma(\beta \cdot \text{logit})$ interpolates between $x/2$ ($\beta \to 0$), standard Swish ($\beta = 1$), and ReLU ($\beta \to \infty$) via the gating_beta parameter (Theorem 6.7.6).

Multi-Head Attention

MultiHeadAttentionLogOddsWeights creates multiple independent attention heads with different random initializations. Each head produces fused log-odds independently, then the results are averaged in log-odds space before converting back to probability via sigmoid (Remark 8.6). Multi-head diversity reduces variance compared to single-head attention. Both single-head and multi-head attention support exact pruning via compute_upper_bounds() and prune() (Theorem 8.7.1).

Neural Score Calibration

PlattCalibrator (sigmoid: $P = \sigma(a \cdot s + b)$) and IsotonicCalibrator (PAVA monotone regression) convert raw neural model scores into calibrated probabilities suitable for Bayesian fusion. Calibrated scores can be combined with BM25 probabilities via log_odds_conjunction.

Vector Similarity Calibration

VectorProbabilityTransform replaces the naive $\frac{1+\cos\theta}{2}$ conversion with a likelihood ratio framework (Paper 3): $P(R|d) = \sigma(\log(f_R(d) / f_G(d)) + \text{logit}(P_{base}))$, where $f_R$ is estimated via weighted KDE or GMM-EM, and $f_G$ is the background Gaussian. Auto-routing selects KDE (gap detected, $K \geq 50$) or GMM (small $K$ or smooth distributions).

calibrate_with_sample() decouples density estimation from evaluation: the local ANN neighborhood (e.g. IVF probed cells) provides the training sample for $f_R$, while probabilities are produced for an arbitrary evaluation set. This is the index-aware calibration path where the density landscape comes from one set of distances and the output probabilities are needed at different points. Standalone helpers ivf_density_prior() and knn_density_prior() provide optional density priors from IVF cell populations or HNSW neighbor distances.

Benchmarks

Evaluated on 5 BEIR datasets using the retrieve-then-evaluate protocol (top-1000 per signal, union candidates, pytrec_eval). Dense encoder: all-MiniLM-L6-v2. BM25: k1=1.2, b=0.75, Lucene variant with Snowball English stemmer.

NDCG@10 (Zero-Shot)

Delta vs BM25 (NDCG@10)

Bayesian-Attn-Norm achieves the highest average NDCG@10 (41.67%), outperforming Convex combination (+0.52), RRF (+1.18), and BM25 (+6.28) in a fully zero-shot setting with no relevance labels. Bayesian-Vector-Balanced (+5.49) uses likelihood ratio calibration (Paper 3) for the dense signal, competitive with Convex (+5.78). See the full benchmark tables with 26 methods, MAP@10, and Recall@10.

Probability Calibration

The base rate prior alone reduces ECE by 68–77% without any labeled data. With batch fitting, calibration error drops below 1%.

Reproduce

# Zero-shot (26 methods, exact dense retrieval)
python benchmarks/hybrid_beir.py -d <beir-data-dir>

# With IVF dense backend (index-aware calibration)
python benchmarks/hybrid_beir.py -d <beir-data-dir> --dense-backend ivf

# With tuning (supervised + grid search)
python benchmarks/hybrid_beir.py -d <beir-data-dir> --tune

# Download BEIR datasets automatically
python benchmarks/hybrid_beir.py -d <beir-data-dir> --download

Adoption

Citation

@preprint{Jeong2026BayesianBM25,
  author    = {Jeong, Jaepil},
  title     = {Bayesian {BM25}: {A} Probabilistic Framework for Hybrid Text
               and Vector Search},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.18414940},
  url       = {https://doi.org/10.5281/zenodo.18414940}
}

@preprint{Jeong2026BayesianNeural,
  author    = {Jeong, Jaepil},
  title     = {From {Bayesian} Inference to Neural Computation: The Analytical
               Emergence of Neural Network Structure from Probabilistic
               Relevance Estimation},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.18512411},
  url       = {https://doi.org/10.5281/zenodo.18512411}
}

@preprint{Jeong2026VectorCalibration,
  author    = {Jeong, Jaepil},
  title     = {Vector Scores as Likelihood Ratios: Index-Derived {Bayesian}
               Calibration for Hybrid Search},
  year      = {2026}
}