How fast is Spanda really from Python?

The claimed 652.1 ns kernel latency only exists inside the Rust process. Measured p50 over 20,000 iterations: 5,605 ns for a raw spanda_eval_c C-FFI roundtrip and 12,742 ns for the full _evaluate_fast wrapper path — about 78,500 evals/sec from Python, roughly 20× below the advertised 1,533,500 evals/sec but still thousands of times faster than 92.4 ms neural Semantic Entropy.

Does spanda.wrap() work after pip install spnda?

No. The PyPI wheel does not ship the compiled Rust cdylib, so the code falls back to the pure-Python CascadedGuardrail path, where wrapper.py line 142 reads receipt.k_samples — an attribute AuditReceipt does not have. First evaluation raises AttributeError. You must clone the repo and cargo build --release the crate yourself.

Is lexical entropy a valid replacement for Semantic Entropy?

For structured reasoning at 7B+ parameters, plausibly yes — the author's AUROC tables show exact-match clustering matching DeBERTa NLI clustering on GSM8K (0.889 at 27B). But the same tables show Confident Mode Collapse at 120B on TriviaQA: inverted AUROC of 0.091 because the model hallucinates the identical wrong answer across all K paths. On ungrounded factual recall, agreement is not evidence.

Spanda Audit: 652 ns Claim vs 12.7 µs in Python

Spanda (spnda) hit the front page twice today with a seductive pitch: LLM hallucination detection via exact-match normalized entropy, "652.1 nanoseconds" per evaluation, 1.53 million evals/sec, 90,000× faster than neural Semantic Entropy, zero dependencies. I cloned it, compiled the Rust core, and benchmarked every layer of that stack. The math is real and the speed is real — but the headline number only exists inside the Rust process boundary, and the pure-Python path you actually get from pip install spnda is 28× slower than advertised and crashes on its flagship feature.

What Spanda claims

The idea is legitimately interesting. Semantic Entropy (Kuhn 2023, Farquhar in Nature 2024) detects hallucinations by sampling K answers and clustering them with a DeBERTa NLI cross-encoder — quadratic forward passes, ~92 ms overhead, a GPU. Spanda replaces the neural clustering with deterministic lexical normalization + normalized Shannon entropy: normalize "The answer is 42", "42.0", "$42", "forty-two" all to 42, then compute R_sc = α·H_norm + (1−α)·(1−w_max). No parameters, no GPU. The README's own tables claim a 652.1 ns kernel, 1,533,500 evals/sec single-core, 2.98 MB RSS, and — my favorite — a "Tests Passing" badge.

What I ran

git clone --depth 1 https://github.com/Adarshent/Spnda.git spanda
cd crates/spanda-core && cargo build --release   # 2m 05s, clean build
cd ../.. && pytest tests/ -q

Test suite: 25 passed, 3 failed. All three failures are the same bug — spanda/wrapper.py:142 reads receipt.k_samples and receipt.agreement_ratio off AuditReceipt, and AuditReceipt (guardrails.py) has no such attributes. That matters because it's not the test path, it's the production fallback path:

# simulate a plain `pip install spnda` user (no compiled Rust cdylib)
import spanda.wrapper as W
W._RUST_LIB = None
W._evaluate_fast(["42", "42"])
# AttributeError: 'AuditReceipt' object has no attribute 'k_samples'

The headline API is spanda.wrap(client). For anyone who didn't manually compile the Rust crate first, wrap() falls into this path and raises AttributeError on the first evaluation. The PyPI wheel doesn't ship the .so — the loader globs for it under crates/spanda-core/target/release/, which only exists in a source checkout. "Zero dependencies" is true in the requirements.txt sense and false in the works-out-of-the-box sense.

The latency receipts

Once I did compile the cdylib, I measured with time.perf_counter_ns over 20,000 iterations (p50, warm):

python compute_rsc (K=10):            p50 = 18,180 ns   (claimed kernel: 652 ns)
python normalize_answer:              p50 =  1,875 ns
raw spanda_eval_c C-FFI roundtrip:    p50 =  5,605 ns
full _evaluate_fast (JSON+ctypes):    p50 = 12,742 ns   → ~78,500 evals/sec

Read that gap. The 652 ns figure is the pure in-Rust compute time — what the CPU spends between the JSON bytes arriving and the response string being assembled. From actual Python, you pay JSON serialization, ctypes marshalling, a string copy, and a json.loads on the way back: 5.6 µs minimum, 12.7 µs through the wrapper. Their own published "1,533,500 evals/sec" is unreachable from Python by a factor of ~20; the Python-batch path in core.py gives you ~55,000 evals/sec.

Is 12.7 µs still good? Emphatically yes — it's still roughly 7,000× faster than the 92.4 ms DeBERTa baseline, still zero GPU, and the core idea survives contact with reality. The normalization is also genuinely well-built: I stress-tested the edge cases and it correctly unifies "42", "42.0", "$42", "forty-two", and "\boxed{42}" into one cluster. Two quirks worth knowing: it silently strips percent signs ("24%" clusters with "24") and parses scientific notation ("1e10" → "10000000000"). Neither bit me in reasoning traces, but both will eventually surprise someone doing factual QA.

graph LR
  A[K sampled answers] --> B[lexical normalize
~1.9 µs Python]
  B --> C[Counter clusters]
  C --> D[H_norm + w_max]
  D --> E[R_sc score]
  E -->|rsc > threshold| F[flag hallucination]
  E -->|rsc ~ 0| G[consensus — or
Confident Mode Collapse]

Verdict: good idea, sloppy floor

Three verdicts, each earned:

The metric: sound, and the honest comparison isn't against Semantic Entropy's 92 ms — it's against just string-matching yourself, which any competent engineer does in an afternoon. Spanda's value is the edge-case normalization table (number words, polar answers, CoT extraction) plus the AUROC evidence that lexical clustering tracks NLI clustering at 7B+ on reasoning. That's a real contribution.
The benchmark claims: technically true, misleadingly framed. 652 ns and 1.53M evals/sec describe an FFI boundary no real user touches. From Python: 78,500 evals/sec. Say that number.
The shipped package: broken at the headline API for anyone without a Rust toolchain, with 3 failing tests wearing a "Tests Passing" badge. That's the part that should cost them stars.

Bottom line: Use the idea, not the install. If you're already sampling K paths for self-consistency, computing normalized entropy over the normalized answers is nearly free and it works — ~13 µs end-to-end from Python is well inside any serving budget, and it will catch genuine disagreement that exact match alone misses. But treat the README's performance table as an internal Rust microbenchmark, not a user-facing spec, and don't pip-install this expecting wrap() to survive first contact. The distance between a microbenchmark and a product is exactly where Spanda fell over.

What Spanda claims

What I ran

The latency receipts

Verdict: good idea, sloppy floor

Related