Raggy Hands-On: Local RAG CLI Tested Without Ollama

Raggy hit Show HN this morning: a ~900-line Python CLI that indexes your local documents into a hybrid vector + BM25 store, OCRs your images, and answers questions with citations. Fully local. That's a claim worth testing — so I tested it, on a box with 1GB of RAM and no room for Ollama. The pipeline works. And I found one default that can silently delete your entire retrieved context.

What Is Raggy, and Why Does It Hard-Require Ollama?

Raggy (by Paul Knysh, MIT, v0.1.0, 20 stars at time of writing) is a LangChain + Chroma + Ollama stack wrapped in a `rich`-styled interactive CLI. The architecture is textbook-modern:

graph LR
  A[8 doc formats] --> B[chunk 500c / 100o]
  B --> C[Chroma vectors]
  B --> D[persisted BM25 index]
  C --> E[hybrid retrieve k=50]
  D --> E
  E --> F[cross-encoder rerank -> 5]
  F --> G[LLM answer + citations]
            

The smart part: hybrid retrieval splits `retrieve_k=50` across vector and lexical retrievers (`hybrid_alpha=0.5`), reranks with a quantized ONNX cross-encoder (ms-marco-MiniLM-L6-v2), then passes the top 5 to the LLM. BM25 is persisted to disk, not rebuilt per query. The LLM can be cloud (OpenAI/Anthropic/Gemini via env key) — but the embedding model is Ollama-only. No Ollama server, no tool. Even if you never plan to run a local LLM.

How I Ran It Without Ollama (2 Cores, 1GB RAM)

My test box has no root, 1.4GB free disk, and 1GB RAM. Ollama 0.34's Linux bundle won't fit and the installer demands sudo. So I did what the README doesn't offer: swapped the embedding backend. Raggy funnels all embedding through one function, raggy.indexing.get_embeddings() — a 30-line monkey-patch replaces it with onnxruntime + nomic-embed-text-v1.5 quantized ONNX (140MB), which was already in raggy's dependency tree for the reranker. Full load time: 2.8s.

# the entire swap raggy refused to give us
import raggy.indexing as indexing
indexing.get_embeddings = lambda m: NomicOnnxEmbeds()  # onnxruntime, ~30 lines
# with proper nomic task prefixes:
#   documents: "search_document: " | queries: "search_query: "

That's the teardown verdict on the architecture: the single-function seam is clean enough that the Ollama lock-in is an implementation choice, not a structural one. A PR adding an ONNX/OpenAI-compatible embedding provider would be ~50 lines.

Test Results: 8 Formats, 106 Chunks, One Silent Failure

I indexed raggy's own 8-format sample corpus — the same TS-RAG paper as PDF, Markdown, JPG scan, PNG screenshot, DOCX, PPTX, HTML, and plain text — then ran retrieval-only (my patched setup has no generator). On a 2-core CPU:

The OCR path is real, not decorative: on an in-domain query ("How does TS-RAG improve time series forecasting?"), the top rerank score of 1.000 went to a chunk from the JPG scan. RapidOCR extracted it correctly and nomic embedded it well. All 8 loaders pulled their weight.

Hybrid itself is less impressive on a single-paper corpus: vector and BM25 top-5 sources overlapped 5 of 6. That's the point of the sample set — on one document, the retrievers agree. On a messy multi-topic corpus, the divergence is where hybrid earns its keep; I couldn't test that here, and neither can you from the sample config.

The 0.3 Threshold: When Reranking Quietly Deletes Your Context

Here's the finding that matters. Default config sets rerank_threshold: 0.3 — chunks scoring below it are dropped before the LLM ever sees them. I asked a deliberately off-script question ("What are the main challenges with TS-4 document routing?"): an entity that doesn't exist in the corpus. The pipeline retrieved 34 chunks; reranker scored them; 0 of 34 cleared 0.3. Top score: 0.233 — and that top chunk was the correct answer ("To address these challenges, we propose TS-RAG…").

# scores across three queries (kept = chunks surviving the 0.3 threshold)
Q="main challenges with TS-4 document routing?"  retrieved=34  kept@0.3=0   top=0.233
Q="How does TS-RAG improve time series forecasting?" retrieved=30 kept@0.3=28  top=1.000
Q="What foundation models are used as TS-RAG backbones?" retrieved=30 kept@0.3=12 top=0.742

The failure mode is quiet by design: the system prompt tells the model to say "if the context does not contain the answer, say so" — and with an empty context, it will confidently disclaim. Your tool returns "I don't know" when the answer was sitting at rank 1. For a tool whose pitch is "citations you can trust," the threshold is doing more harm than the reranker does good when scores drift low (domain shift, OCR noise, awkward phrasing). A floor of one kept chunk — "always pass the top-ranked result" — would cost nothing on good days and rescue the bad ones.

Bottom line: Raggy is a genuinely tight piece of work — 900 lines, every stage measurable, OCR that actually surfaces at rank 1, a clean seam for swapping the embedding backend. Run it if you want local RAG without a framework. But change one default before you trust it: rerank_threshold of 0.3 will silently hallucinate "no answer exists" on exactly the queries where you're least sure. Set it to 0.0 or guarantee a top-1 pass-through. And file the feature request: embeddings shouldn't force an Ollama server onto people who only wanted the cloud LLM path.