#Searching inside documents

A library, and a librarian.

Until recently the librarian could only read the spines. Ask "where does it say the budget is forty thousand?" and the best it could do was spot a file called budget.pdf and offer to open it. If the sentence lived in CR-réunion-mars.docx, it was never going to be found.

files_grep gives the librarian a card catalogue. One card per passage:

Three things about that card, and the third is the one that matters.

1. The section is stored, so a citation names §3. Budget prévisionnel rather than an anonymous 200-word fragment.
2. The position is in bytes, and files_read already takes a byte offset. The citation is verifiable by re-reading it. Nothing new was invented.
3. Nowhere does it say who may read this. No user column. The catalogue does not know, cannot be asked, and must never be taught.

#The catalogue proposes; Nextcloud disposes

Step 4 is the whole guarantee. The other 188 are not filtered out — they are never obtained. Three things follow from that, and none of them is available to a design that copies permissions into its index:

#Filling it, holding no credential

This server has no Nextcloud account, and must never be pointed at an admin one. So a global crawl does not exist here. Two mechanisms, doing deliberately different jobs:

The webhook cannot read the file, and does not try. Its payload carries a fileid, a path and a uid — but no token, because it fires from a background job where no session exists. So it marks the row dirty and stops there. The next search by somebody who can actually open that file does the reading, with their own credential.

Which produces the right policy without anyone having to write it down:

A file nobody can open is never indexed, because there is nobody to open it.

Discovery is cheap and metadata-only, so the watermark races ahead of the reading. Extraction is capped at 20 documents per pass at concurrency 2, because the document server is a single instance and a large PDF is about five seconds. Nothing here ever blocks an answer: the pass is fire-and-forget.

A cold account is told so, in those words. "The index is still being built" is a different fact from "no match", and conflating them would be a confident lie.

#Chunking: structure, and nothing clever

Split on markdown headings; on labelled sections (Article 5, 3.2 Budget) in prose that has none; on blank lines otherwise. No overlap — Chroma measured the popular 800/400 default at 1.5 precision against 7.0 for the same size with none, at equal recall.

Each passage carries the heading above it, and that heading is the whole parent-child mechanism. The child is the indexed passage; the parent is files_read(path, offset), which already existed. No parent table, no deduplication, no 5–15× token blow-up at generation time. A citation names a section, and nothing is stored twice.

The evidence for taking exactly this much and no more: HiChunk (ACL 2026) puts +6.97 evidence recall over fixed-200 chunks on hierarchical splitting plus the merge back to the parent — and only +3.6, flat on Fact-Cov, for the hierarchy without it. The merge is the load-bearing half, and it costs nothing here.

#Measured, on the default skeleton drive

52 files: three PDFs including the 15 MB Nextcloud manual, plus ODT, ODP, ODS, DOCX and Markdown.

Asked for "two-factor authentication", the manual answers:

That section title was recovered from a PDF with no structure of its own, by one regex. It is the good idea behind PageIndex — navigate by structure — obtained from an index instead of from LLM calls on every query.

#Scans are recorded, not ignored

Extraction under 200 characters means a PDF with no text layer, and there is no model in this process to look at it. The row is parked as needs_ocr and never retried — so the day an OCR service exists, the backlog is one SELECT.

The same small local model that will do embeddings can do the OCR. One service, two jobs, arriving together.

#Three bugs the end-to-end test found

Worth keeping, because each one is invisible from a unit test and each now has a regression guard.

A configuration fault ate the whole corpus. A missing ONLYOFFICE_JWT_SECRET fails identically to an unreadable file, so every PDF in the drive got parked as permanently unreadable — and fixing the config would never have brought them back. A failure is no longer a verdict: the row stays dirty and counts, and only the third attempt gives up.

The drain read the oldest files first. A document written a minute ago — the one someone is actually asking about — went to the back of a 500-file queue. Now newest-first, with failed rows behind everything untried, because otherwise a handful of broken files sit at the head and no pass ever reaches the rest.

The webhook's own rows sorted last. A row the webhook creates has no mtime yet, and SQLite puts NULL last on ORDER BY … DESC. The single file we knew for certain had just changed went to the very back of the queue — exactly inverting the reason for having a webhook. COALESCE(mtime, 9e18): an unknown mtime means "just now".

#Why not a vector store

The short version, with the numbers. The long version, including what would change the answer, is in Decisions.

Meanwhile the strongest peer-reviewed number in the whole comparison favours the architecture that was already here: Amazon (AAAI 2026) measured agentic grep-and-reread at 94.5% of RAG's faithfulness, and beating it on FinanceBench (32.7% vs 24.2%). The missing piece was never a vector store — it was content-level candidate generation. That is what this page is.

#The trigger is instrumented, not argued

FTS5 fails on exactly one thing: vocabulary mismatch. The question says "congés payés", the document says "RTT". So every call records what happened:

CREATE TABLE queries (at, query, candidates, survivors, returned);

• candidates = 0 → a hole in the index.
• candidates > 0, survivors = 0 → a permission boundary doing its job.
• Non-zero candidates the model keeps rephrasing past → vocabulary mismatch, the only failure embeddings actually fix.

When that last category is common enough to count, sqlite-vec goes on the same rowids in the same file — a migration, not a rewrite. Its brute-force scan is a feature here: exact, and it pages in rank order as cheaply as FTS5, which the ef_search cliff above notably does not.

One thing that must not be bolted on casually: hybrid BM25+dense fusion without a reranker was measured as actively harmful (0.303 → 0.289, ECIR 2025), and three of four popular rerankers made MTEB-R worse than dense alone.

#Operating it

./scripts/webhooks.sh          # idempotent; enables webhook_listeners, registers three events

Backfilling the existing drive needs nothing: it happens by bounded passes on every files_grep. Watch it warm up:

docker exec nextcloud-ai-mcp-nextcloud-1 node -e '
const {DatabaseSync}=require("node:sqlite");
const db=new DatabaseSync("/app/data/index.db");
console.log(db.prepare(`SELECT
  (SELECT COUNT(*) FROM docs) docs,
  (SELECT COUNT(*) FROM docs WHERE indexed IS NOT NULL) read,
  (SELECT COUNT(*) FROM docs WHERE dirty=1) pending,
  (SELECT COUNT(*) FROM chunks) passages`).get());'

The .db holds the extracted text of every document a search has reached, which makes it exactly as sensitive as the drive itself. It lives on a volume inside mcp-nextcloud, on the credential side of the boundary, and never in the orchestrator.