Linuxknowledge-base

Build and query a local embeddings index for a Markdown vault

Section-chunk a Markdown vault, embed with nomic-embed-text on a local Ollama cluster, cache by content hash, swap the index atomically stamped with the vault commit, and query it with fail-closed staleness checks.

Distrosdebian, ubuntu, opensuse
Shellbash
Updated
Script
bash
# ─────────────────────────────────────────────────────────────
# Local semantic search over a Git vault, as a revision-labeled
# derived consumer. Ollama serves nomic-embed-text on your nodes.
# ─────────────────────────────────────────────────────────────

# 1. Pull the embedding model on each Ollama node
for host in 10.0.0.65 10.0.0.66 10.0.0.67; do
  ssh "root@$host" "ollama pull nomic-embed-text"
done

# 2. Embed a single chunk (the indexer loops this over all chunks,
#    round-robining across nodes for throughput)
curl -s http://10.0.0.65:11434/api/embeddings \
  -d '{"model":"nomic-embed-text","prompt":"the section text here"}' \
  | jq '.embedding | length'          # → vector dimension

# 3. Build the index: section-chunk, hash-cache, embed, atomic swap,
#    stamp the vault commit
cd ~/wiki
node tools/embeddings/build-index.mjs .
#   - split each page on headings (## …) into section chunks
#   - hash each chunk; re-embed ONLY chunks whose hash changed
#   - write new index beside the old, then rename() over it (atomic)
#   - record vault HEAD into the index (.index-commit)

# 4. Verify the index reports the current vault commit
node tools/embeddings/verify-index.mjs .
#   → index commit == git rev-parse HEAD  (else: STALE)

# 5. Query — returns ranked chunks, each with source page + commit
node tools/embeddings/query.mjs "recover a locked keyring after reboot"

# ─────────────────────────────────────────────────────────────
# Reference: the hash-cache + atomic-swap core (pseudocode).
# ─────────────────────────────────────────────────────────────
#   const commit = sh('git rev-parse HEAD');
#   const cache  = loadCache('.embed-cache.json');   // {chunkHash: vector}
#   const chunks = pages.flatMap(splitOnHeadings);   // section chunks
#   const index  = [];
#   const nodes  = ['10.0.0.65','10.0.0.66','10.0.0.67'];
#   for (const [i, c] of chunks.entries()) {
#     const h = sha256(c.text);
#     const vec = cache[h] ?? await embed(nodes[i % nodes.length], c.text);
#     cache[h] = vec;
#     index.push({ path: c.path, heading: c.heading, vec });
#   }
#   writeFile('.index.tmp', JSON.stringify({ commit, index }));
#   writeFile('.embed-cache.json', JSON.stringify(cache));
#   renameSync('.index.tmp', '.index.json');         // atomic swap
#   writeFile('.index-commit', commit);
#
#   // query: cosine(queryVec, index[i].vec), take top-k, return with commit

What this does

Builds local semantic search over a Markdown vault as a revision-labeled derived consumer. It section-chunks each page on its headings, embeds chunks with nomic-embed-text served by Ollama across your nodes, caches vectors by content hash so rebuilds only re-embed changed chunks, swaps the new index in atomically, and stamps it with the vault commit. Queries return ranked chunks with their source page and the index commit, so callers can detect and reject stale results.

Why this approach

Keyword search is limited by vocabulary. If you write a runbook titled “Recover GRUB after a failed EFI update,” grep for “boot repair” finds nothing — the words don’t overlap. An embeddings-based semantic index finds it because the embedding model understands that “boot repair” and “recover GRUB after a failed EFI update” describe related concepts. Once you’ve built a vault with more than a few dozen pages, this becomes the difference between finding what you need in seconds and spending ten minutes paging through titles hoping you remember exactly what you wrote.

Building the index locally matters for a homelab. Your notes include IP addresses, hostnames, and service architecture details that you probably don’t want passing through a third-party API. A local embedding model on your Ollama nodes keeps all of that on-premises: the model runs on hardware you control, there are no per-query fees, and the index works when your internet connection doesn’t. The model used here — nomic-embed-text — is compact enough to run on CPU if your nodes don’t have GPUs, though a GPU speeds up bulk rebuilds significantly.

The content-hash cache is the detail that makes regular rebuilds practical. Naively, rebuilding the index means re-embedding every chunk in the vault — for a 750-page vault with 3–4 sections per page, that can take several minutes even with multiple Ollama nodes. Caching by content hash means only changed sections need to be re-embedded; an incremental rebuild after a single-page edit takes seconds. The first build is still slow, but after that the incremental cost is proportional to how much content actually changed.

The commit stamp is what makes the index trustworthy in automated workflows. Every query returns the vault commit the index was built from alongside the hits. A caller — an AI agent’s begin step, for instance — checks this commit against vault HEAD and can decide whether the hits are current enough to trust. This is the same pattern used by the Wiki.js projection: never hide the revision a derived consumer was built from, because a stale result presented without that context is invisible misinformation.

Prerequisites

  • An Ollama cluster (one or more nodes) reachable on your LAN; replace 10.0.0.65/66/67 with your node IPs.
  • A Git vault checked out or mounted locally (~/wiki); Node.js 18+ and jq.
  • Enough RAM/VRAM on a node to serve nomic-embed-text (it is compact — CPU works, a GPU is faster).

Notes

  • Substitute your own values before running any line — this playbook uses example stand-ins. Replace 10.0.0.65/.66/.67 (your Ollama node IPs — a single host is fine), ~/wiki (your repo path), and the tools/… script paths.
  • Chunk by section, not by fixed size. Splitting on headings keeps each chunk a coherent unit of meaning, which retrieves far better than arbitrary byte windows.
  • Cache by content hash. Most rebuilds change a handful of chunks; re-embedding only those makes an incremental rebuild seconds, not minutes.
  • Swap atomically. Build the new index beside the old and rename() over it, so a query never sees a half-written index.
  • Stamp and verify the commit. The index is a derived consumer, not authority. Record the vault commit; a caller (an agent’s begin step) consults the index read-only to find candidate pages, then confirms each hit against the canonical page. If the index commit lags HEAD, label its hits stale.
  • Spread embedding across nodes by round-robin for throughput; the model is identical on each, so results are consistent.
  • Mark tested: true only after building, verifying, and querying against your own vault and Ollama nodes.
All playbooks