Knowledge Graphs

Why Flat Vector Search Fails Solo Researchers

If you have maintained a personal research archive for more than a year - whether in Obsidian, Logseq, plain Markdown folders, or Zotero annotations - you have likely hit the ceiling of standard Retrieval-Augmented Generation (RAG). Standard vector search embeds each paragraph into a dense numerical vector and retrieves the top-k chunks closest to your query in cosine space. That works well for localized fact lookup, such as finding a specific CLI flag you documented last month.

Vector search collapses, however, when your question spans multiple documents, timeframes, or domains. Suppose you ask your archive: "Which open-source licensing risks appeared across the infrastructure tools I evaluated in Q1, and which of those tools were recommended by authors I cited in my distributed systems notes?" No single paragraph contains the answer. The answer lives in the connections between entities: Tool A is mentioned in Evaluation Note 12, shares a dependency with Library B in Note 44, and Author C critiqued Library B's license in a 2025 paper summary.

Manual wiki-linking (such as typing [[brackets]] in Obsidian) helps, but it does not scale. As a solo practitioner, you do not have hours each week to curate backlinks across thousands of papers, meeting logs, and code snippets. A Personal Knowledge Graph (PKG) automates this structural layer. It parses your unstructured notes into explicit nodes (Concepts, People, Projects, Tools, Claims, Decisions) and typed, directional edges (DEPENDS_ON, CONTRADICTS, AUTHORED_BY, SUPERSEDES), turning your personal archive into a queryable reasoning substrate.

Property Graphs vs. RDF Triplestores for Individual Practitioners

Before writing a single line of extraction code, you must choose a graph data model. In graph engineering, the two dominant paradigms are Resource Description Framework (RDF) triplestores and Labeled Property Graphs (LPG). Understanding the trade-off saves weeks of unnecessary complexity.

RDF represents every fact as a strict three-part statement: (Subject, Predicate, Object), where subjects and predicates are globally unique URIs governed by formal ontologies such as OWL or SHACL. RDF excels in linked open data, biomedical registries, and cross-organization semantic web publishing. For a solo practitioner, however, RDF introduces heavy friction: attaching metadata to a relationship (such as confidence score, extraction date, or source Markdown line number) requires reification or RDF-star extensions, which complicate queries.

Labeled Property Graphs - implemented by Neo4j, KuzuDB, NetworkX, and lightweight SQLite adjacency tables - allow both nodes and edges to carry arbitrary key-value properties natively. For a personal knowledge graph, an edge can directly store {"source_file": "notes/systems.md", "chunk_hash": "9f2a...", "confidence": 0.91, "valid_from": "2025-11-04"}.

For individual practitioners, start as lightweight as possible: an embedded SQLite database paired with in-memory NetworkX (or KuzuDB for native Cypher queries without a background server daemon) gives you zero-ops portability, single-file backups, and sub-10ms traversals across 50,000+ edges. You only need Neo4j Desktop or a containerized graph server when you require built-in Graph Data Science algorithms or rich interactive browser visualization at scale.

Designing a Lean Personal Ontology That Evolves Cleanly

The fastest way to ruin a personal knowledge graph is to let an LLM invent arbitrary node labels and edge types on every document. Without a constrained schema, one note produces (Person)-[:WROTE]->(Paper), the next produces (Author)-[:AUTHORED]->(Article), and a third produces (Researcher)-[:PUBLISHED_IN]->(Journal). Within a month, your graph fragments into dozens of synonymous relationship types, making structured traversal impossible.

A production personal ontology should be small, rigid at the core, and extensible through properties. For most solo practitioners, six to eight canonical entity types cover 95% of personal research:

• Person: Authors, colleagues, mentors, clients, historical figures.
• Organization: Labs, companies, standards bodies, universities.
• Concept: Theories, algorithms, mental models, architectural patterns.
• Artifact: Papers, books, code repositories, datasets, tools, notes.
• Project: Active personal initiatives, experiments, client engagements.
• Claim: Specific empirical findings, benchmarks, or hypotheses with truth conditions.
• Decision: Personal architectural choices, career moves, or policy commitments.

Pair these node types with a closed vocabulary of 10 to 14 uppercase relationship types such as CREATED, DEPENDS_ON, SUPPORTS, CONTRADICTS, APPLIES_TO, MENTIONS, PART_OF, and SUPERSEDES. Any nuance that does not fit the edge label belongs in an edge property (for example, role: "lead maintainer" or context: "benchmark evaluation"), preserving clean graph topology while retaining rich detail.

Automated Entity-Relation Extraction with Local Models

Extracting structured triples from raw Markdown and PDFs requires strict schema enforcement. If you ask a local model running in Ollama to "extract a knowledge graph from this text" in free-form prose, it will drift in formatting, invent relationships outside your ontology, and occasionally hallucinate entities not present in the source paragraph.

Instead, combine three engineering controls: JSON Schema constrained decoding (supported natively via Ollama's format parameter and Pydantic validation), short structure-aware chunks (400 to 700 tokens per extraction call with the parent Markdown heading injected as context), and exact evidence span grounding. Requiring the model to return a verbatim evidence_quote substring from the chunk allows your Python pipeline to verify programmatically that the claim actually appears in the text before committing the edge.

Canonical Entity Resolution and Alias Deduplication

Entity resolution is the single hardest operational problem in personal knowledge graphs. Across six months of notes, you will refer to the same concept or tool by multiple surface forms: "PostgreSQL", "Postgres", "PG", and "psql"; or "Kuzu" and "KuzuDB"; or a colleague's full name versus their first name in a daily journal entry. If your pipeline creates a separate node for each spelling, your graph's connectivity collapses.

A robust local resolver uses a three-stage cascade before inserting a new node:

1. Deterministic Normalization & Alias Dictionary: Strip punctuation, normalize case, and check a personal YAML/JSON alias map (e.g., mapping "pg" and "postgres" to canonical ID artifact:postgresql).
2. Fuzzy Lexical & Embedding Similarity: Compute Jaro-Winkler or token-set similarity plus cosine similarity against existing node name embeddings of the same entity type. Candidates above 0.92 cosine similarity auto-merge; candidates between 0.80 and 0.92 are routed to a lightweight LLM adjudication step or a weekly human review queue.
3. Context-Aware Disambiguation: Never merge entities solely by name when types differ. "Mercury" (Planet) and "Mercury" (Banking Platform) must remain distinct nodes keyed by (type, canonical_name).

Personal GraphRAG: Community Summaries and Multi-Hop Traversal

Once your notes are materialized into a clean property graph, how do you query it? Modern GraphRAG (pioneered by Microsoft Research and adapted into lightweight local toolkits) combines two complementary retrieval modes: Local Graph Search and Global Community Search.

Local Graph Search starts from seed entities extracted from your question (or matched via vector similarity on node descriptions), expands outward 1 to 2 hops along relevant edge types, and collects both the subgraph structure and the original Markdown chunks attached to those edges. This answers targeted relational questions such as "What are all the bottlenecks I noted across projects using LanceDB?"

Global Community Search solves macro-level synthesis questions such as "What are the overarching themes across my last six months of systems engineering notes?" Using the Leiden community detection algorithm, your pipeline partitions the graph into tightly connected clusters of entities, generates a concise summary for each cluster offline, and embeds those community summaries. At query time, the retriever performs map-reduce synthesis over the top matching communities, giving you a bird's-eye view of your entire intellectual landscape without exceeding local context windows.

Temporal Edges, Confidence Scoring, and Source Provenance

Personal knowledge is never static. In 2024, you might have documented that a project used ChromaDB; in 2026, you migrated that project to SQLite-vec. If your knowledge graph overwrites or blindly accumulates edges without timestamps, your GraphRAG assistant will cite both architectures as simultaneously active.

To prevent temporal confusion, treat your graph as an append-and-supersede ledger. Every edge stores observed_at (derived from the note's YAML frontmatter or file modification date), status (active or superseded), and source_hash (the SHA-256 digest of the source chunk). When a newer note records a conflicting decision or tool migration, you preserve the historical edge with status = "superseded" and link the new edge with SUPERSEDES, giving you a complete intellectual audit trail.

Failure-Mode Recovery: Orphan Nodes, Cycles, and Hallucinated Edges

Even well-tuned local extraction pipelines accumulate structural debt over time. Three failure modes appear most frequently in solo practitioner graphs, and each has a deterministic recovery path:

• Orphan & Low-Degree Noise Nodes: Extraction of overly generic nouns (e.g., "System", "Approach", "Data") creates hub-like garbage nodes or thousands of degree-1 islands. Fix this with a stoplist filter and periodic pruning of degree-1 nodes older than 30 days that lack manual pins.
• Unintended Self-Loops & Semantic Cycles: Models occasionally extract (Concept A)-[:DEPENDS_ON]->(Concept A) or contradictory bidirectional edges. Enforce a pre-commit invariant: reject any edge where source_id == target_id, and flag symmetric contradictions for review.
• Stale Graph After Note Deletion or Edits: When you refactor or delete a Markdown file, its old edges linger as phantom facts. Key every edge by source_file and chunk_hash so an incremental watcher can delete all edges belonging to a modified file hash before re-extracting.

Production Python Pipeline: Local Triple Extractor and SQLite/NetworkX Graph

This complete, self-contained Python module scans a local Markdown vault, extracts schema-validated triples using a local Ollama model, verifies that the quoted evidence exists in the source text, resolves aliases deterministically, and persists the graph into a single portable SQLite file that loads directly into NetworkX.

import hashlib
import json
import sqlite3
from pathlib import Path
from typing import Literal
import urllib.request
import networkx as nx
from pydantic import BaseModel, Field

ENTITY_TYPES = Literal["Person", "Organization", "Concept", "Artifact", "Project", "Claim", "Decision"]
REL_TYPES = Literal["CREATED", "DEPENDS_ON", "SUPPORTS", "CONTRADICTS", "APPLIES_TO", "MENTIONS", "PART_OF", "SUPERSEDES"]

ALIAS_MAP = {
    "postgres": "PostgreSQL",
    "pg": "PostgreSQL",
    "kuzu": "KuzuDB",
    "nx": "NetworkX",
}

class Triple(BaseModel):
    subject: str = Field(min_length=2)
    subject_type: ENTITY_TYPES
    predicate: REL_TYPES
    object: str = Field(min_length=2)
    object_type: ENTITY_TYPES
    evidence_quote: str = Field(min_length=10)
    confidence: float = Field(ge=0.0, le=1.0)

class ExtractionResult(BaseModel):
    triples: list[Triple]

def normalize_entity(name: str) -> str:
    cleaned = " ".join(name.strip().split())
    return ALIAS_MAP.get(cleaned.lower(), cleaned)

def init_db(db_path: Path) -> sqlite3.Connection:
    conn = sqlite3.connect(db_path)
    conn.execute("PRAGMA journal_mode=WAL;")
    conn.execute("""
        CREATE TABLE IF NOT EXISTS nodes (
            node_id TEXT PRIMARY KEY,
            entity_type TEXT NOT NULL,
            canonical_name TEXT NOT NULL
        )
    """)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS edges (
            edge_id TEXT PRIMARY KEY,
            source_id TEXT NOT NULL,
            predicate TEXT NOT NULL,
            target_id TEXT NOT NULL,
            evidence_quote TEXT NOT NULL,
            confidence REAL NOT NULL,
            source_file TEXT NOT NULL,
            chunk_hash TEXT NOT NULL
        )
    """)
    conn.commit()
    return conn

def extract_with_ollama(chunk: str, model: str = "qwen2.5:7b") -> ExtractionResult:
    prompt = (
        "Extract factual entity-relation triples strictly from the text below. "
        "Every triple must include an exact verbatim evidence_quote from the text.\n\n"
        f"TEXT:\n{chunk}"
    )
    payload = {
        "model": model,
        "prompt": prompt,
        "stream": False,
        "options": {"temperature": 0.0},
        "format": ExtractionResult.model_json_schema(),
    }
    req = urllib.request.Request(
        "http://127.0.0.1:11434/api/generate",
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=90) as resp:
        raw = json.loads(resp.read().decode("utf-8"))["response"]
    return ExtractionResult.model_validate_json(raw)

def ingest_markdown_file(conn: sqlite3.Connection, md_path: Path) -> int:
    text = md_path.read_text(encoding="utf-8")
    file_key = str(md_path)
    conn.execute("DELETE FROM edges WHERE source_file = ?", (file_key,))
    inserted = 0
    for idx, block in enumerate(text.split("\n\n")):
        chunk = block.strip()
        if len(chunk) < 80:
            continue
        chunk_hash = hashlib.sha256(chunk.encode("utf-8")).hexdigest()[:16]
        result = extract_with_ollama(chunk)
        for t in result.triples:
            if t.confidence < 0.70 or t.evidence_quote.lower() not in chunk.lower():
                continue
            subj = normalize_entity(t.subject)
            obj = normalize_entity(t.object)
            if subj == obj:
                continue
            src_id = f"{t.subject_type}:{subj}"
            tgt_id = f"{t.object_type}:{obj}"
            conn.execute("INSERT OR IGNORE INTO nodes VALUES (?, ?, ?)", (src_id, t.subject_type, subj))
            conn.execute("INSERT OR IGNORE INTO nodes VALUES (?, ?, ?)", (tgt_id, t.object_type, obj))
            edge_id = hashlib.sha256(f"{src_id}|{t.predicate}|{tgt_id}|{chunk_hash}".encode()).hexdigest()[:20]
            conn.execute(
                "INSERT OR REPLACE INTO edges VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
                (edge_id, src_id, t.predicate, tgt_id, t.evidence_quote, t.confidence, file_key, chunk_hash),
            )
            inserted += 1
    conn.commit()
    return inserted

def load_networkx(conn: sqlite3.Connection) -> nx.MultiDiGraph:
    g = nx.MultiDiGraph()
    for node_id, etype, name in conn.execute("SELECT node_id, entity_type, canonical_name FROM nodes"):
        g.add_node(node_id, entity_type=etype, name=name)
    for src, pred, tgt, quote, src_file in conn.execute(
        "SELECT source_id, predicate, target_id, evidence_quote, source_file FROM edges"
    ):
        g.add_edge(src, tgt, key=pred, predicate=pred, quote=quote, source_file=src_file)
    return g

Production Prompt Template: Constrained Schema Entity-Relation Extractor

When running extraction across diverse personal notes - such as research paper summaries, architecture logs, and book highlights - use this strict system prompt to prevent ontology drift and force verbatim quote grounding.

SYSTEM:
You are a deterministic Knowledge Graph Extraction Engine for a personal research archive.
Extract ONLY explicit, verifiable relationships stated in INPUT_CHUNK.

ALLOWED ENTITY TYPES:
Person, Organization, Concept, Artifact, Project, Claim, Decision

ALLOWED PREDICATES:
CREATED, DEPENDS_ON, SUPPORTS, CONTRADICTS, APPLIES_TO, MENTIONS, PART_OF, SUPERSEDES

STRICT RULES:
1. Do NOT invent entities or extrapolate beyond INPUT_CHUNK.
2. Do NOT emit generic stop-entities ("System", "Tool", "Method", "Thing", "User").
3. Resolve pronouns ("it", "they", "this library") to their explicit antecedent from HEADING_CONTEXT or return nothing if ambiguous.
4. Every triple MUST include an exact, character-for-character substring in "evidence_quote".
5. Output valid JSON matching {"triples": [...]} with no markdown fences or commentary.

INPUT:
HEADING_CONTEXT: {{parent_headings}}
SOURCE_DATE: {{note_date}}
INPUT_CHUNK:
{{chunk_text}}

Step-by-Step Operational Guide: Building Your Local Graph Workstation

Follow these operational steps to stand up a completely local, zero-cloud knowledge graph pipeline on Linux, macOS, or Windows WSL2 in under 20 minutes:

1. Isolate Your Python Environment: Create a dedicated virtual environment (python3 -m venv ~/.venvs/pkg-graph && source ~/.venvs/pkg-graph/bin/activate) and install networkx, pydantic, igraph, and leidenalg for fast local community detection.
2. Install and Lock Local Inference: Install Ollama, pull an instruction-following model with strong JSON adherence (7B to 14B parameters depending on your RAM/VRAM) plus a local embedding model such as nomic-embed-text, and verify the server binds strictly to 127.0.0.1:11434.
3. Initialize Your Ontology & Alias File: Create ontology.yaml inside your vault root listing your 7 entity types, 8-12 predicates, and a starting aliases.json mapping abbreviations you personally use (project codenames, colleague nicknames, tool acronyms).
4. Dry-Run on 20 Representative Notes: Run the extractor on a 20-note sample spanning daily logs, paper summaries, and technical specs. Inspect the generated SQLite table to verify that no generic stop-nouns or self-loops slipped through.
5. Schedule Incremental Watchers: Configure a background file watcher (using watchdog or a nightly cron/launchd timer) that hashes modified Markdown files, purges stale edges by source_file, and upserts updated triples automatically.

Production Python Pipeline: Multi-Hop GraphRAG Retriever with Citations

Once your graph is populated in SQLite and loaded into NetworkX, this retriever performs seed entity matching, executes a bounded 2-hop breadth-first traversal, ranks connected evidence edges, and passes the structured subgraph context to your local Ollama model for cited synthesis.

import json
import sqlite3
import urllib.request
from pathlib import Path
import networkx as nx

def find_seed_nodes(g: nx.MultiDiGraph, query: str, limit: int = 5) -> list[str]:
    terms = {w.lower() for w in query.split() if len(w) > 2}
    scored = []
    for node_id, data in g.nodes(data=True):
        name = data.get("name", "").lower()
        overlap = sum(1 for t in terms if t in name)
        if overlap > 0:
            scored.append((overlap, g.degree(node_id), node_id))
    scored.sort(reverse=True)
    return [n for _, _, n in scored[:limit]]

def extract_k_hop_context(g: nx.MultiDiGraph, seeds: list[str], hops: int = 2, max_edges: int = 30) -> list[str]:
    visited_nodes = set(seeds)
    frontier = set(seeds)
    for _ in range(hops):
        next_frontier = set()
        for node in frontier:
            if node in g:
                next_frontier.update(g.successors(node))
                next_frontier.update(g.predecessors(node))
        frontier = next_frontier - visited_nodes
        visited_nodes.update(frontier)

    subgraph = g.subgraph(visited_nodes)
    evidence_lines = []
    for u, v, data in list(subgraph.edges(data=True))[:max_edges]:
        line = (
            f"- ({u}) -[{data['predicate']}]-> ({v}) | "
            f"Quote: \"{data['quote']}\" [Source: {data['source_file']}]"
        )
        evidence_lines.append(line)
    return evidence_lines

def answer_with_graphrag(db_path: Path, question: str, model: str = "qwen2.5:7b") -> str:
    conn = sqlite3.connect(db_path)
    g = nx.MultiDiGraph()
    for nid, etype, name in conn.execute("SELECT node_id, entity_type, canonical_name FROM nodes"):
        g.add_node(nid, entity_type=etype, name=name)
    for src, pred, tgt, quote, src_file in conn.execute(
        "SELECT source_id, predicate, target_id, evidence_quote, source_file FROM edges"
    ):
        g.add_edge(src, tgt, predicate=pred, quote=quote, source_file=src_file)

    seeds = find_seed_nodes(g, question)
    if not seeds:
        return "No matching entities found in the local knowledge graph."

    context_lines = extract_k_hop_context(g, seeds, hops=2)
    prompt = (
        "Answer the question using ONLY the verified graph edges and quotes below. "
        "Cite the exact [Source: ...] path for every claim.\n\n"
        f"GRAPH EVIDENCE:\n" + "\n".join(context_lines) + f"\n\nQUESTION: {question}"
    )
    payload = {"model": model, "prompt": prompt, "stream": False, "options": {"temperature": 0.1}}
    req = urllib.request.Request(
        "http://127.0.0.1:11434/api/generate",
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=90) as resp:
        return json.loads(resp.read().decode("utf-8"))["response"]

Production Prompt Template: Entity Resolution and Merge Adjudicator

When your Stage 2 vector similarity check flags two entity names in the gray zone (cosine similarity between 0.80 and 0.92), run this adjudication prompt with each node's 1-hop neighborhood to decide safely whether to merge them or keep them separate.

SYSTEM:
You are an Entity Resolution Adjudicator for a Personal Knowledge Graph.
Decide whether ENTITY_A and ENTITY_B refer to the exact same real-world entity.

CRITERIA:
- Compare entity types, shared neighbors, and source context quotes.
- Acronyms, versioned tool names (e.g., "Python 3.12" vs "Python"), and spelling variants usually MERGE.
- Distinct products, distinct people sharing a last name, or homonyms across domains MUST NOT MERGE.

OUTPUT JSON ONLY:
{
  "decision": "MERGE" | "KEEP_SEPARATE",
  "canonical_name": "string",
  "reasoning": "one concise sentence"
}

ENTITY_A: {{entity_a_name}} (Type: {{entity_a_type}})
Neighbors A: {{entity_a_neighbors}}

ENTITY_B: {{entity_b_name}} (Type: {{entity_b_type}})
Neighbors B: {{entity_b_neighbors}}

Step-by-Step Operational Runbook: Graph Pruning, Snapshots, and Rebuild

Execute this monthly maintenance runbook to keep your personal knowledge graph fast, accurate, and free of semantic drift:

1. Create an Atomic SQLite Backup: Never run destructive graph cleanup on your live database. Execute sqlite3 pkg.db ".backup pkg_backup_$(date +%F).db" first.
2. Audit High-Degree Hub Nodes: Query the top 20 nodes by degree (ORDER BY degree DESC LIMIT 20). If generic terms like "Model" or "API" appear, add them to your stoplist and delete those nodes and their incident edges.
3. Purge Disconnected Orphan Nodes: Remove nodes with zero incident edges (left behind after note deletions or edits) using a single SQL DELETE FROM nodes WHERE node_id NOT IN (SELECT source_id FROM edges UNION SELECT target_id FROM edges).
4. Review Alias Merge Candidates: Inspect the weekly gray-zone queue, append confirmed synonyms to aliases.json, and rewrite incident edge IDs to point to the surviving canonical node.
5. Recompute Leiden Communities & Verify Golden Queries: Re-run community detection and summary generation, then execute five benchmark multi-hop questions from your personal test suite to confirm citation accuracy before retiring the backup.

Live Technical References

• Microsoft GraphRAG Documentation - Official architecture for hierarchical Leiden community summaries and local/global graph search.
• Neo4j Graph Database Documentation - Labeled property graph modeling, Cypher query language, and Graph Data Science guides.
• KuzuDB Embedded Graph Database - High-performance, serverless, in-process property graph database with native Cypher and vector support.
• NetworkX Documentation - Standard Python library for complex network analysis, k-hop traversal, and graph algorithms.
• Ollama Documentation - Local LLM runtime with structured JSON schema output and zero-egress execution.
• Pydantic V2 Validation Documentation - Strict runtime schema validation for LLM-extracted triples and structured outputs.
• leidenalg Documentation - Python implementation of the Leiden algorithm for guaranteed well-connected community detection.
• W3C RDF 1.1 Concepts and Abstract Syntax - Foundational specification for subject-predicate-object semantic web triples.
• Anthropic: Building Effective Agents - Engineering patterns for tool-augmented retrieval and deterministic verification loops.
• SQLite Online Backup API - Atomic snapshot and recovery guarantees for local-first single-file databases.