Markdown for RAG

What the converter actually hands you

Markdown is worth converting to because it is the only text format that keeps the author's structure. How much of it survives depends entirely on the input format.

HTML and DOCX keep what the author typed: h1 becomes #, a table becomes a pipe table, a pre block becomes a fence. PDF keeps nothing. Both PDFs came back as text and only text: forty pages of clauses produced zero # lines, hard line breaks where the page broke them, and thirty-nine form feeds (0x0C) standing in for page boundaries.

Headings are the only boundary the document declares itself

A heading is the author saying that the subject changed, which is the question a chunker is trying to answer. Cut there and every chunk starts with the words that make it findable on its own; keep the path and the chunk still knows which section it belongs to:

{"id": "manual.md#1", "source": "manual.md",
 "heading": "Field service handbook > Safety", "chars": 181,
 "text": "## Safety\n\nIsolate the unit from mains power and wait two minutes for the capacitors to discharge. Wear gloves when handling the heat exchanger."}

A 181-character chunk on its own says nothing about which manual it came from, so a query has to match the body text alone. With the path in the metadata the same chunk is findable by its section as well as by its content.

Fixed windows cut where there is no boundary

A thousand characters is the default in most tutorials. On the HTML page that produced three chunks, the boundary at character 1,000 landed inside the only table, and both internal boundaries landed mid-sentence. On the forty-page PDF the same window produced sixteen chunks with fourteen of its fifteen boundaries inside a sentence.

A cut table is worse than a cut sentence: the second half has no header row, so it renders as loose lines full of pipes and the model reads rows without column names. Overlap does not repair it — a hundred characters of overlap added two hundred duplicated characters, nine percent of the document, and still cut the same table.

Blank lines are a boundary only sometimes

Splitting on an empty line is the other common default. On the DOCX handbook it produced ten chunks averaging 89 characters, and five of them are a heading with nothing under it: in converted Markdown the heading and its paragraph are separated by a blank line. The HTML page broke into nineteen fragments of 112 characters on average, with the table row block intact at 432.

On the PDFs the same split behaves differently: eighty fragments averaging 193 characters, thirty-nine of them beginning with a form feed, and not one carrying a section name — because the PDF produced no section names to carry.

Cutting on headings: the script

The chunker below always starts a chunk at a heading, never cuts a table or a fenced block, splits an oversized section at paragraph boundaries instead of mid-sentence, and reports the heading path each chunk was cut from.

#!/usr/bin/env python3
"""Markdown to retrieval chunks: headings are boundaries, tables and code stay whole.

    python3 chunk.py handbook.md                 # one chunk per section, plain text
    python3 chunk.py handbook.md --max 1200      # smaller chunks
    python3 chunk.py handbook.md --jsonl out.jsonl  # metadata for a vector store

Every chunk carries the heading path it came from, so a chunk knows what it is about even
when the heading itself is in a different chunk.
"""

import argparse
import json
import re
import sys

HEADING = re.compile(r"^(#{1,6})\s+(.*)$")
FENCE = re.compile(r"^\s{0,3}(`{3,}|~{3,})")


def units(text):
    """Split Markdown into atomic units: (offset, kind, text)."""
    out, lines, pos, i = [], text.splitlines(keepends=True), 0, 0
    while i < len(lines):
        line, start = lines[i], pos
        if not line.strip():
            pos += len(line)
            i += 1
            continue
        if HEADING.match(line):
            out.append((start, "heading", line))
            pos += len(line)
            i += 1
            continue
        fence = FENCE.match(line)
        if fence:
            mark = fence.group(1)
            block = line
            pos += len(line)
            i += 1
            while i < len(lines):
                block += lines[i]
                pos += len(lines[i])
                closed = FENCE.match(lines[i]) and lines[i].strip().startswith(mark)
                i += 1
                if closed:
                    break
            out.append((start, "fence", block))
            continue
        if line.lstrip().startswith("|"):
            block = ""
            while i < len(lines) and lines[i].lstrip().startswith("|"):
                block += lines[i]
                pos += len(lines[i])
                i += 1
            out.append((start, "table", block))
            continue
        block = ""
        while i < len(lines) and lines[i].strip() and not HEADING.match(lines[i]) \
                and not FENCE.match(lines[i]) and not lines[i].lstrip().startswith("|"):
            block += lines[i]
            pos += len(lines[i])
            i += 1
        out.append((start, "para", block))
    return out


def chunk(text, max_chars=1500):
    """-> [{"start", "end", "heading"}] — slices of the source, nothing invented."""
    chunks, path, cur = [], [], None
    for start, kind, body in units(text):
        if kind == "heading":
            head = HEADING.match(body)
            level, title = head.group(1), head.group(2)
            while path and path[-1][0] >= len(level):
                path.pop()
            path.append((len(level), title.strip()))
            if cur:
                chunks.append(cur)
            cur = {"start": start, "end": start + len(body),
                   "heading": " > ".join(t for _, t in path)}
            continue
        if cur is None:
            cur = {"start": start, "end": start, "heading": ""}
        if cur["end"] > cur["start"] and cur["end"] - cur["start"] + len(body) > max_chars:
            chunks.append(cur)
            cur = {"start": start, "end": start, "heading": cur["heading"]}
        cur["end"] = start + len(body)
    if cur:
        chunks.append(cur)
    return chunks


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("source")
    ap.add_argument("--max", type=int, default=1500, help="soft limit in characters")
    ap.add_argument("--jsonl", help="write JSON Lines here instead of stdout")
    args = ap.parse_args()

    text = open(args.source, encoding="utf-8").read()
    parts = chunk(text, args.max)
    if args.jsonl:
        with open(args.jsonl, "w", encoding="utf-8") as fh:
            for n, part in enumerate(parts):
                body = text[part["start"]:part["end"]].strip()
                fh.write(json.dumps({
                    "id": f"{args.source}#{n}",
                    "source": args.source,
                    "heading": part["heading"],
                    "offset": part["start"],
                    "chars": len(body),
                    "text": body,
                }, ensure_ascii=False) + "\n")
        print(f"{len(parts)} chunks -> {args.jsonl}", file=sys.stderr)
        return
    for n, part in enumerate(parts):
        if n:
            print("\n" + "-" * 60 + "\n")
        print(text[part["start"]:part["end"]].strip())


if __name__ == "__main__":
    main()

Measured on the same four files with a 1,500-character cap: the HTML page became eight chunks averaging 268 characters with no table or code block cut, the DOCX handbook five averaging 180, the two-page contract one chunk of 1,136, the forty-page PDF twelve averaging 1,300 with no heading path. Run it as python3 chunk.py manual.md, or add --jsonl out.jsonl for records with metadata.

When there are no headings, promote what looks like one

Clause numbers are structure the PDF never marked as structure. One substitution on the forty-page output — lines matching ^(\d+)\.\s+([A-Z][^\n]{2,60})$ rewritten to ## \1. \2 — turned twelve chunks averaging 1,300 characters with no heading path into forty chunks averaging 391, every one of them headed. Not a word of text changed; only the boundaries moved. Do it only where the pattern really is a heading: a wrong heading misleads retrieval more than a missing one.

A checklist for a corpus

• Convert to Markdown, not to plain text: the headings, tables and fences are what you are buying.
• Count headings in one file per format before batch-converting. Zero means every chunk will be context-free.
• Cut on headings with a character cap, not on characters alone.
• Never split a table or a fenced block; a table that exceeds the cap stays whole.
• Keep the heading path, and prepend the document title when chunks travel between collections.
• Strip boilerplate first: the HTML page's navigation became a chunk of its own.
• Replace form feeds in PDF output — text.replace("\f", "\n\n") — if you split on blank lines.
• Scanned PDFs have no text layer at all: the converter answers 422 and asks for OCR.

Office formats keep structure best for retrieval: DOCX, PPTX, XLSX, EPUB and HTML all come through with headings or tables a chunker can see. When the source is PDF, count the headings before you embed anything.