Documents for LLMs

What each format hands the model

Token counts below are tiktoken with the o200k_base encoding, the one GPT-4o-class models use. Another tokenizer will differ by a few percent; the ratios between formats will not.

Read the size column and then the token column. The 2.2 MB PDF produced 9,939 tokens; the 271 KB HTML page produced 23,560 — two and a half times more from a file a tenth the size. Bytes per token is 4.5 for the PDF and 87 for the HTML, because a PDF stores fonts and page objects while HTML repeats its own markup. Budgeting a corpus by file size gets you a number that is wrong in both directions.

Structure is data in Office files and layout in a PDF

Why the heading column splits so cleanly is not the converter being lazy. In a DOCX, an XLSX or an EPUB the author marked structure explicitly, and the converter copies those marks. In a PDF the structure is a side effect of where the ink landed, so there is nothing to copy. Measured, element by element:

• DOCX — five headings out of five heading styles, one table, lists intact. The table arrived with an empty header row, a known Word artefact explained in the DOCX guide.
• XLSX — each sheet name becomes a heading (## Revenue, ## Notes) and each sheet becomes a table with its real header row. This is the cleanest office format to ingest: 376 characters, two headings, two tables, nothing to repair.
• PPTX — four slides, four # headings: the slide title becomes the heading. Bullets do not: they arrive as separate plain lines, and the counter finds zero list items in the file.
• EPUB — a **Title:** / **Language:** / **Identifier:** preamble, then every chapter at level one. Chapter nesting is flat: a two-level book becomes a list of # siblings.
• CSV — the table and nothing else. No heading, no title, no file name: a chunk cut from it cannot say what it describes.
• JSON — the heading is the file name (# tickets.json), not the content; rows become a table whose columns are the union of every record's keys, and a null arrives as an empty cell.
• PDF — zero headings in fifteen pages. Not one #.

The slide deck is worth a second look, because it is where a reasonable expectation fails. This is the whole converted deck, 346 characters:

<!-- Slide number: 1 -->
# HX-40 service rollout
Field operations · October 2026

<!-- Slide number: 2 -->
# Why now
Gasket failures doubled in Q3
Two units offline for a week

The two bullets of slide 2 are on two lines but carry no list markers, so a chunker splitting on blank lines gets one chunk and a chunker splitting on lists gets none. The slide comments are the useful part: they are a boundary the converter did declare, so treat <!-- Slide number: N --> as a section marker when you build the corpus.

What the markup costs in tokens

The usual objection to Markdown is that the pipes and hashes waste tokens. Measured by converting each output back to the same text with the markers removed, the cost is small and it buys the only boundaries the model gets:

*For the HTML page the comparison counts link URLs separately, because they are navigation rather than structure: 17,662 tokens after removing URLs, 17,344 with the markers stripped too — 318 tokens for 32 headings, two tables and 83 code fences. Ten percent is the ceiling for a dense table; on prose it is nearer one percent. The PDF shows 0% because the PDF output has no markup to remove: there is nothing in it except text.

The tokens go to navigation, not to structure

If markup is cheap, what made the docs page 23,560 tokens? Of that total, 5,311 tokens (22.5%) sit inside the URLs of 474 relative links — the sidebar, the table of contents, the previous/next links, the Sphinx footer. Another 1,562 tokens (6.6%) are the 103 [¶](#…) permalink anchors Sphinx puts after every heading. That is 29% of the page spent telling a model where the navigation is on a website it cannot visit.

That is the whole page, as it arrives when you ask for it whole — strip_nav=false — or convert with a tool that has no cleaning pass. The converter drops this chrome from the HTML before parsing, so the same page arrives by default as 73,349 characters and 19,015 tokens with 22 headings and 222 links: the navigation above is not in that output at all.

When the page does have to stay whole, the fix is a pass over the Markdown before chunking: drop the permalinks, drop list items whose target is a relative URL, and turn external links into their text. It keeps every heading, table and fence:

import re

def strip_navigation(md: str) -> str:
    """Remove site chrome from converted Markdown, keep the document.

    Drops Sphinx permalink anchors, list items that link to relative URLs
    (sidebar, table of contents, previous/next) and external link URLs.
    Headings, tables and code fences are untouched.
    """
    md = re.sub(r"\[¶\]\(#[^)]*\)", "", md)
    md = re.sub(r"^\s*[-*+]\s+\[[^\]]*\]\((?!https?:)[^)]*\).*$", "", md, flags=re.M)
    md = re.sub(r"\[([^\]]+)\]\(https?://[^)]+\)", r"\1", md)
    return re.sub(r"\n{3,}", "\n\n", md).strip() + "\n"

On the Sphinx page that pass takes 23,560 tokens down to 21,039 — 2,521 tokens, 10.7%, removed — and the counters still find 32 headings, two tables and 83 code fences afterwards. Repeat the measurement on your own corpus before trusting it: a page whose content is mostly links, such as a changelog or a link directory, has no text left once you strip navigation, and that is a signal to convert a different page.

PDF: text with no structure at all

Here is the same measurement applied to the paper, and the whole diagnosis in one line:

• 39,662 characters in 1,059 lines — an average of 37 characters per line, so every line is a hard wrap, not a sentence.
• 14 form feeds (0x0C) standing in for page breaks, which some chunkers treat as whitespace and some as a word character.
• 14 words hyphenated across a line break, which arrive split in two.
• 33 lines of a rotated arXiv stamp at the top, extracted one letter per line — 62 tokens of 3 / 2 / 0 / 2 / g u 2 garbage before the first sentence.
• Zero headings. Fifteen pages, no structural marker anywhere.

Two of those are mechanical and worth fixing before the text reaches a model: page breaks and hard wraps.

import re

def unwrap_pdf(text: str) -> str:
    """Turn PDF text into paragraphs: pages, hyphenated words and hard wraps."""
    text = text.replace("\f", "\n\n")                 # form feed -> paragraph break
    text = re.sub(r"(\w)-\n(\w)", r"\1\2", text)      # hyphenated word across a line
    text = re.sub(r"(?<=[a-z,;:])\n(?=[a-z(])", " ", text)  # wrap inside a sentence
    return re.sub(r"\n{3,}", "\n\n", text)

Measured on the same paper: 1,059 lines become 810, the average line grows from 37 to 49 characters, all 14 hyphen joins are repaired and no form feed survives. The token count barely moves (9,939 to 9,634 — shorter text, longer words), because this changes the shape, not the size. Headings stay at zero: no post-processing invents structure that was never there. For that, split the PDF by chapter before converting or promote its numbered clauses to headings, as measured in Markdown for RAG.

A folder into a context budget: the script

The pipeline that produced the table above is this script. It sends a folder to POST /api/convert/batch twenty files per request, writes one .md per source file plus manifest.json, and prints what each document will cost you. Files with no headings are flagged, because those are the ones that will produce context-free chunks.

#!/usr/bin/env python3
"""Convert a folder of mixed documents to Markdown and report what a model will read.

    python3 tollm.py docs/ --out corpus/ --budget 128000

Sends files to POST /api/convert/batch (twenty per request), writes one .md per source
file plus manifest.json, and prints a table: characters, tokens, headings, tables, code
blocks. Documents with no headings are flagged, because a chunker cannot cut on a
boundary it cannot see. Token counts use tiktoken when installed, otherwise chars / 4.
"""
import argparse
import json
import os
import re
import sys
import urllib.error
import urllib.request
import uuid
from pathlib import Path

SUPPORTED = {
    ".pdf", ".docx", ".pptx", ".xlsx", ".xls", ".csv", ".html", ".htm", ".epub",
    ".json", ".yaml", ".yml", ".toml", ".xml", ".ini", ".env", ".rtf", ".svg",
    ".msg", ".ipynb", ".md", ".txt", ".log",
}
BATCH = 20


def encoder(name):
    try:
        import tiktoken
        enc = tiktoken.get_encoding(name)
        return (lambda text: len(enc.encode(text))), "tiktoken:" + name
    except Exception:
        return (lambda text: (len(text) + 3) // 4), "chars/4 estimate"


def post_batch(api, paths):
    """One multipart request with up to twenty files. Returns the parsed JSON."""
    boundary = uuid.uuid4().hex
    body = bytearray()
    for path in paths:
        body += (f"--{boundary}\r\n"
                 f'Content-Disposition: form-data; name="files"; filename="{path.name}"\r\n'
                 "Content-Type: application/octet-stream\r\n\r\n").encode()
        body += path.read_bytes() + b"\r\n"
    body += f"--{boundary}--\r\n".encode()
    req = urllib.request.Request(
        api.rstrip("/") + "/api/convert/batch", data=bytes(body), method="POST",
        headers={"Content-Type": f"multipart/form-data; boundary={boundary}"})
    with urllib.request.urlopen(req, timeout=600) as resp:
        return json.loads(resp.read())


def stats(md, count_tokens):
    lines = md.split("\n")
    tables = sum(1 for line in lines
                 if line.lstrip().startswith("|")
                 and re.fullmatch(r"\|?[\s:|-]+\|?", line.strip()) and "-" in line)
    return {
        "chars": len(md),
        "tokens": count_tokens(md),
        "headings": sum(1 for line in lines if re.match(r"^#{1,6} ", line)),
        "tables": tables,
        "fences": md.count("```") // 2,
    }


def main():
    ap = argparse.ArgumentParser(description="Convert a folder for a model.")
    ap.add_argument("src", type=Path, help="folder with documents")
    ap.add_argument("--out", type=Path, default=Path("markdown"), help="where .md files go")
    ap.add_argument("--api", default=os.environ.get("LLMFILES_API", "https://llmfiles.com"))
    ap.add_argument("--budget", type=int, default=0,
                    help="context budget in tokens; reports how many documents fit")
    ap.add_argument("--encoding", default="o200k_base", help="tiktoken encoding")
    args = ap.parse_args()

    files = sorted(p for p in args.src.rglob("*")
                   if p.is_file() and p.suffix.lower() in SUPPORTED
                   and not p.name.startswith("."))
    if not files:
        sys.exit(f"no supported documents under {args.src}")

    count_tokens, token_source = encoder(args.encoding)
    args.out.mkdir(parents=True, exist_ok=True)
    manifest, failed = [], []

    for start in range(0, len(files), BATCH):
        chunk = files[start:start + BATCH]
        try:
            data = post_batch(args.api, chunk)
        except urllib.error.HTTPError as exc:
            detail = exc.read().decode("utf-8", "replace")[:300]
            sys.exit(f"API error {exc.code} on batch {start // BATCH + 1}: {detail}")
        for item in data.get("files", []):
            name = item.get("filename") or "file"
            if not item.get("ok"):
                failed.append({"source": name,
                               "reason": item.get("error") or "no text extracted"})
                continue
            md = item["markdown"]
            target = args.out / (Path(name).stem + ".md")
            n = 2
            while target.exists():
                target = args.out / f"{Path(name).stem}-{n}.md"
                n += 1
            target.write_text(md, encoding="utf-8")
            record = {"source": name, "markdown": str(target)}
            record.update(stats(md, count_tokens))
            manifest.append(record)

    manifest.sort(key=lambda r: r["source"])
    (args.out / "manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8")

    print(f"{'document':20} {'chars':>7} {'tokens':>7} {'head':>5} {'tbl':>4} {'code':>5}  note")
    total = 0
    for r in manifest:
        total += r["tokens"]
        note = "no headings" if r["headings"] == 0 else ""
        print(f"{Path(r['source']).name[:20]:20} {r['chars']:>7} {r['tokens']:>7} "
              f"{r['headings']:>5} {r['tables']:>4} {r['fences']:>5}  {note}")
    for f in failed:
        print(f"{Path(f['source']).name[:20]:20} {'-':>7} {'-':>7} {'-':>5} {'-':>4} {'-':>5}  "
              f"failed: {f['reason'][:60]}")

    print(f"\n{len(manifest)} converted, {len(failed)} failed, {total} tokens "
          f"({total / max(1, len(manifest)):.0f} per document), tokens: {token_source}")
    empty = [r["source"] for r in manifest if r["headings"] == 0]
    if empty:
        print(f"{len(empty)} document(s) have no headings at all: {', '.join(empty)}")
    if args.budget:
        used, fits = 0, 0
        for r in manifest:
            if used + r["tokens"] > args.budget:
                break
            used += r["tokens"]
            fits += 1
        print(f"budget {args.budget}: {fits} of {len(manifest)} documents fit "
              f"({used} tokens, {100 * used // args.budget}% of the window)")


if __name__ == "__main__":
    main()

Run on the eight probe documents with a 32,768-token budget, it prints exactly this:

document               chars  tokens  head  tbl  code  note
handbook.docx            666     153     5    1     0
notes.epub               465     122     3    1     0
paper.pdf              39662    9939     0    0     0  no headings
revenue.xlsx             376     164     2    2     0
rollout.pptx             346      92     4    0     0
sensors.csv              231     101     0    1     0  no headings
spec.html              89649   23560    32    2    83
tickets.json             319     103     1    1     0

8 converted, 0 failed, 34234 tokens (4279 per document), tokens: tiktoken:o200k_base
2 document(s) have no headings at all: paper.pdf, sensors.csv
budget 32768: 6 of 8 documents fit (10571 tokens, 32% of the window)

The budget line is the part worth reading twice. Six of eight documents fit in 10,571 tokens, and the seventh — the paper — would have pushed the total past the window on its own at 9,939. A single fifteen-page PDF costs as much context as five office documents and a documentation page, and it brings no headings, so the chunks cut from it will not know what they are about.

Before you embed a corpus

• Convert, do not paste. A pasted PDF keeps its 37-character hard wraps; a converted one can be unwrapped.
• Count headings per format on one real file before batch-converting. Zero means every chunk from that format will be context-free.
• Prefer the Word or Excel original over its PDF export: the structure is data in one and a drawing in the other, and that difference is the whole heading column above.
• Strip navigation before chunking, and re-measure afterwards: 10.7% of the docs page was sidebar.
• Budget in tokens, not megabytes. Bytes per token ranged from 4.5 to 87 in this set.
• Flag headings that are file names (# tickets.json) — they say nothing about the content and should be replaced with a real title.
• Scanned PDFs have no text layer at all: the endpoint answers 422 and asks for OCR instead of returning bytes.