Batch converting documents to Markdown

The corpus these numbers come from

Nine files, 60 344 bytes: a Word handbook with a table, a spreadsheet with two sheets and 120 rows, a six-slide deck, a two-page PDF, an HTML page, a CSV export with 300 rows, a JSON schema, a Markdown runbook and a text note. They were written for this test, so the sizes are honest but the documents are not customer files. Output is measured in characters and words of the Markdown that came back.

Three routes, side by side

One request, nine files

The batch endpoint took all nine files in a single multipart request and returned in 0.153 s. The same nine files sent one at a time took 0.261 s in total, so on loopback the saving is real but modest — what a batch actually saves over a tunnel is nine round trips and nine quota checks, not nine conversions' worth of work. The response is a JSON object with count, converted, failed, skipped and a files array that carries the Markdown of each file, so one failing file does not lose the other eight.

curl -s -X POST https://llmfiles.com/api/convert/batch \
  -F [email protected] -F [email protected] \
  -D - -o batch.json | grep -i x-batch
# x-batch-total: 2
# x-batch-converted: 2
# x-quota-remaining: 3

The X-Batch-* headers repeat the four counts without parsing the body, which is the cheap way to watch a job. Per-file cost of the endpoint itself is small: twenty tiny files came back in 0.024 s. Twenty-one files are refused outright with HTTP 413 and the sentence "Too many files in one batch: 21, the limit is 20. Split the batch or send the files in a .zip to /api/convert/zip."

What came out, file by file

The CSV is the loudest line: 13.9 KB of rows became a 17.9 KB pipe table with 4 526 words, because Markdown pads every cell and repeats the separator row — faithful, and the most character-expensive shape in the corpus. The deck is the opposite: 32.6 KB in, 89 words out, since the rest was XML plumbing around six slides. A PDF returns flat text with no headings at all, which is a property of the format and is covered in the PDF guide.

The quota is per file; the archive is not

This is the part that decides which route you want. A batch charges one conversion per file that converts; an archive charges one for the whole thing. Measured on 01.11, when the daily allowance was five: from an IP with nothing used that day, a batch of seven files returned converted: 5 and skipped: 2, and the two skipped files carried the reason Daily limit reached (5 conversions per day). The request itself was HTTP 200, because five files did convert — the status only becomes 402 when nothing could be converted. Sending the same corpus as a ZIP cost one conversion: the response reported X-Quota-Used: 1 and X-Quota-Remaining: 4 for nine converted files. The allowance has been 25 a day since 01.11, so the same batch of seven now converts in full; the accounting itself did not change.

Archives: structure stays, junk is listed

A ZIP is unpacked one member at a time, each member goes through its own converter, and the folder structure is rebuilt in the output. In a test archive of 15 659 bytes with seven members, docs/reports/, docs/ and data/ were preserved, extensions were replaced with .md, and four files converted in 0.083 s. The other three were not dropped silently: the manifest lists them under "Not converted" with a reason each — __MACOSX/._runbook.md and .DS_Store as archive service files, images/logo.png as "binary file, no extractable text".

The manifest — _manifest.md at the top of the output archive — opens with the count ("Converted 4 of 7 files, skipped 3"), then a table of source name, output name and size, then the skipped entries, then a note on how the archive was produced. It is written for a person, not a script, but it is the only record of the members that did not convert. Nested archives are not opened: a ZIP inside a ZIP stays a ZIP at depth 1.

One thing the archive route will not do is appear inside a batch. Send a ZIP as a member of multipart to the batch endpoint and that member comes back failed with "Archives are not unpacked inside a batch: send this .zip on its own to /api/convert/zip." The two endpoints stay separate on purpose, so that one archive cannot quietly turn twenty files into two thousand.

When the folder is bigger than that

The CLI runs the same converter with no upload and no quota. The nine-file corpus took 1.8 s as a folder command, and a hundred copies of it took 6.2 s — about 60 ms per file on this machine, with Python's start-up on top once. Recursion is on by default, --no-clobber protects files that already exist, and --json prints a summary a build script can read.

llmfiles ~/Documents/contracts -o out/ --json
# { "ok": true, "converted": 100, "failed": 0, "skipped": 0, ... }
llmfiles ~/Documents/contracts --no-recursive --no-clobber

Wiring it into a pipeline

For a corpus that has to land in a repository or a vector store, the archive route is the easiest to script: post the ZIP, save the response, unzip it, read the manifest to see which members need attention, then commit. Everything the endpoints accept and return is on the API page, including the error codes: 413 for too many files, 402 when the daily limit leaves nothing to convert, 422 for a file with no extractable text.

Limits, as they actually are