Brendan Long's Shortform

Brendan Long

This is a special post for quick takes by Brendan Long. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I thought everyone serving Markdown content for LLMs would make make it easier to write save-for-later / readability apps. After actually trying this, it's still easier to extract content from HTML than to guess how people want their Markdown rendered. Some sites include navigation in the Markdown^[1], where it's nearly impossible to extract^[2], and others use obscure extensions^[3].

I guess there's something to be said about standards which are actually standard.

^{^}
I think Cloudflare's automatic Markdown converter is doing this.
^{^}
How do you know the difference between site navigation and a table of contents?
^{^}
And even if I wanted to support every extension, they can conflict with each other.

2dynomight4mo

I'm a little bit out of the loop... How and why would you serve markdown content for LLMs?

4Brendan Long4mo

Markdown has a lot fewer tokens since it's less verbose and is usually just the text in a page (on some pages the actual text is a minority of the bytes). LLM tooling typically requests the Markdown version of pages (with a fallback to HTML), and the easiest way to support that is with HTTP-to-Markdown processing like what Cloudflare does. And people who write their blogs in Markdown anyway sometimes just serve that version. I assume that's what Gwern is doing, and I'm considering doing it this way too.

sometimes just serve that version. I assume that’s what Gwern is doing

Correct. I serve Markdown in two ways: first, every page includes the HTML-standardized metadata, which points to the Markdown file version of pages (using a simple template line: <link rel="alternate" type="text/markdown" href="https://gwern.net$url$.md" title="Markdown source of ‘$title-plain$’ page">); second, I support standard HTTP content-negotiation, so any HTTP request which header specifies anywhere in it that it would accept a text/markdown or text/plain result will th... (read more)

Brendan Long's Shortform

6

Brendan Long's Shortform

6