Converting Sitemaps and Raw HTML into Token-Efficient llms-full.txt Context Files

When developers paste a documentation URL into Cursor (@Docs), Claude Code, or a custom Retrieval-Augmented Generation (RAG) pipeline, feeding raw HTML into the context window is extraordinarily wasteful.

A typical documentation page rendered by Docusaurus, Nextra, or Mintlify contains:

• 120 KB of hydrated React/Next.js <script> payloads and inline JSON state
• 45 KB of SVG icon paths, top navigation bars, sidebar trees, and footer links
• Only 12 KB of actual technical prose, tables, and code blocks

If an LLM ingests 20 pages of raw HTML, it burns through a 200,000-token context window mostly reading className="flex items-center justify-between" instead of your API documentation.

Here is how to build a deterministic, in-browser HTML-to-Clean-Markdown compiler and LLM Token Budget Profiler to produce llms.txt, llms-full.txt, and llms-ctx.txt deliverables.

1. Stripping Non-Content DOM Subtrees Before Markdown Serialization

Converting HTML to Markdown for LLM ingestion requires aggressive semantic pruning before converting tags:

1. Prune Chrome &Executable Nodes: Strip <script>, <style>, <noscript>, <svg>, <canvas>, <iframe>, <nav>, <header>, <footer>, <aside>, and elements matching [role="navigation"], [aria-hidden="true"], or cookie/ad container classes.
2. Preserve Semantic Structure & Code Blocks:
   • Convert <h1>–<h6> into ATX # headings.
   • Extract <pre><code class="language-ts"> blocks into fenced Markdown code blocks (```ts ... ```) with HTML entity decoding (< \(\rightarrow\) <, & \(\rightarrow\) &).
   • Convert <table><thead><tbody> grids into pipe-delimited GitHub Flavored Markdown (GFM) tables so LLMs preserve row-column relationships.

2. Profiling Context Window Utilization Across 5 Frontier Models

Once your pages are compiled into /llms.txt (the lightweight directory index) and /llms-full.txt (the full corpus), you need to know whether /llms-full.txt fits inside a single model prompt without truncation.

By estimating BPE token counts across code and prose segments, you can map exact context window utilization against current frontier models:

• DeepSeek R1 / V3: 64,000 token context window
• Perplexity Sonar: 127,000 token context window
• OpenAI GPT-4o / o3: 128,000 token context window
• Anthropic Claude 3.7 Sonnet: 200,000 token context window
• Google Gemini 2.5 Pro: 1,000,000 token context window

If your llms-full.txt exceeds 64k tokens, generating a middle-tier llms-ctx.txt (which excludes links marked under ## Optional such as legal policies, changelogs, and legacy v1 docs) keeps the context bundle compact enough to fit inside every major model's single-shot window.

You can test this HTML-to-Markdown compiler, 5,000-URL sitemap.xml parser, and multi-model token budget profiler locally in your browser using the KivaroTools llms.txt & AI Crawler robots.txt Studio.