# Converting Sitemaps and Raw HTML into Token-Efficient llms-full.txt Context Files

When developers paste a documentation URL into Cursor (`@Docs`), Claude Code, or a custom Retrieval-Augmented Generation (RAG) pipeline, feeding raw HTML into the context window is extraordinarily wasteful.

A typical documentation page rendered by Docusaurus, Nextra, or Mintlify contains:

- `120 KB` of hydrated React/Next.js `<script>` payloads and inline JSON state
- `45 KB` of SVG icon paths, top navigation bars, sidebar trees, and footer links
- Only `12 KB` of actual technical prose, tables, and code blocks

If an LLM ingests 20 pages of raw HTML, it burns through a **200,000-token context window** mostly reading `className="flex items-center justify-between"` instead of your API documentation.

Here is how to build a deterministic, in-browser **HTML-to-Clean-Markdown compiler** and **LLM Token Budget Profiler** to produce `llms.txt`, `llms-full.txt`, and `llms-ctx.txt` deliverables.

---

## 1\. Stripping Non-Content DOM Subtrees Before Markdown Serialization

Converting HTML to Markdown for LLM ingestion requires aggressive semantic pruning before converting tags:

1. **Prune Chrome &Executable Nodes:** Strip `<script>`, `<style>`, `<noscript>`, `<svg>`, `<canvas>`, `<iframe>`, `<nav>`, `<header>`, `<footer>`, `<aside>`, and elements matching `[role="navigation"]`, `[aria-hidden="true"]`, or cookie/ad container classes.
2. **Preserve Semantic Structure & Code Blocks:**
   - Convert `<h1>`–`<h6>` into ATX `#` headings.
   - Extract `<pre><code class="language-ts">` blocks into fenced Markdown code blocks (` ```ts ... ``` `) with HTML entity decoding (`&lt;` $\rightarrow$ `<`, `&amp;` $\rightarrow$ `&`).
   - Convert `<table><thead><tbody>` grids into pipe-delimited GitHub Flavored Markdown (GFM) tables so LLMs preserve row-column relationships.

---

## 2\. Profiling Context Window Utilization Across 5 Frontier Models

Once your pages are compiled into `/llms.txt` (the lightweight directory index) and `/llms-full.txt` (the full corpus), you need to know whether `/llms-full.txt` fits inside a single model prompt without truncation.

By estimating BPE token counts across code and prose segments, you can map exact context window utilization against current frontier models:

- **DeepSeek R1 / V3:** `64,000` token context window
- **Perplexity Sonar:** `127,000` token context window
- **OpenAI GPT-4o / o3:** `128,000` token context window
- **Anthropic Claude 3.7 Sonnet:** `200,000` token context window
- **Google Gemini 2.5 Pro:** `1,000,000` token context window

If your `llms-full.txt` exceeds `64k` tokens, generating a middle-tier `llms-ctx.txt` (which excludes links marked under `## Optional` such as legal policies, changelogs, and legacy v1 docs) keeps the context bundle compact enough to fit inside every major model's single-shot window.

You can test this HTML-to-Markdown compiler, 5,000-URL `sitemap.xml` parser, and multi-model token budget profiler locally in your browser using the [KivaroTools `llms.txt` & AI Crawler `robots.txt` Studio](https://kivarotools.com/seo/llms-txt-generator).