with classes
→ Must infer content from class names
→ "content", "main", "article" are hints
→ But no standards, site-specific
```
### CSS Display and Visibility
HTML content may be visually hidden:
**Display: None:**
```html
Mobile menu (hidden on desktop)
```
**Extraction Issue:**
```
Text extraction sees:
→ "Mobile menu (hidden on desktop)"
→ Even though visually hidden
Should extracted text:
→ Include hidden content? (it exists in DOM)
→ Exclude hidden content? (user doesn't see it)
Use case dependent:
→ Mobile menu: Exclude (duplicate of main nav)
→ Spoiler/accordion: Include (real content, just collapsed)
```
**Visibility: Hidden vs Opacity:0:**
```html
Content A
Content B
Screen reader only text
All invisible to sighted users
→ visibility:hidden: Layout space reserved
→ opacity:0: Transparent but present
→ sr-only: For accessibility
Should extract:
→ sr-only: Yes (valuable alt text)
→ others: Debatable
```
### Navigation and UI Elements
Page chrome contamination:
**Navigation Extraction:**
```html
Text extraction: "Home About Products"
Appears in every page:
→ 50 pages on site
→ All have "Home About Products" in extracted text
→ Repetitive noise
→ Dilutes unique content signal
```
**Button and Link Text:**
```html
Click here
Learn more
Read documentation →
Extracted: "Click here Learn more Read documentation →"
Out of context:
→ "Click here" meaningless (click where?)
→ "Learn more" vague (learn about what?)
→ Arrow "→" is decorative (visual only)
Better to:
→ Extract link destination as context
→ "Sign up (Learn more)"
→ "Documentation (Read documentation)"
```
### Forms and Input Fields
Form elements have special extraction needs:
**Form HTML:**
```html
```
**Extraction Variants:**
```
Option 1: Extract labels only
"Email: Submit"
Option 2: Extract labels + placeholders
"Email: you@example.com Submit"
→ Placeholder looks like content (wrong)
Option 3: Skip forms entirely
(no text extracted)
Best practice:
→ Extract labels (field names)
→ Skip inputs, buttons, placeholders
→ "Email:" is content
→ "you@example.com" is just hint
```
### Script Tags and Style Blocks
Non-content elements:
**JavaScript Inline:**
```html
```
**Text Extraction:**
```
Naive extraction includes:
"function trackEvent() { analytics.send('pageview'); }"
This is code, not content!
→ Should be excluded
→ But: How to distinguish from
blocks (which should be included)?
Solution:
→ Strip
```
**Extraction Opportunity:**
```
Structured data provides:
→ Article title
→ Author
→ Date
→ Other metadata
Can augment extracted text:
"Getting Started Guide (by John Smith, published 2024-01-15)"
Adds context beyond visible text
```
### Single Page Applications (SPAs)
JavaScript-rendered content:
**Initial HTML (before JS):**
```html
```
**After JavaScript Executes:**
```html
Welcome
Actual content rendered by React...
```
**The Empty Shell Problem:**
```
HTTP request fetches:
→ Empty
→ No content visible
Text extraction:
→ Nothing to extract!
Solution required:
→ Headless browser (Puppeteer, Playwright)
→ Execute JavaScript
→ Wait for content to load
→ Then extract
10-100x slower than static HTML extraction
→ But necessary for SPAs
```
***
## How to Solve
**Use semantic HTML tags to identify content areas (article, main) + strip navigation, headers, footers + exclude display:none elements + extract alt text from images + use headless browser for JavaScript-rendered content + convert tables to markdown format.** See [HTML Extraction](/rag-scenarios-and-solutions/chunking/html-conversion.md).
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://help.twig.so/rag-scenarios-and-solutions/chunking/html-conversion.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.