Getting Started
This is the actual content...
This is the content...
``` **Modern SPA (hard to scrape):** ```html HTTP Request → Server returns: Actual content lives in app.js: → Browser downloads JavaScript → JavaScript executes → React renders content dynamically → DOM populated with, etc. ``` **Why This Breaks Scraping:** ``` Standard HTTP Scraper: 1. GET https://docs.company.com/api-reference 2. Receive HTML response 3. Parse HTML with BeautifulSoup/Cheerio 4. Extract text from
Result: ← Empty! The actual content never appears because: → Scraper doesn't execute JavaScript → No browser engine to render React components → Only sees the empty shell HTML ``` **The Headless Browser Requirement:** ``` To scrape JavaScript sites: 1. Launch headless Chrome (Puppeteer/Playwright) 2. Navigate to URL 3. Wait for JavaScript to execute 4. Wait for DOM to fully render 5. Extract rendered HTML 6. Parse content Cost: → 50-100x slower than HTTP-only scraping → Requires Chrome/Chromium binary → High memory usage (each page = browser instance) → Prone to timeouts and crashes ``` ### Robots.txt and Crawl Restrictions Many sites explicitly block scrapers via `robots.txt`: **Example robots.txt:** ``` User-agent: * Disallow: /admin/ Disallow: /internal/ Disallow: /api/ Crawl-delay: 10 User-agent: GPTBot Disallow: / User-agent: anthropic-ai Disallow: / ``` **The Compliance Dilemma:** ``` Ethical Scraping: → Must respect robots.txt → If /api-reference blocked → skip it → If crawl-delay: 10 → wait 10 seconds between requests But: → User expects ALL documentation in knowledge base → "Why isn't /api-reference indexed?" → "The scraper is broken!" Reality: It's obeying robots.txt User wants: Override restrictions Twig must: Follow legal/ethical guidelines ``` **The Dynamic Robots.txt Problem:** ``` Scenario: 1. Twig crawls site at 10:00 AM 2. robots.txt allows everything 3. 500 pages scraped successfully 4. Site owner adds GPTBot block at 2:00 PM 5. Next sync at 3:00 PM 6. Twig re-crawls, now blocked Result: → Previous 500 pages stay in vector DB → New pages can't be added → Knowledge base becomes stale → User sees "sync stopped working" ``` ### Infinite Crawl and Link Cycle Detection Web crawlers can get stuck in infinite loops: **The Pagination Problem:** ``` Blog with infinite scroll: /blog → /blog?page=2 → /blog?page=3 → ... → /blog?page=999 Each page has "Next →" link Crawler follows every link Never terminates Or: /search?q=test → /search?q=test&sort=date → /search?q=test&sort=date&page=2 Each URL is unique, but content is identical ``` **Cycle Detection Challenge:** ``` URL normalization issues: → https://docs.company.com/guide → https://docs.company.com/guide/ → https://docs.company.com/guide?ref=sidebar → https://docs.company.com/guide#overview Are these the same page? To a crawler: → Different URLs (should crawl all) To a human: → Same content (only crawl once) Deduplication logic must: 1. Normalize URLs (remove trailing slash, query params, anchors) 2. Compare content hashes 3. Skip if already seen ``` **The Subdomain Explosion:** ``` Start URL: https://docs.company.com Crawler finds links: → https://api.company.com (different subdomain) → https://blog.company.com (different subdomain) → https://status.company.com (different subdomain) → https://company.com (main site) Should crawler follow these? → Yes? Might scrape entire company website (thousands of pages) → No? Might miss important API documentation on api.company.com User expects: "Just crawl the docs" Reality: "Docs" spans 5 subdomains ``` ### Authentication and Gated Content Many documentation sites require login: **The Auth Wall:** ``` Public docs: https://docs.company.com/getting-started Internal docs: https://docs.company.com/internal (401 Unauthorized) ``` **HTTP Authentication Methods:** ``` 1. Basic Auth: Authorization: Basic base64(username:password) 2. Cookie-based session: → Login at /login → Receive session cookie → Include cookie in subsequent requests 3. JWT token: Authorization: BearerThis is the actual content...