Firecrawl

Firecrawl crawls a site and converts its content into clean markdown formatted specifically for AI context windows, handling dynamic pages, authentication, and rate limiting so developers don't have to build their own crawler from scratch. It's become a common piece of infrastructure behind RAG pipelines and research agents that need real web content, not just a stale static dataset.