# webcrawling

- Page: https://shared.btbytes.com/p/webcrawling
- Date: 2024-08-29
- Tags: webcrawling

[mendableai/firecrawl: 🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.](https://github.com/mendableai/firecrawl) is an API service. The github repo is the examples.

> Crawls all accessible subpages and give you clean data for each. No sitemap required. The greatest benefit is that the extracted data is catered for LLM-based pipelines -- [via X/rohanpaul_ai](https://x.com/rohanpaul_ai/status/1829158662964691159).

![](https://files.btbytes.com/images/20224/08/firecrawl.webp)