Firecrawl Setup: Turn Any Website Into LLM-Ready Data

Firecrawl is the API that turns any website into clean, structured markdown an LLM can actually use, handling the JavaScript rendering, proxies and parsing that make DIY scraping miserable. With more than 146,000 GitHub stars it is one of the most popular AI-infrastructure projects in existence, and it is the plumbing behind countless RAG pipelines and agents. This tutorial gets you scraping a page into markdown in a few minutes, with no scraping code of your own.

• Firecrawl returns clean markdown, not raw HTML, so pages drop straight into a prompt or a vector store.
• Get started in minutes: sign up for an API key at firecrawl.dev, then call the API by curl, Python, or Node.
• Beyond scrape it can search the web, crawl whole sites, and interact (click, navigate) with pages.
• The core is open source (AGPL-3.0) and self-hostable; the hosted API is the zero-ops path.

What is Firecrawl and why is it so popular?

Firecrawl is a web data API built for AI. Give it a URL and it renders the page, strips the noise, and hands back markdown ready for a model or an embedding pipeline. It also does more than single pages: a search endpoint finds information across the web, a crawl endpoint walks an entire site, and an interact capability can click and navigate pages that hide content behind actions. Its popularity comes from removing a universally hated chore. Every team building RAG or agents needs clean web text, almost nobody wants to maintain a scraping stack, and Firecrawl turns that stack into a single call.

RelatedStrix Setup: Run an AI Penetration Tester on Your Code

How do you scrape your first page?

Sign up at firecrawl.dev to get an API key (it looks like fc-...). The fastest smoke test needs no install at all, just curl:

curl -X POST 'https://api.firecrawl.dev/v2/scrape' \
  -H 'Authorization: Bearer fc-YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{ "url": "firecrawl.dev" }'

In a real project you will use an SDK. The Python and Node clients are a two-line setup:

# Python
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
result = app.scrape("firecrawl.dev")

// Node
import { Firecrawl } from 'firecrawl';
const app = new Firecrawl({ apiKey: "fc-YOUR_API_KEY" });
await app.scrape("firecrawl.dev");

What can it do beyond a single page?

The same client exposes the bigger operations. search runs a web query and returns results already scraped to markdown, which is a one-call replacement for "search then fetch then clean." crawl walks a site from a starting URL and returns every page, useful for ingesting an entire docs site into a knowledge base. The interact flow lets an agent operate a page: scrape a site, then send prompts like "search for mechanical keyboard" and "click the first result" against the returned session. For RAG, scrape and crawl are the workhorses; for agents that need to act, interact is the interesting one.

What are the gotchas?

Three. The hosted API is a paid service beyond its free allowance, so a large crawl has a cost, and you should cache aggressively rather than re-scraping the same URLs. The self-hosted option is real but licensed AGPL-3.0, which has copyleft obligations that matter if you fold it into a proprietary product, so read the license before you build on it. And scraping is subject to the target site's terms and to rate limits, so be a good citizen: respect robots, throttle, and do not hammer sites you do not own.

RelatedSet Up OpenAI Codex Inside Claude Code

Hosted API or self-hosted, which should you pick?

The choice comes down to volume, sensitivity, and how much ops you want to own. The hosted API is right for almost everyone starting out: no infrastructure, a free tier to prototype on, and someone else keeping the scrapers working as sites change their defenses. You reach for self-hosting in three situations. When volume gets large enough that per-request pricing dominates your costs and running your own becomes cheaper. When the URLs you scrape are sensitive and you do not want them touching a third-party service. And when you need to customize behavior beyond what the API exposes. The catch on self-hosting is the AGPL-3.0 license, which carries copyleft obligations that matter the moment you fold Firecrawl into a product you distribute, so read it before you commit. For most teams the honest answer is: start hosted, measure your real volume and sensitivity, and only move to self-hosted when the numbers or the compliance requirements actually push you there.

Our take

Firecrawl won its enormous star count by being the least glamorous, most necessary piece of the AI stack: clean web text on demand. The markdown-first output is the right default, because it is what models actually want, and the search and crawl endpoints quietly replace a whole category of glue code. The two things to keep front of mind are cost on big jobs and the AGPL terms if you self-host into a closed product. For prototyping an agent or a RAG pipeline this weekend, the hosted free tier plus three lines of SDK code is the fastest path from a URL to usable data that exists right now.

Original analysis by GenZTech. Tool documentation: firecrawl/firecrawl on GitHub.