Crawlee for web scraping: what breaks in production

Crawlee is an open-source library for web crawling and scraping. Its docs say it "covers your crawling and scraping end-to-end": fast HTTP crawling with Cheerio or JSDOM, real browsers with Playwright or Puppeteer behind the same interface, a persistent request queue, retries and sessions. It gets a crawler running quickly. Keeping it running is the hard part.

Below is how Crawlee works, what breaks around crawlers in production, and the checks that catch it.

What is Crawlee and which crawler should you pick?

Crawlee handles the plumbing every crawler needs: a queue of URLs that survives restarts, retries, concurrency limits and sessions. You write the handler that turns a page into a record.

The main choice is the crawler class:

Both use the same router and request queue. You can start with Cheerio and move a single difficult source to a browser later.

What does a minimal Crawlee crawler look like?

A router with two handlers: list pages enqueue detail pages, detail pages save a record.

import { CheerioCrawler, createCheerioRouter, Dataset } from 'crawlee'

const router = createCheerioRouter()

router.addDefaultHandler(async ({ enqueueLinks }) => {
  await enqueueLinks({ selector: 'a.listing', label: 'DETAIL' })
})

router.addHandler('DETAIL', async ({ request, $ }) => {
  await Dataset.pushData({
    url: request.loadedUrl,
    title: $('h1').text().trim(),
    price: $('.price').text().trim(),
  })
})

const crawler = new CheerioCrawler({
  requestHandler: router,
  maxConcurrency: 4,
  maxRequestRetries: 2,
  failedRequestHandler: async ({ request }) => console.error('gave up', request.url),
})

await crawler.run(['https://example.com/listings'])

A production crawler needs one more idea. It starts from sitemaps and remembers where the last run stopped, so each run fetches only what changed since then. A full walk of a big catalog on every run is how a crawler hits its time limit.

What breaks in production?

None of these are Crawlee bugs. They are what happens around any crawler that runs every day:

The first row is the most treacherous: a crawl that runs green but brings almost no new records. Monitoring should start with exactly that case.

How do you monitor a Crawlee crawler?

Crawlee tells you about requests that failed. After its retries a request lands in failedRequestHandler, and you can log or count it. It does not tell you that a run which succeeded brought nothing. That check is yours.

Compare each run with the median of recent runs, not with a fixed number. A source that usually brings many new records and suddenly brings a handful is a failure, even when every request succeeded. A run still marked running long after its time limit is a failure too, because a killed process never writes its own end.

How do you make a Crawlee crawler production-ready?

The first step is about manners and the law. A crawler should introduce itself with its own name and a link to a page that explains who is collecting and why. If a source's terms forbid automation, do not collect from it. What is legal in scraping, and when a crawler pays off at all, is in Web scraping for business: when a crawler pays off.

When is Crawlee the wrong tool?

• The source has an API or an export. Use it. It is more stable and its terms are written down.
• You need the data once. A one-off script or copying by hand is enough.
• The data is behind a login, or the terms forbid collection. Agree on access with the source instead.
• Your team writes Python. Take Scrapy or Crawlee for Python. A crawler in a language nobody maintains dies with the first redesign.

The pattern behind most such incidents is the same one we wrote about in Claude Code hooks that silently do nothing: green does not mean it worked.

Is your crawler production-ready?

Need a crawler that tells you when it stops?

If you regularly need data the source allows you to collect, we build you collection with monitoring that alerts you when it stops working. We do it as part of business process automation. Since 2023 we have delivered 15+ production projects.

We get back to you within two business days with a few questions about your work. Then we agree on what we do first.

If you'd rather talk first, book a 15-minute intro call.

How would you find out today that your crawler brought nothing last night?

Frequently asked questions

What is Crawlee?

An open-source library for web crawling and scraping. It gives one interface for fast HTTP crawling with HTML parsers and for real browsers with Playwright or Puppeteer, plus a persistent request queue, retries and sessions.

CheerioCrawler or PlaywrightCrawler?

Start with CheerioCrawler. It downloads the HTML and parses it, which is fast and light. Move a source to PlaywrightCrawler only when its pages render content with JavaScript or need clicks. Both use the same router and queue.

Crawlee or Scrapy?

Pick the language your team will maintain, because a crawler needs fixing again and again. A TypeScript team takes Crawlee. A Python team takes Scrapy, or Crawlee for Python.

Why does a Crawlee crawler stop working?

The site changes or blocks you, or the run itself fails: a time limit, a killed process, a schedule that is not what you think. The worst case is a run that ends green and saves nothing. That is why you check the yield after every run.

How do you monitor a crawler?

After every run, check that it ran, how many new records it brought compared with recent runs, and that the key fields are filled. Alert a named person. A run still marked running long after its time limit counts as failed.