Crawling a whole site
Some data is spread over hundreds of pages no single list links to — every product, every article, every listing. Ask for “every product page of the shop” and the plan gains a crawl step: the program follows the site’s links, recognises the pages that hold your data, and reads each one with the same fields.
- Ask for a crawl and recognise its step in the plan
- Read the page groups and which one the scraper uses
- Know what a crawl costs and when it stops
- Continue a stopped crawl and inspect its requests
Groups of pages
During the build, a short discovery crawl (up to 300 pages, never charged) looks at the site and sorts the addresses it finds into groups that share a shape — /item/12, /item/13 and so on. The verifier picks the group that holds what you asked for, the build reads one sample page of it, and the printed program loops over every page of that group.
CRAWL_START = "https://shop.example.com/category/{{category}}"
TARGETS = ["/item/*"]
# … inside main():
with waiso.step("Read every product page", page, mark=ROW_SELECTOR):
for p in waiso.crawl(page, CRAWL_START, targets=TARGETS, inputs=inputs):
rows += waiso.extract(p.page, ROW_SELECTOR, FIELDS, LISTS)Pages are fetched with plain requests whenever the site allows, which is many times faster and cheaper than opening each one in a browser. Only the start of the walk — a search to submit, a filter to choose — needs the browser, and the crawl takes over from the page it reached.
Cost and limits
A crawling run is charged per page fetched, in complete blocks as they fill, with the last block rounded up once. It replaces the flat charge of an ordinary run.
| What | Rule |
|---|---|
| Plain pages | 1 credit per 100 pages |
| Pages that need a browser | 1 credit per 10 pages |
| Pages per crawl | Free 1,000 · Pro 10,000 · Team 50,000 · Business 200,000 |
| Time | 30 minutes per crawl |
Why a crawl stopped
- Found everything the plan asked for. or Crawled the whole site: no new pages left. — finished.
- Stopped at your plan’s page limit, Stopped: a crawl runs at most 30 minutes. or Stopped: your credits ran out. — The results are partial.
- Stopped: the site refused or stopped answering. — try again later, or through a proxy.
Continue and inspect
A crawl that stopped early can pick up where it left off: press Continue in the crawl panel. The new run starts from the pages still waiting and skips the ones already read.
The network record
Every request a crawl made is kept with its status, size, time and headers, and the body of each response. Search by address, filter by status (2xx, 3xx…), and open a request’s General, Request and Response tabs. A body opens on its first 4 KB; Show more and Load all read further.