AI ScraperGuide

Call a scraper from a flow or a script

A published scraper is a callable, like a script. Drop it into a flow as a step and bind its inputs, or call it from code with one line — scraper.run("shop_products", {"category": "2"}) — and get its rows back. Both always run the published version, and the run appears under its caller on the Runs page.

8 min readBefore this: The program and its versions, What a flow isOpen in Waiso
What you will learn
  • Add a scraper step to a flow and bind its inputs
  • Call a scraper from code in every script language
  • Read the result: rows, count, doubts and notFound
  • Handle a scraper’s errors and timeouts

A scraper as a step

In a flow, + → AI Scraper lists only published scrapers. The step’s sidebar shows the scraper’s declared inputs as fields you bind like any other step’s parameters, a Step timeout, Open scraper, and Run test. Its output is the rows, ready for a loop or a SQL step after it.

A scraper step runs the published program — no AI at run time — and its rows feed the next step.

What the step returns

The step’s output is { rows, count, runId, downloadPath }: bind output.rows to a loop, or output.count to a condition. When the result is larger than 5 MB it is stored outside the run, rows is empty and downloadPath says where to get it. A Step timeout bounds the step alone: past it, the scraper run is cancelled and the step fails with timeout, which an On error step can catch.

In the flow file, the step references the scraper by id; its name is only a readable hint.

YAMLcampaign.flow.yaml
collect:
  kind: aiScraper
  label: Collect products
  scraper: 0f1c2d3e-4a5b-6c7d-8e9f-0a1b2c3d4e5f
  scraperName: shop_products
  timeoutMs: 120000
  inputs:
    category: { from: start, path: output.category }
  next: each_product

Call it from code

The path is the scraper’s folder path with exact names ("dir/sub/name") or its id. Inputs are checked against what the published version declares — text, number or date.

collect.py
from waiso import scraper

def main(category: str = "2"):
    res = scraper.run("shop_products", {"category": category}, timeout=120)
    if res.not_found:
        return []
    print(f"{res.count} rows, doubts: {res.doubts}")
    return res.rows

Start now, collect later

  • run waits for the rows. It never outlives the calling script: past its timeout (or the script’s own remaining time) the scraper run is cancelled and a timeout error is raised.
  • run_async / runAsync returns a run id at once; status reads its state without blocking; wait collects the rows, obeying only its own timeout and never cancelling the run.

The result and errors

  • rows — every row, even when a large result was stored outside the run (then downloadPath is set too); count — how many.
  • doubts — fields or steps the build was unsure of, [] when clean. Check it before trusting the rows.
  • notFound — the input matched nothing; rows is empty and this is an answer, not a failure.

Errors carry a stable code: scraper_not_found, scraper_not_published, scraper_failed, credit_exhausted and timeout, plus the run id when a run exists. In a flow, publish and run refuse a step pointing at a scraper that is gone or unpublished.

Related topics