Example: scrape and store

Three SDK calls

scraper.run returns the rows of the published version. s3.upload takes text — here the JSON of the rows — and writes it to the project’s S3 connection, returning its path. state keeps small values from one run to the next: this script stores how many rows the last run saw.

The code

import json
from datetime import date
from waiso import scraper, s3, state

def main(category: str = "2"):
    res = scraper.run("shop_products", {"category": category}, timeout=300)
    if res.not_found:
        return {"stored": None, "rows": 0, "change": 0}
    obj = s3.upload(json.dumps(res.rows), name=f"{date.today()}.json",
                    folder=f"products/{category}")
    before = state.get(f"count:{category}", 0)
    state.set(f"count:{category}", res.count)
    return {"stored": obj.s3.path, "rows": res.count,
            "change": res.count - before, "doubts": res.doubts}

The file layout does the bookkeeping. Each category gets its own folder and each file carries the date in its name; every upload also gets a short unique prefix, so two runs on the same day leave two files and nothing is ever overwritten. The returned path is the exact key, which is why the script hands it back. The state key carries the category too, so running the script for several categories never mixes their counts.

1. Publish the scraper first: the script always runs its published version.
2. Run the script twice. The second time, change is 0 unless the site changed.

Make it a routine

Publish the script and add a daily schedule in its Triggers tab. Pass doubts on: when it is not empty, an alert or a flow condition can ask a person to look before the data is trusted.