eddy@homestorage539/787 TiB· 1.17 PB rawUsed/usable is space across every pool, after parity and ZFS overhead. Raw is every disk's label size added up. Per-pool detail is on the homelab page.
↑↓ move↵ openesc close#tag · area:homelab · in:journal

ScraperX

A Python web scraping library that runs the same scraper on a laptop or scaled out across AWS.

ScraperX is a Python web scraping library that covers everything from a tiny scraper that’s quick to get running, up to a complex, hard-to-scrape site that scales as far as your infra budget allows.

Why I built it

I needed a way to quickly test scraping a site at a small scale and try out different methods of scraping it. I wanted to be able to run a scraper fully on my laptop from start to finish, but then also be able to scale any single part of it out to places like AWS, using AWS’s services to scale as the task requires.

None of this started from scratch. It’s a rebuild of web scraping code I had been using for years before, just packaged up into something that’s easy to use.

How it works

A scraper is split into 3 parts:

  • Dispatcher: builds the list of tasks to run and sends them out at whatever rate limit you set.
  • Downloader: fetches the page and saves the raw source, handling retries, proxies and user-agents for you.
  • Extractor: parses the saved source into structured data and saves data based on templates.

The thing that ties them together is a “task”, which is just a JSON object. By default all 3 steps run in order in the same process, which is great locally when developing and testing a scraper. In production that exact same code runs at scale: rather than each step running in one process on a single machine, the task gets sent to a message queue and any type of worker can pick it up and run the next part. Sources and extracted data can be saved locally or to S3.

A minimal scraper

This scrapes the first few pages of quotes.toscrape.com, a site made for practicing web scraping. It’s based on the minimal example in the repo. Each class handles one step, and anything you don’t override uses the built-in defaults.

# quotes.py
from scraperx import Scraper, run_cli, Dispatch, Download, Extract


class QuotesDispatch(Dispatch):
    def submit_tasks(self):
        # A task is just a dict, it gets passed along to every step
        return [
            {'url': f'https://quotes.toscrape.com/page/{page}/', 'page': page}
            for page in range(1, 4)
        ]


class QuotesDownload(Download):
    def download(self):
        # A requests response, with retries, user-agent and proxy handled for you
        r = self.request_get(self.task['url'])
        # Saved using the downloader's `file_template` from the config
        self.save_request(r)


class QuotesExtract(Extract):
    def extract(self, raw_source, source_idx):
        yield self.extract_task(
            name='quotes',
            selectors=['div.quote'],  # each match is passed to the callback
            callback=self.extract_quote,
            post_extract=self.save_as,
            post_extract_kwargs={'file_format': 'json'},
        )

    def extract_quote(self, element, idx, **kwargs):
        return {
            'text': element.css('span.text::text').get(),
            'author': element.css('small.author::text').get(),
            'tags': element.css('a.tag::text').getall(),
        }


quotes_scraper = Scraper(
    dispatch_cls=QuotesDispatch,
    download_cls=QuotesDownload,
    extract_cls=QuotesExtract,
)

if __name__ == '__main__':
    run_cli(quotes_scraper)

Plus a small config next to it that sets the rate limit and where files get saved. Any key from the task can be used in the file names:

# config.yaml
default:
  dispatch:
    ratelimit:
      type: qps
      value: 1
  downloader:
    file_template: "source/{scraper_name}/page_{page}.html"
  extractor:
    file_template: "extracted/{scraper_name}/page_{page}.json"

Then pip install scraperx and python quotes.py dispatch runs the whole thing start to finish, leaving the raw HTML in source/ and a JSON file of quotes per page in extracted/.

Rerunning just the part you need

Since every step can run on its own, you don’t have to run the whole thing every time. Say you’re building out the extraction for a page you already downloaded, and you don’t want to get blocked by the site from hitting it over and over while testing locally? Just run only the extractor against the source file the downloader saved in a previous run:

python quotes.py extract source/quotes/page_2.html

This also makes fixing an extractor bug very simple. Once you update the extractor, you can re-run it over every source file already downloaded, no need to scrape the site again. And to keep a fix from breaking pages that already worked, you can turn a downloaded source into a test case with a hand-checked copy of its extracted data to compare against.

Trade-offs

Web scraping is an ever-changing game of cat and mouse, so I needed this system to be very flexible, but I also wanted it to be quick and simple to get up and running. That comes with a few trade-offs, the main one being it’s a bit “black boxy”. If you know how it works, you need very little code to get a full web scraper running, but if you want to override something and do something very custom, you need to know how it works under the hood.

That’s also the beauty of this library though. You can fully swap out how each component works, or the code running in it, and still reuse the way the scraper runs and passes tasks around.

Where it stands

ScraperX is pretty much done at this point. I don’t plan on making any major changes to it, as I have other scraping projects in the works that will replace it at some point.

↑ top