Spiders

A Spider is a class which defines what to crawl (a site or a group of sites), how to perform the crawl (i.e. follow links) and how to extract structured data from the pages (i.e. scraping items).

Basic concepts

qCrawl spiders should subclass qcrawl.core.spider.Spider, override the async parse() method, and define the name and start_urls attributes.

Class attributes:

• name: str — unique spider identifier (required).

• start_urls: list[str] — initial URLs to crawl (required).

• custom_settings: dict — spider-specific settings that override global project settings.

• allowed_domains: list[str] — restrict crawling to these domains.

parse(response) — async generator that can yield:

• Item (container for scraped fields and internal metadata) or plain dict (engine wraps into Item).
• Request (data class representing an HTTP crawl request).
• str URL (engine or middlewares convert to Request).

The parse() method processes all downloaded pages. It receives a Page object representing the HTTP response, and yields Item objects, Request objects, or string URLs.

Simple example spider that uses css selectors and yields Item:

from datetime import datetime, timezone

from qcrawl.core.spider import Spider
from qcrawl.core.response import Page
from qcrawl.core.item import Item


class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    async def parse(self, response: Page):

        # ResponseView provides helper methods for parsing:
        # CSS/XPath selectors, link extraction, url resolution
        rv = self.response_view(response)

        # `response_view(response).doc` exposes a lazy-loaded `lxml` document tree
        # for selectors guide see: https://www.qcrawl.org/concepts/selectors/
        for q in rv.doc.cssselect(".quote"):
            text_nodes = q.cssselect("span.text")
            author_nodes = q.cssselect("small.author")
            if not text_nodes or not author_nodes:
                continue

            text = text_nodes[0].text_content().strip()
            author = author_nodes[0].text_content().strip()

            ts = datetime.now(timezone.utc).isoformat()

            yield Item(
                data={"text": text, "author": author},
                metadata={"scraped_at": ts})

        next_link = rv.doc.cssselect("li.next a")
        if next_link:
            href = next_link[0].get("href")
            if href:
                yield rv.follow(href)

Scraping lifecycle

flowchart LR
  Spider -->|"yield Request / URL / Item"| Scheduler
  Scheduler -->|"next Request"| Engine
  Engine -->|"fetch"| Downloader
  Downloader -->|"Response"| Engine
  Engine -->|"call parse(response)"| Spider

  Spider -.->|"yield Item"| Export@{ shape: bow-rect, label: "Export process" }

The simplified scraping cycle works as follows:

1. You generate the initial requests to crawl the first URLs. These requests come from the start_requests() method of your spider, which by default yields a Request for each URL in the start_urls: list[str].
2. Each request is placed in the scheduler's queue. The engine pulls the next request, sends it to the downloader, and waits for the response. Once downloaded, the response is passed back to the engine.
3. The engine calls your spider's parse(response) method with the downloaded response. Inside this method, you parse the page content (using CSS selectors, XPath) and yield either Item objects containing extracted data or new Request objects for additional URLs to crawl.
4. Any yielded Request / URL / Item object are returned to the scheduler, enqueued, and processed in the same way — forming a continuous loop until no more requests remain.
5. Any yielded Item objects are sent to the export process: item pipelines (drop, transform), exporters (data formating), and storage backends (save data).

Per-request callbacks

By default every response is handled by parse(). To route a specific request's response to a different spider method, pass callback (a method or its name) and optional cb_kwargs when creating the request:

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]

    async def parse(self, response):
        rv = self.response_view(response)
        for link in rv.doc.cssselect("a.detail"):
            yield rv.follow(link.get("href"), callback=self.parse_detail, cb_kwargs={"source": "list"})

    async def parse_detail(self, response, source):
        yield {"url": response.url, "source": source}

The callback is stored by name so requests stay serializable across the disk/Redis queues; the engine resolves it on the spider at dispatch (defaulting to parse). A lambda or other unnamed callable is rejected.