How to Handle Pagination, Infinite Scroll, and Load More Buttons in Scrapers

Overview

If you scrape enough modern websites, you eventually see the same content-loading patterns repeated with different visual styling. A listing page may expose page numbers in the URL, append more results when you scroll, or hide the next batch behind a button. The frontend looks different each time, but the scraper decision tree is usually the same.

The practical goal is not to “simulate browsing” as closely as possible. The goal is to collect complete, structured data with the fewest moving parts. In many cases, the best scraper does not scroll at all. It finds the underlying request, reproduces it directly, and paginates through the data source instead of the rendered UI. When that is not possible, browser automation becomes the fallback.

A good pagination scraping guide starts with one question: where does the next batch of records really come from? The answer is typically one of four sources:

• A predictable URL pattern such as ?page=2 or /page/2/
• An XHR or fetch request returning JSON
• A GraphQL request using cursors or offsets
• DOM updates triggered only by client-side interaction

Before writing code, inspect the target page in your browser’s network tab. Filter by XHR or fetch, trigger the next page or scroll event manually, and watch what changes. This one step often saves more time than any framework choice.

If you are deciding which browser runtime to use for dynamic pages, pair this article with Best Headless Browsers for Scraping and Testing. If you are still choosing a broader stack, Scrapy vs Beautiful Soup vs Requests helps frame where browser automation fits.

Core framework

Use the framework below whenever you need to handle dynamic pagination in a scraper. It is designed to keep your implementation simple first and interactive only when necessary.

1. Classify the loading pattern

Start by placing the page into one of these categories:

• Static pagination: page links, page parameters, or canonical next URLs are visible in the HTML.
• Load more button: a button requests and appends the next batch without full navigation.
• Infinite scroll: scrolling triggers new requests and appends additional items automatically.
• Hybrid pattern: page numbers exist, but content is still injected dynamically; or a load more button appears after initial scroll.

Do not assume the visible UI tells the full story. A page with infinite scroll may still use a clean paginated API under the hood.

2. Prefer direct requests over browser events

The most maintainable strategy is usually:

1. Find the request responsible for the next batch.
2. Replicate its headers, params, and payload as needed.
3. Iterate offsets, pages, or cursors directly.
4. Parse the JSON or returned HTML fragments.

This approach is usually faster, cheaper, and less brittle than trying to scrape infinite scroll through repeated viewport interactions. It also produces cleaner logs because request boundaries are explicit.

Use browser automation only when one of these conditions applies:

• Requests are hard to reproduce because tokens are generated in-page.
• The site depends heavily on client-side state.
• Data is embedded only after scripts run.
• The interaction sequence itself is required to unlock additional content.

3. Define a stop condition before you loop

Most scraper bugs around pagination come from weak termination logic. “Keep scrolling until nothing changes” sounds simple, but in practice it can trap your job in long waits or duplicate extraction.

Good stop conditions include:

• No next-page link exists
• Returned JSON contains an empty result set
• The cursor is null or absent
• The load more button becomes disabled or disappears
• The item count no longer increases after a bounded number of retries
• A known maximum page count is reached

Always track the unique identifiers of extracted items. If the same IDs keep repeating, your loop is not advancing, even if the page visually changes.

4. Separate navigation from extraction

Keep your scraper organized around two distinct tasks:

• Navigator: moves to the next batch of content
• Extractor: parses the current batch into structured records

This separation makes it easier to swap strategies later. For example, you can begin with Playwright infinite scroll logic, then later replace only the navigator with direct API requests while keeping the same extractor.

5. Build around stable selectors and stable signals

Dynamic pages are noisy. Avoid tying your scraper to cosmetic classes or brittle DOM depth if there are better anchors available. Prefer selectors based on semantic attributes, data attributes, link patterns, or predictable container structures. If you need help choosing between selector styles, XPath vs CSS Selectors for Web Scraping is a useful companion.

Also wait on meaningful signals. Instead of sleeping for three seconds after every click, wait for one of the following:

• A network response matching the next-batch endpoint
• An increase in the number of listing elements
• A specific loading spinner to disappear
• A cursor value in the DOM or page state to change

6. Normalize records as they arrive

As new batches load, normalize fields immediately. Store source URL, page number or cursor, scrape timestamp, and any unique item key available on the page. This makes deduplication and recovery easier downstream. For storage decisions after extraction, see How to Store Scraped Data.

Practical examples

This section shows reusable patterns rather than site-specific recipes. The point is to give you scaffolding you can adapt quickly.

Example 1: Static numbered pagination

This is the simplest case. You inspect the HTML and see links like ?page=2. In this situation, do not launch a browser unless the target really requires it.

import requests
from bs4 import BeautifulSoup

base_url = "https://example.com/products?page={}"
page = 1
items = []

while True:
    resp = requests.get(base_url.format(page), timeout=30)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")

    cards = soup.select(".product-card")
    if not cards:
        break

    for card in cards:
        items.append({
            "title": card.select_one(".title").get_text(strip=True),
            "url": card.select_one("a")["href"],
            "page": page
        })

    next_link = soup.select_one("a[rel='next']")
    if not next_link:
        break

    page += 1

The important detail here is the stop condition. You can stop on the absence of a next link, on an empty result set, or both.

Example 2: Load more button with Playwright

When you need to scrape a load more button, the main challenge is avoiding duplicate extraction. Extract only the newly added items or keep a set of seen IDs.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/listings", wait_until="networkidle")

    seen = set()
    results = []

    while True:
        cards = page.locator(".listing-card")
        count = cards.count()

        for i in range(count):
            card = cards.nth(i)
            url = card.locator("a").first.get_attribute("href")
            if url in seen:
                continue
            seen.add(url)
            results.append({
                "url": url,
                "title": card.locator(".title").inner_text()
            })

        button = page.locator("button:has-text('Load more')")
        if button.count() == 0 or not button.is_visible():
            break

        previous_count = count
        button.click()
        page.wait_for_function(
            "prev => document.querySelectorAll('.listing-card').length > prev",
            previous_count
        )

    browser.close()

This pattern works because it waits for a measurable increase in item count instead of relying on arbitrary delays. If the button triggers an API request you can reliably reproduce, switch to direct requests later.

Example 3: Playwright infinite scroll

To scrape infinite scroll, scroll in steps, measure growth, and stop after a limited number of no-change cycles. This is far safer than endlessly scrolling to the bottom.

from playwright.sync_api import sync_playwright

MAX_IDLE_ROUNDS = 3

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1280, "height": 2000})
    page.goto("https://example.com/feed", wait_until="networkidle")

    idle_rounds = 0
    last_count = 0

    while idle_rounds < MAX_IDLE_ROUNDS:
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        page.wait_for_timeout(1500)

        current_count = page.locator(".feed-item").count()
        if current_count == last_count:
            idle_rounds += 1
        else:
            idle_rounds = 0
            last_count = current_count

    data = []
    cards = page.locator(".feed-item")
    for i in range(cards.count()):
        card = cards.nth(i)
        data.append({
            "title": card.locator("h2").inner_text(),
            "link": card.locator("a").first.get_attribute("href")
        })

    browser.close()

For many teams, this is the first working implementation. That is fine. But once it works, inspect the network requests and see whether the scroll sequence can be replaced by a cleaner JSON endpoint. The initial browser-based version is often best treated as a discovery tool.

Example 4: Cursor-based API behind infinite scroll

A common pattern is a frontend that sends a request like:

POST /api/search
{
  "query": "laptops",
  "cursor": "eyJwYWdlIjoyfQ=="
}

The response returns items plus the next cursor. If you can reproduce that request, your scraper becomes much simpler:

import requests

url = "https://example.com/api/search"
cursor = None
results = []

while True:
    payload = {"query": "laptops", "cursor": cursor}
    resp = requests.post(url, json=payload, timeout=30)
    resp.raise_for_status()
    data = resp.json()

    items = data.get("items", [])
    if not items:
        break

    results.extend(items)
    cursor = data.get("nextCursor")
    if not cursor:
        break

When this option exists, prefer it. It is usually the cleanest answer to the question of how to handle dynamic pagination in a scraper.

Example 5: Combined list and detail scraping

Many real projects require you to collect list pages, then visit each detail page. In that case, think in phases:

1. Enumerate all list items across pagination or infinite loading.
2. Deduplicate by canonical URL or item ID.
3. Queue detail pages for a second pass.

This structure is easier to scale and monitor than trying to scrape detail data during every click or scroll. For broader pipeline design, How to Build a Web Scraping Pipeline is worth bookmarking.

Common mistakes

Most scraper instability around pagination comes from a handful of avoidable errors.

Relying on fixed sleeps

sleep(2) can appear to work during testing and fail under different network conditions. Replace fixed waits with conditions tied to DOM count, specific responses, or element state changes.

Stopping too early

Infinite scroll implementations often pause before the next batch arrives, especially on slower sites. If your scraper stops after one unchanged count, you may miss records. Use a bounded idle threshold such as two or three rounds instead of a single no-change check.

Not deduplicating

Some sites re-render existing items when new batches load. If you scrape the whole DOM after every button click without tracking seen IDs, duplicates are almost guaranteed.

Parsing the UI when the API is available

If the site already gives you structured JSON for each batch, scraping card text from the rendered DOM adds fragility for little benefit. Use the page to discover the request, then scrape the request directly where feasible.

Missing hidden limits

Some pages cap visible results even though more pages exist elsewhere in search facets, categories, or API parameters. If the total count displayed on the page is much higher than the extracted count, inspect whether the frontend is segmenting results in additional ways.

Ignoring structure drift

Pagination selectors, button labels, and request payloads change over time. Add checks that alert you when expected item counts drop or when next-page signals disappear. How to Detect Website Structure Changes Before Your Scraper Fails covers useful monitoring patterns.

Overusing browser automation

Browser tools like Playwright are excellent for discovery and for truly dynamic pages, but they are not always the best production default. A requests-based implementation is often easier to scale, proxy, and retry. If anti-bot friction becomes part of the problem, review your legal and operational posture as well, including Web Scraping Legal Checklist and CAPTCHA Bypass Strategies for Web Scraping.

When to revisit

You should revisit your pagination, infinite scroll, or load-more handling whenever the target site changes how it delivers records. In practice, that usually happens more often than developers expect. A working scraper can remain stable for months, then break after a frontend rewrite, an API parameter change, or a small shift in the stop condition.

Review your implementation when any of the following happens:

• The site moves from server-rendered pages to client-side rendering
• A numbered pagination flow becomes infinite scroll
• The load more button disappears or changes text and state behavior
• Network requests switch from page numbers to cursor-based pagination
• Item counts suddenly drop even though the page still loads
• The site introduces stronger anti-automation checks
• Your extraction now takes much longer than before

A practical maintenance checklist looks like this:

1. Open the page manually and trigger the next batch once.
2. Inspect network requests and compare them to your current scraper assumptions.
3. Confirm your stop condition still reflects the page’s real terminal state.
4. Verify deduplication keys still exist and remain stable.
5. Run a small scrape and compare item counts against visible expectations.
6. Log enough metadata—page number, cursor, request URL, batch size—to debug quickly later.

If you are starting a new target today, use this order of operations:

1. Check whether a direct paginated or cursor-based request exists.
2. If yes, build around the request.
3. If no, test whether a load more button can be handled with click-and-wait logic.
4. If not, implement bounded infinite scroll with a robust no-growth stop condition.
5. Separate enumeration from detail scraping and store stable identifiers early.

That sequence keeps your scraper focused on the least fragile mechanism available. It also makes future upgrades easier. A browser-based Playwright infinite scroll solution can get you unstuck, but the long-term win is understanding the site’s data flow well enough to replace simulated interaction with simpler, more explicit pagination logic.

As a final rule, treat every pagination strategy as provisional. The UI pattern is what users see; the request pattern is what your scraper depends on. If you keep your implementation anchored to that distinction, you will be able to handle most content-loading patterns with less guesswork and fewer rewrites.