API Pagination Guide: Compare Offset, Cursor, and Keyset Pagination

Overview

Pagination divides a large result set into smaller responses. A client requests one portion at a time instead of downloading every matching record in a single response. Most paginated APIs expose a page size such as limit or pageSize, along with a value that identifies the requested position.

The best pagination method depends on how frequently data changes, how large the collection can become, whether users need to jump to a specific page, and whether the endpoint supports a stable sort order. For a small, mostly static collection, offset or page-number pagination may be sufficient. For feeds, exports, activity logs, and frequently changing datasets, cursor or keyset pagination is usually easier to make consistent.

Before implementing an endpoint, document the contract: the parameter names, default and maximum page sizes, sort behavior, filter behavior, response metadata, and the meaning of an empty page. An OpenAPI specification can make these details visible to client developers and easier to validate during testing.

Checklist by scenario

Use page-number or offset pagination when navigation matters

Page numbers are a reasonable choice for administrative screens, search results, and collections where users need to move directly between pages. Offset pagination is also straightforward for internal tools and small datasets.

• Define whether page starts at zero or one.
• Define a maximum per_page or limit value and enforce it server-side.
• Apply a deterministic sort, preferably including a unique tie-breaker.
• Return enough metadata for the client to know whether another page exists.
• Test behavior when records are added or deleted between requests.

A request might look like GET /orders?page=3&per_page=25&sort=created_at:desc. A response can include an array of items and a navigation object such as currentPage, pageSize, hasNext, and optional links. Avoid requiring clients to calculate the total number of pages unless an accurate count is affordable and meaningful.

Use cursor pagination for changing feeds and forward-only clients

Cursor pagination gives the client a position token rather than a numeric location. The server creates the cursor from the relevant ordering and may encode additional state, such as filters or a snapshot boundary. Clients should treat the cursor as opaque: store it and send it back without parsing or modifying it.

• Return a cursor for the next request only when more results are available.
• Document whether an absent, null, or empty cursor means the collection is complete.
• Keep the filter and sort parameters unchanged while following a cursor.
• Decide how expired, malformed, or invalid cursors are reported.
• Make the client stop when the next cursor is missing or unchanged.

Typical exchanges look like this:

GET /events?limit=100

{
  "items": [
    { "id": "evt_1001", "type": "login" }
  ],
  "nextCursor": "opaque-token"
}

The next request is GET /events?limit=100&cursor=opaque-token. A robust client records the cursor after successfully processing the page, not before. That distinction helps prevent data loss when a request succeeds but downstream processing fails.

Use keyset pagination for large ordered datasets

Keyset pagination, sometimes called seek pagination, uses the last record's sort key as the boundary for the next query. For example, a dataset ordered by ascending id can use WHERE id > :last_id ORDER BY id ASC LIMIT :limit. The database can often use an index to seek directly to the boundary instead of scanning and skipping many earlier rows.

The ordering must be stable and unique. If timestamps can repeat, use a compound boundary such as (created_at, id). The next query then selects records after that pair, using the same comparison rules as the sort order. For descending order, reverse both the comparison and the ordering consistently.

Keyset pagination is a strong fit for exports, synchronization jobs, audit logs, and background workers. It is less suitable when a client must jump to an arbitrary page or display an exact page count.

Paginate third-party APIs with defensive client logic

When integrating an external API, follow its documented pagination contract rather than assuming that every service uses page and limit. Some APIs return a URL for the next request; others return a cursor under a nested metadata property. Preserve query parameters, authentication headers, and required filters when following a next link. If authentication or cross-origin behavior is involved, the CORS and API troubleshooting guide can help separate browser restrictions from pagination errors.

What to double-check

Sorting and filtering

Pagination is only reproducible when the result order is defined. Always specify an explicit sort, and add a unique secondary key when the primary field is not unique. Keep filters unchanged throughout a traversal. If a client changes the filter while reusing a cursor, the server should reject the request or clearly define the result; silently mixing states can produce confusing gaps.

Response semantics

Choose a consistent shape for items and navigation data. Useful fields include items, nextCursor, hasNext, and links for the next or previous request. Do not make clients infer completion solely from a short page unless that behavior is explicitly documented. A short page can result from filtering, permissions, or a changing dataset.

Retries, rate limits, and processing

Retry transient failures with bounded exponential backoff and jitter where appropriate. Do not blindly retry every client or validation error. Respect the API's rate-limit signals and avoid launching one request per page concurrently unless the API contract and ordering requirements allow it.

For data exports, make processing idempotent. Record a durable checkpoint such as the last successful cursor, keyset boundary, or source identifier. If a page is processed twice after a retry, a unique key or upsert strategy should prevent duplicate writes. For sensitive tokens, avoid logging full cursors if they may contain encoded filters or internal state.

Contract and test coverage

Test the first page, a middle page, the final page, an empty collection, a page-size boundary, invalid pagination parameters, and records inserted or deleted during traversal. Also test duplicate sort values, expired cursors, repeated next cursors, server errors, and rate-limit responses. Use a JSON formatter or validator during debugging when response nesting makes it difficult to inspect whether the navigation field is present and correctly typed.

Common mistakes

• Using an unstable sort: Sorting only by a non-unique timestamp can cause records to move between pages. Add a unique tie-breaker.
• Changing filters mid-traversal: A cursor or offset belongs to a particular result definition. Start a new traversal when filters change.
• Assuming every short page is the last page: Confirm completion using documented metadata or the absence of a next link or cursor.
• Incrementing an offset after a failed request: Advance the checkpoint only after the page has been received and safely processed.
• Retrying with a new cursor: When a request fails transiently, retry the same request and cursor unless the API explicitly says otherwise.
• Ignoring repeated cursors: Track the previous cursor. If the API returns the same next cursor repeatedly, stop and report the condition instead of creating an endless loop.
• Fetching an unsafe page size: Large limits can increase latency, memory use, and timeout risk. Use a documented maximum and tune it through measurement.
• Counting everything unnecessarily: An exact total count may be expensive on a large or changing dataset. Return it only when the product genuinely needs it.

For scraping or synchronization work, pagination should be considered alongside selectors, delays, retries, and deduplication. The web scraping workflow guide provides a broader checklist for reliable data extraction.

When to revisit

Review pagination before launching a new client, adding a large data export, changing the endpoint's sort order, or migrating the underlying database. Revisit it when users report missing or repeated records, when response times increase at higher offsets, or when a third-party provider changes its SDK or response format.

Use this practical review checklist:

1. Write down the endpoint's stable sort, filters, page-size limits, and completion signal.
2. Choose page-number, offset, cursor, or keyset pagination based on the scenario rather than implementation habit.
3. Run tests with inserts, deletes, duplicate sort values, empty results, failures, and repeated cursors.
4. Verify that retries do not skip pages and that downstream writes are idempotent.
5. Inspect logs and metrics for latency, page size, retry counts, rate-limit responses, and traversal duration.
6. Update the API contract and client examples whenever pagination behavior changes.

A well-designed pagination contract makes ordinary reads simple while giving exports and background jobs a safe way to resume. Treat the ordering, checkpoint, and completion rule as one system: if any one of them is ambiguous, clients can appear to work while quietly missing or duplicating data.