SerpApi CSE Handbook

Case 2: The "Missing Data" Issue (Pagination)

Problem Statement

Scenario: A user wants 50 results but their loop logic is flawed, fetching only the first page repeatedly.

Common Mistake

Forgetting to implement pagination logic results in only getting the default 10 results, no matter how many times you call the API.

The Broken Code

broken.js
import * as Dotenv from "dotenv";
import { getJson } from "serpapi";

Dotenv.config();

console.log("Starting job search...");

// BROKEN LOGIC:
// This script only fetches the default first page (10 results).
// It misses the 'start' parameter and the loop structure needed for pagination.
const json = await getJson({
  engine: "google",
  api_key: process.env.SERPAPI_KEY,
  q: "Barista Job",
  location: "Bali",
});

const count = json.organic_results?.length || 0;
console.log(`Done. Found ${count} jobs.`);
console.log("Where are the rest of the jobs? (Missing pagination logic)");

Output:

Done. Found 10 jobs. Where are the rest of the jobs? [Missing pagination]

The Root Cause

Google Search returns results in pages of 10 (by default). To get more results, you need to:

  1. Loop through multiple API calls
  2. Increment the start parameter (offset) by 10 each time
  3. Handle edge cases where no more results exist

The Solution

Implement proper pagination with the start parameter and loop logic:

solution.js
import * as Dotenv from "dotenv";
import { getJson } from "serpapi";

Dotenv.config();

const maxPages = 3;
const resultsPerPage = 10;

console.log(`Starting pagination fetch for ${maxPages} pages...`);

for (let i = 0; i < maxPages; i++) {
  const currentOffset = i * resultsPerPage; // Offset: 0, 10, 20...

  try {
    console.log(`Requesting Page ${i + 1} (offset: ${currentOffset})...`);

    // Serial fetching using await inside the loop
    const json = await getJson({
      engine: "google",
      api_key: process.env.SERPAPI_KEY,
      q: "Barista Job",
      location: "Bali",
      start: currentOffset,
    });

    const count = json.organic_results?.length || 0;
    console.log(`Page ${i + 1}: Retrieved ${count} jobs.`);

    // Optimization: Stop if no results are returned to save credits
    if (count === 0) {
      console.log("No more results. Stopping loop.");
      break;
    }
  } catch (error) {
    console.error(`Failed to fetch page ${i + 1}:`, error.message);
    break;
  }
}

console.log("Pagination complete.");

Output:

Requesting Page 1 (offset: 0)... Page 1: Retrieved 10 jobs. Requesting Page 2 (offset: 10)... Page 2: Retrieved 10 jobs. Requesting Page 3 (offset: 20)... Page 3: Retrieved 10 jobs. Pagination complete. [Success]

Key Takeaways

Best Practice

Always implement the start parameter with proper offset calculation when fetching multiple pages.

  • Use start parameter with offset: i * resultsPerPage
  • Implement break condition when count === 0 to save API credits
  • Handle errors gracefully to prevent infinite loops
  • Consider storing results in an array for further processing

Optimization Tips

// Store all results
const allResults = [];

for (let i = 0; i < maxPages; i++) {
  const json = await getJson({ /* ... */ start: i * 10 });
  
  if (json.organic_results?.length > 0) {
    allResults.push(...json.organic_results);
  } else {
    break; // No more results
  }
}

console.log(`Total results collected: ${allResults.length}`);

Related Resources

ON THIS PAGE