Consuming Paginated API's in Javascript

Several providers adopt a notion of pagination in their API’s when the number of results from a query is large. This is common to see when the API provides a list of things, like users, billing statements, posts and so on.

A pattern that is adopted by several providers works as follows:

1. Developer makes a request to provider (e.g. for the billing statements)
2. The API returns a “page” with say 10 statements, and a link to the next page
3. Developer makes a request to the provided link
4. The API returns another “page” with statements, and no link
5. Seeing no link, the developer now knows that all the required entries have been fetched

The number of pages can be dynamic, so the developer needs to keep following the links until they exist if they want to fetch all the results. The code to do this manually can get quite convoluted, with many custom solutions requiring some sort of queue. An easier solution is to use the excellent async library, which works both in the browser and in NodeJS.

Let us first create a sample API provider that returns paginated results:

var sample_paginated_api = function(query)
{
  return new Promise(function(resolve, reject) {
    var page_number = parseInt(query);
    if (page_number == 5) {
      // Final page
      return resolve({
        "results": [0, 1].map((x) => (x*page_number)),
      });
    } else {
      // Intermediate pages
      return resolve({
        "results": [0, 1, 2, 3, 4].map((x) => (x*page_number)),
        "nextLink": (page_number+1).toString()
      });
    }
  });
}

This little snippet emulates a request to a API provider that returns a Promise, which resolves to the response. The response may contain a nextLink key if there are more pages. For example, Using the API with query as 0, we get the following output:

sample_paginated_api('0').then((res) => console.log(res))
// Output: { results: [ 0, 0, 0, 0, 0 ], nextLink: '1' }

Now let us use the aformentioned async library to implement our consumer that can follow the next links until all the pages are exhausted. Specifically, we will use the async.whilst control flow helper.

// Tested in node v14
const async = require('async');

all_results = []
sample_paginated_api('0').then((results)  => {
  latest_results = results;
  all_results.push(latest_results);
  async.whilst(
    (cb) => cb(null, latest_results.nextLink),
    (iteration_cb) => {
      sample_paginated_api(latest_results.nextLink).then((res) => {
        latest_results = res;
        all_results.push(latest_results);
        iteration_cb(null, res)
      });
    },
    (err, results) => {
      console.log(all_results)
    }
  )
});

Let’s try to understand whats going on here. The async.whilst method accepts three arguments:

1. The test callback: This function should return a boolean indicating if the while loop should continue. In our case, we simply check for the presence or absense of latest_results.nextLink
2. The iteration callback: This function is called for every iteration where the test callback returned true. In our case, since there is a nextLink (since the test passed), we call the API again with the nextLink and store the result in latest_results for the next iteration’s test
3. The final callback: This function is called at the end of all iterations, but only contains the results from the last iteration. Hence, we store the results from all iterations in a global variable all_results

The output of the above snippet is:

[
  { results: [ 0, 0, 0, 0, 0 ], nextLink: '1' },
  { results: [ 0, 1, 2, 3, 4 ], nextLink: '2' },
  { results: [ 0, 2, 4, 6, 8 ], nextLink: '3' },
  { results: [ 0, 3, 6, 9, 12 ], nextLink: '4' },
  { results: [ 0, 4, 8, 12, 16 ], nextLink: '5' },
  { results: [ 0, 5 ] }
]

And thats all! all_results will contain the results from each iteration, i.e. each page for you to process as needed.