Timi Ajiboye

How to implement URL Pagination for Sails.js API

Complete code for this tutorial can be found here

Sails.js has a lot of nifty in-built features and tools. There's one however, that I wish it had. The ability to get a paginated response by doing just this:

GET localhost:1337/post?page=3&perPage=20  

find()

To accomplish this we're going to override the find() Blueprint default action. (For more information about Blueprints, check my previous post on blueprints)

The find() action is responsible for the out-of-the box URL querying functionality we get when we create a model-controller pair.

To see it in action, let's get started:

sails new pagination-example --no-front-end  
sails new api Post  

The sails new api <Model> command automatically creates Post.js and PostController.js files for one in the proper folders.

Let us give our Post.js model one attribute that we can play with.

module.exports = {  
  attributes:{
        body: {
          type: "string",
          required: true
        }
  }
}

Because Blueprints are automatically turned on, you can sails lift the application, then test out the find() action in your browser by pasting this URL localhost:1337/post?where={'body':'hahaha'}&limit=3.
This will automatically get you 3 posts that have their body equal to 'hahaha'.

Note: You'll need to create dummy Post entries before you can see any of this work.

All this querying and filtering functionality happens right out of the box, without writing any extra code.

What we want to do, is to extend that functionality. To do that we, need to create a /api/blueprints folder then create a find.js file in it.

We're going to copy the sails default find blueprint code from the repository on github, paste it in our own find.js, then make some additions to it.

actionUtils

If you look at the very top of the find.js code, you'll see:

var actionUtil = require('../actionUtil')  

You can also easily find that actionUtil's methods are used pretty much everywhere in the file. In our find() override, we want to make use of these same methods and we'd also like to add our own.

To do this, we can create an ActionUtilityService.js in api/services. (For more information about Services, check my previous post on services)

In api/services/ActionUtilityService.js

const actionUtil = require('../../node_modules/sails/lib/hooks/blueprints/actionUtil')

module.exports = {  
  util: actionUtil
}

With this Service, actionUtil in it's entirety is exposed throughout our sails app via ActionUtilityService.util. Also, it's a pretty simple matter to add functions to actionUtil

Let the overriding begin

First of all, look at line(s) 40 - 44 in the find.js. You'll find the Waterline query that's responsible for querying the database and returning the results that match your request's filters.

...
  var query = Model.find()
  .where(actionUtil.parseCriteria(req))
  .limit(actionUtil.parseLimit(req))
  .skip(actionUtil.parseSkip(req))
  .sort(actionUtil.parseSort(req));
...

It follows, that it is at this point we would want to add something like .paginate().

paginate() is an in-built Waterline helper method that can accomplish a sort of combination of skip() and limit(). It's used like this:

Model.find()  
...
.paginate({page:1, limit: 10})
...

You can read the more about the query language in the sails official documentation.

We need to replace all usages of actionUtil in the find.js code with ActionUtilityService.util and then add paginate() to the query in find.js

    var query = Model.find()
    .where(ActionUtilityService.util.parseCriteria(req))
    .limit(ActionUtilityService.util.parseLimit(req))
    .skip(ActionUtilityService.util.parseSkip(req))
    .sort(ActionUtilityService.util.parseSort(req))
    .paginate({page: ActionUtilityService.util.parsePage(req), limit: ActionUtilityService.util.parsePerPage(req)})

Now we need create our perPage(req) and parsePerPage(req) functions in our Service.

In /api/services/ActionUtilityService

const actionUtil = require('../../node_modules/sails/lib/hooks/blueprints/actionUtil')

actionUtil.parsePage = (req) => {  
  const DEFAULT_PAGE = 1;
  let page = req.param('page') || (typeof req.options.page !== 'undefined' ? req.options.page : DEFAULT_PAGE);
  if (page) {
    page = +page;
  }
  return page;
}

actionUtil.parsePerPage = (req) => {  
  const DEFAULT_PER_PAGE = req._sails.config.blueprints.perPage || 20;
  let perPage = req.param('perPage') || (typeof req.options.perPage !== 'undefined' ? req.options.perPage : DEFAULT_PER_PAGE);
  if (perPage) {
    perPage = +perPage;
  }
  return perPage;
}

module.exports = {  
  util: actionUtil
}

The parsePage() and parsePerPage() functions are pretty much copies of the original actionUtil's parseSkip() and parseLimit().

With the above code in place, you can even set a default perPage value in sails.config.blueprints.

Now it seems like we're all done, and that GET localhost:1337/post?page=3&perPage=20 should work.

Unfortunately, it won't.

parseCriteria(req)

Remember when I mentioned this URL localhost:1337/post?where={'body':'hahaha'}&limit=3? The function in actionUtils that's responsible for making the where filter work is parseCriteria(req).

parseCriteria() has functionality to take all parameters in the URL and construct a query object to be passed into Waterline's .where() query function. It even works when you don't do ?where=.

In the code of parseCriteria(), there are arrays of query parameter keys that should be ignored like limit, skip, etc. We need to add page and perPage to those lists.

To do that, we need to override/write our own parseCriteria().

Below is the complete code for ActionUtilityService

const actionUtil = require('../../node_modules/sails/lib/hooks/blueprints/actionUtil')  
const isArray = require('lodash.isarray');  
const isUndefined = require('lodash.isundefined');  
const isString = require('lodash.isstring');  
const isObject = require('lodash.isobject');  
const _ = require('lodash')

// Parameter used for jsonp callback is constant, as far as
// blueprints are concerned (for now.)
const JSONP_CALLBACK_PARAM = 'callback';

actionUtil.parsePage = (req) => {  
  const DEFAULT_PAGE = 1;
  let page = req.param('page') || (typeof req.options.page !== 'undefined' ? req.options.page : DEFAULT_PAGE);
  if (page) {
    page = +page;
  }
  return page;
}

actionUtil.parsePerPage = (req) => {  
  const DEFAULT_PER_PAGE = req._sails.config.blueprints.perPage || 20;
  let perPage = req.param('perPage') || (typeof req.options.perPage !== 'undefined' ? req.options.perPage : DEFAULT_PER_PAGE);
  if (perPage) {
    perPage = +perPage;
  }
  return perPage;
}

actionUtil.parseCriteria = (req) => {  
  // Allow customizable blacklist for params NOT to include as criteria.
  req.options.criteria = req.options.criteria || {};
  req.options.criteria.blacklist = req.options.criteria.blacklist || ['limit', 'skip', 'sort', 'populate', 'page', 'perPage'];

  // Validate blacklist to provide a more helpful error msg.
  var blacklist = req.options.criteria && req.options.criteria.blacklist;
  if (blacklist && !isArray(blacklist)) {
    throw new Error('Invalid `req.options.criteria.blacklist`. Should be an array of strings (parameter names.)');
  }

  // Look for explicitly specified `where` parameter.
  var where = req.params.all().where;

  // If `where` parameter is a string, try to interpret it as JSON
  if (isString(where)) {
    where = tryToParseJSON(where);
  }

  // If `where` has not been specified, but other unbound parameter variables
  // **ARE** specified, build the `where` option using them.
  if (!where) {

    // Prune params which aren't fit to be used as `where` criteria
    // to build a proper where query
    where = req.params.all();

    // Omit built-in runtime config (like query modifiers)
    where = _.omit(where, blacklist || ['limit', 'skip', 'sort', 'page', 'perPage']);

    // Omit any params w/ undefined values
    where = _.omit(where, function (p) {
      if (isUndefined(p)) {
        return true;
      }
    });

    // Omit jsonp callback param (but only if jsonp is enabled)
    var jsonpOpts = req.options.jsonp && !req.isSocket;
    jsonpOpts = isObject(jsonpOpts) ? jsonpOpts : {callback: JSONP_CALLBACK_PARAM};
    if (jsonpOpts) {
      where = _.omit(where, [jsonpOpts.callback]);
    }
  }

  // Merge w/ req.options.where and return
  where = _.merge({}, req.options.where || {}, where) || undefined;

  return where;
}

function tryToParseJSON (json) {  
  if (!isString(json)) return null;
  try {
    return JSON.parse(json);
  }
  catch (e) { return e; }
}

module.exports = {  
  util: actionUtil
}

A few things to note:
1. We included all the modules that parseCriteria uses. You'll need to npm i --save <module> for the following modules: lodash.isstring, lodash.isarray, lodash.isundefined, lodash.isobject and lodash itself.
2. The above code is a copy of the original parseCriteria's code with tiny changes to the two blacklist arrays. We simply add page and perPage to them.
3. Remember to copy the tryToParseJSON(json) function too.

Back to find()

Let's clean up find.js by using Promises (Waterline query supports them by default and this is what we'll have.

/**
 * Module dependencies
 */
_ = require('lodash');

/**
 * Find Records
 *
 *  get   /:modelIdentity
 *   *    /:modelIdentity/find
 *
 * An API call to find and return model instances from the data adapter
 * using the specified criteria.  If an id was specified, just the instance
 * with that unique id will be returned.
 *
 * Optional:
 * @param {Object} where       - the find criteria (passed directly to the ORM)
 * @param {Integer} limit      - the maximum number of records to send back (useful for pagination)
 * @param {Integer} skip       - the number of records to skip (useful for pagination)
 * @param {String} sort        - the order of returned records, e.g. `name ASC` or `age DESC`
 * @param {String} callback - default jsonp callback param (i.e. the name of the js function returned)
 */

module.exports = function findRecords(req, res) {

  // Look up the model
  var Model = ActionUtilityService.util.parseModel(req);


  // If an `id` param was specified, use the findOne blueprint action
  // to grab the particular instance with its primary key === the value
  // of the `id` param.   (mainly here for compatibility for 0.9, where
  // there was no separate `findOne` action)
  if (ActionUtilityService.util.parsePk(req)) {
    return require('./findOne')(req, res);
  }

  // Lookup for records that match the specified criteria
  var query = Model.find()
    .where(ActionUtilityService.util.parseCriteria(req))
    .limit(ActionUtilityService.util.parseLimit(req))
    .skip(ActionUtilityService.util.parseSkip(req))
    .sort(ActionUtilityService.util.parseSort(req))
    .paginate({page: ActionUtilityService.util.parsePage(req), limit: ActionUtilityService.util.parsePerPage(req)})
  query = ActionUtilityService.util.populateRequest(query, req);
  query.then((matchingRecords) => {
    if (req._sails.hooks.pubsub && req.isSocket) {
      Model.subscribe(req, matchingRecords);
      if (req.options.autoWatch) {
        Model.watch(req);
      }
      // Also subscribe to instances of all associated models
      _.each(matchingRecords, function (record) {
        ActionUtilityService.util.subscribeDeep(req, record);
      });
    }  
      res.ok(matchingRecords);
  }).catch((err) => {
    if (err) return res.serverError(err);
  })
};

One more thing

What good is a pagination API if the client cannot tell how many records there are?
How else will the client calculate how many pages exist in the database?

To do this, we can add a little code to our find.js to add the Total count of the model to the headers of the response.

...
query.then((matchingRecords) => {  
    ...
    Model.count().then((found) => {
      res.set('Total', found)
      res.ok(matchingRecords);
    })
    ...
}

That's all. You can fire up Postman (or your browser) to test.

Remember, the complete code for this tutorial can be found here

O dabọ.