Scheduling Tasks in the MEAN stack

As I was still learning my way around the MEAN stack, I couldn’t wrap my mind around how to schedule future tasks in the MEAN stack (such as sending an email). This was because MEAN is so stateless, transactional. Or at least it seems to be. But, recall that we have ExpressJS and Node.js in the picture, which are always running on the server side.

Given this, the solution turns out to be very simple. You can schedule tasks in Node using the node-schedule package. To install node-schedule, run the command:

npm install node-schedule

Then in ExpressJS you can set up tasks like so:

var schedule = require("node-schedule");
var dailyRule = new schedule.RecurrenceRule();
dailyRule.second = 0;
dailyRule.minute = 0;
dailyRule.hour = 0;
var onceADay = schedule.scheduleJob(dailyRule, function(){
  //Do something here
});

I’ve found that it’s easy to put this in ExpressJS’s app.js (or wherever your server startup script is), since that is called on the MEAN application’s server side startup.

MEAN application, getting all sockets during a request

In my MEAN application I had a need to get the list of all open websockets during a particular request. Turns out with socket.io this is easily doable if can save a reference to the socketio variable during initialization.

For a typical socket.io init in Express, the app.js would contain the following code:

var app = express();
var socketio = require('socket.io')(server, {
    path: '/socket.io-client'
});
require('./config/socketio')(socketio);

So what you need to do is save a reference to the “socketio” variable, which can be retrieved from the request passed down in Express, and used to get all sockets. To do so, you can set it as an application-wide variable in Express like so in app.js:

app.set('socketio', socketio);

Next, a typical Express function handling a request looks like so:

function(req, res) {
}

From your request (the “req” variable) you can get a reference to the Express app, which allows you to get a reference to the socketio variable you set application wide, which you can use to get the list of sockets. Like so:

function(req, res)  {
    var sockets = req.app.get('socketio').sockets.sockets;
    //...
}

And now you can loop through all the sockets like below, and do whatever with them you’d like:

for (var socketId in sockets) {
    var socket = sockets[socketId];
}

As easy as that!

Mongoose model hasOwnProperty() workaround

When using a Mongoose model, if you have a reference to a queried document, you’ll notice that you cannot call hasOwnProperty() to check if it has a certain property. For example, if you have a User model, and the schema has a property called “verified”, if you call user.hasOwnProperty(‘verified’), it will not return true!

This is because the object you get back from your mongoose query doesn’t access properties directly.

An easy and quick workaround is like so:

var userObject = user.toObject();
if(userObject.hasOwnProperty('verified'))
  //...

And now hasOwnProperty() will return true.

Mongoose save() to update value in array

This had me scratching my head for a while until I figured it out. Let’s say you have a MongoDB model which has an array in it. If you’ve got a reference to a document which you pulled using Mongoose, and you update a value in the array, then you call document.save(), the array will not get updated!

Turns out you need to use the document.markModified(arrayName) function calling document.save().

Here’s an example. Let’s say your model looks like this:

var UserSchema = new Schema({
  //...
  emailSubscriptions: {
    type: Array,
    'default': ["all"]
  }
});
exports.model = mongoose.model('User', UserSchema);

So after you get a user and modify an array (in this case emailSubscriptions), you’ll need to mark it as modified before saving it:

User.findById(userId, function(err, user) {
  user.emailSubscriptions = ['none'];
  user.markModified('emailSubscriptions');
  user.save(function(err, user) {
    //...
  });
});

socket.io: Getting all connected sockets in Node/Express

Sometimes you need a list of all connected sockets in your server side API. This could be, for example, if you want to loop through all the sockets and emit something when a certain event happens, such as some a data model updating or something else.

As of socket.io version 1.3, you have access to all sockets using “socketio.sockets.sockets” programatically.

So for example if you needed all sockets in an Express controller running on node.js, you could access it this way.

In your server side app.js, socket.io is usually configured in express this way:

var app = express();
//...
var socketio = require('socket.io')(server, {
  //...
});

And then, you can save the reference for socketio for later using Express’s app.set() and app.get(). In app.js you would do:

app.set('socketio', socketio);

Then in an express controller, you can access socketio like so:

exports.whateverController = function(req, res, next) {
  var socketio = req.app.get('socketio');
  //...
}

And then you can loop through the sockets:

var sockets = socketio.sockets.sockets;
for(var socketId in sockets)
{
  var socket = sockets[socketId]; //loop through and do whatever with each connected socket
  //...
}

Simple, right?!

Paginating documents/items in MEAN

If you’ve ever scrolled through the Facebook newsfeed, you’ve noticed that the topmost stories are the most recent ones, and as you scroll to the bottom, older ones get loaded over and over as you keep scrolling.

This feature is firstly kind of “cool”, and fits in perfectly in a single page application. It’s also pretty useful from a performance standpoint, since not all of the documents (items your page is displaying, such as Facebook news stories, classified ads, search results, etc.) need to be loaded up front all at once when the user first lands on the page.

Paginating your documents in a MEAN application can be accomplished fairly easily, though it isn’t necessarily obvious. So I thought I’d write about the process I took, and the code I wrote, to get it done.

Let’s start with the AngularJS side of things. I used the ngInfiniteScroll module (https://github.com/sroze/ngInfiniteScroll) to accomplish the continuous scrolling effect. It’s pretty simple to configure, so please read up on the documentation. Essentially it can just be wrapped around an Angular ng-repeat directive, and be configured with a function to call to fetch more documents when the bottom of the page is reached (ngInfiniteScroll does all the calculations internally). Here is an example of what it would look like for getting more “classifieds” from the database to add them to the view:

loading-classifieds

So in the example above, the getMorePosted() function in your controller is called whenever ngInfiniteScroll detects that the user is at the bottom of the page. Note here that ngInfiniteScroll will most likely trigger right when the user lands on the page, unless you pre-load some documents in your controller. I elected getMorePosted() to fetch both the initial set of documents, and every successive set of documents as well. Depending on how you set things up, this may or may not make a difference, but it did for me.

My getMorePosted() function in the controller looks like this (note: it uses a factory called Classified to do the actual getting of classifieds from the API (Express/MongoDB on the server side of MEAN) which I’ll define later):

$scope.initialLoadDone = false;
$scope.loadingClassifieds = false;
$scope.getMorePosted = function() {
  if($scope.loadingClassifieds) return;

  $scope.loadingClassifieds = true;

  if(!$scope.initialLoadDone) {
    Classified.getPosted(function (postedClassifieds) {
      $scope.postedClassifieds = postedClassifieds;
      $scope.loadingClassifieds = false;
      $scope.initialLoadDone = true;
    });
  }
  else
  {
    Classified.getMorePosted(function(err,numberOfClassifiedsGotten) {
      $scope.loadingClassifieds = false;
      if(numberOfClassifiedsGotten==0)
        $scope.noMoreClassifieds=true;
    });
  }
}

A couple things to note here. When the classifieds are being loaded, the $scope.loadingClassifieds flag is set to true. This disables ngInfiniteScroll from attempting to keep loading more classifieds when the bottom is reached, and it can also be used to put up a message to the user that loading is underway (in case it doesn’t happen near instantly due to a slow connection). Furthermore, getMorePosted() also tracks through the $scope.noMoreClassifieds flag when the end has reached (if ever, depending on how many thousands or millions of documents are in your database, and how far down the user scrolls). It does this by measuring the number of documents returned, and if the number equals zero, it means the end of pagination has been reached.

This is how getPosted() and getMorePosted() look like in the Classified factory:

app.factory('Classified', function Classified(ClassifiedResource, ...) {
      var postedClassifieds = [];
      var postedClassifiedsLoaded = false;
      //...
      getPosted: function(callback) {
          var cb = callback || angular.noop;
          if (postedClassifiedsLoaded) {
            //console.log("Sending already-loaded postedClassifieds");
            return cb(postedClassifieds);
          } else {
            return ClassifiedResource.Posted.query(
              function(_postedClassifieds) {
                //console.log("Loading postedClassifieds from webservice");
                postedClassifieds = _postedClassifieds;
                postedClassifiedsLoaded = true;
                return cb(postedClassifieds);
              },
              function(err) {
                return cb(err);
              }).$promise;
          }
        },
        getMorePosted: function(callback) {
          var cb = callback || angular.noop;
          if (!postedClassifiedsLoaded)
            callback();
          else {
            return ClassifiedResource.Posted.query({
                startTime: new Date(postedClassifieds[postedClassifieds.length - 1].posted).getTime()
              },
              function(_postedClassifieds) {
                //console.log("Loading more postedClassifieds from webservice, from before startTime="+postedClassifieds[postedClassifieds.length-1].posted);
                for (var i = 0; i < _postedClassifieds.length; i++)
                  replaceOrInsertInArray(postedClassifieds, _postedClassifieds[i], true);
                return cb(null, _postedClassifieds.length);
              },
              function(err) {
                return cb(err);
              }).$promise;
          }
        },
        //...

And this is how ClassifiedResource looks like:

app.factory('ClassifiedResource', function ($resource) {
  return {
    Posted: $resource(
      '/api/classified/getPosted/:startTime',
      {
      },
      {
      }
    ),
}

So note that in my setup, the service loads and maintains the list of documents (postedClassifieds) within memory. And getPosted() returns that list if it is already loaded, and it also gets the first set of documents. getMorePosted() is where the magic happens. It gets the timestamp of the last classified, and transmits that to the API (server side, Express) which then loads the next “page” after for all documents (classifieds in this case) after that timestamp.

Before we continue to examine the server side, it’s important to note that you’ll need a field to sort by in a descending order (or ascending if you want you want the oldest documents up front). A timestamp value will work great. Otherwise a MongoDB ID could work too, since those are incremental. It will depend on your data. In my case, a timestamp called “posted” was available in my data, and very consistent. Documents could only be removed from before a past timestamp, but not added to in a past timestamp (even then, this wouldn’t be a huge problem). So that works just fine with this pagination approach.

Here is what the server side looks like in Express/NodeJS:

var Classified = require('./classified.model');
exports.getPosted = function(req, res) {
  var startTime = req.params.startTime ? req.params.startTime : null;

  var query = Classified.find(
      {posted: { $ne: null }}
  );
  query.sort('-posted -_id');
  query.limit(20);
  if(startTime)
    query.where({posted: {$lt: new Date().setTime(startTime)}});
  query
    .exec(function (err, classifieds) {
      if(err) { ... }
      return res.status(200).json(classifieds);
    });

}

Note that “Classified” defines my model, which is queried from using Mongoose. I limit the number of documents returned to 20, which works well for my application. And the query is sorted in descending order by the “posted” field, which is a timestamp. You’ll notice a where clause added, which gets only the classifieds posted before the time sent in (“startTime”) from the UI, so that works in conjunction with the sort and returns 20 more classifieds before the “startTime”. Also note that I send the timestamp in milliseconds, which gives a nice clean number that can be sent down to the API from the UI.

And, that’s it!

Something I want to add is that on your client side (in AngularJS) if you end up loading too many documents/items in your ng-repeat, the application performance will greatly degrade. With ngInfiniteScroll, all items on the page are always kept once they’re loaded, even if they’re not in the view currently. There’s another module: https://github.com/angular-ui/ui-scroll which will allow you to destroy and re-create items as they go in and out of the view from the user’s browser as the user scrolls through. This will vastly improve performance when a lot of documents are loaded.

Giving the user a message before resizing images through ng-file-upload

This had been bothering me for a while until I stumbled upon an answer that led me to a solution.

With ng-file-upload (https://github.com/danialfarid/ng-file-upload) for AngularJS, you have the capability to resize images using the ngf-resize directive. It’s very handy since you can put some of the CPU processing burden resizing giant images to smaller sizes on the user, rather than to put it on your own server (if you resize the images after the files are uploaded).

HOWEVER, the problem with ngf-resize is when resizing starts, and especially if the user selects multiple images at once, the user’s browser hangs while the images resize. The bigger the image, the longer it takes. This is irritating, and also confusing, causing the user to wonder what is going on. For the longest time I was trying to figure out how to give the user a message before the resizing actually starts.

I eventually stumbled upon the ngf-before-model-change directive part of ng-file-upload. This allows you to define a function that is called before the model changes (and the images start resizing). This is a perfect place to put up a message to the user that their images are about to be resized, and for them to sit tight for the next few seconds.

Then the ngf-select directive can be used to define a function which is called AFTER the resizing is complete, and this is where you can remove the message to the user.

Full example follows like this. In your JavaScript side of things (in your AngularJS controller) you would do:

$scope.beforeResizingImages = function(images)
{
  blockUI.start($translate.instant('FORMATTING_IMAGES')+"...");
  $scope.$apply();
}

$scope.afterResizingImages = function(images)
{
  blockUI.stop();
}

And then in HTML:

<div ngf-before-model-change="beforeResizingImages($files)" ngf-select="afterResizingImages($files)" />

And that’s it! beforeResizingImages() and afterResizingImages() will be called in the correct order, putting the message up before resizing images (and before the browser hangs for a few seconds for the CPU intensive process), and taking it off after resizing. Note that I use angular-block-ui (https://github.com/McNull/angular-block-ui) to block the UI and put the message up, and of course angular-translate to translate the text for the block message.