Streaming HTML5 video through Node.js

The server side to support streaming HTML5 video needs to be able to handle headers sent in from the browser. So unfortunately you can’t simply just read the video bytes and send everything back in the response.

The main thing to note here is that the browser may send in a Range HTTP header, which will specify what byte range from the video the browser is requesting. If the range is missing, we can send the whole video starting from byte 0. If it’s there, we’ll want to send only the range of bytes requested. The range header will be in this format:

Range: bytes=0-

…meaning it’s requesting the whole video (or at least it’s the initial request so the size of the video can be determined by the browser from the Content-Length header in the response).

Or:

Range: bytes=5000-10000

…meaning the browser is requesting the video starting from 5000 bytes to 10000 bytes (the user may have skipped ahead).

Also important to note the response headers sent back from the server. These should include:

Accept-Ranges: bytes
Content-Type: video/html
Content-Length: (length)
Content-Range: bytes (start)-(end)/(total)

The Accept-Ranges tells the browser that the server side supports HTML5 video streaming and can take byte ranges. Content-Length sends the total length of the file in bytes. And Content-Range sends the range of the content being returned, in bytes.

So in our Node.js API that handles the HTML5 video streaming requests, we need to be able to handle these headers, and ship the video file back accordingly in ranges (if so requested).

Here’s what the code would look like:

exports.stream = function(req, res) {
  var fileName = req.params.fileName ? req.params.fileName : null;
  if(!fileName)
    return res.status(404).send();

  fs.stat(fileName, function(err, stats) {
    if (err) {
      if (err.code === 'ENOENT') {
        return res.status(404).send();
      }
    }

    var start;
    var end;
    var total = 0;
    var contentRange = false;
    var contentLength = 0;

    var range = req.headers.range;
    if (range)
    {
      var positions = range.replace(/bytes=/, "").split("-");
      start = parseInt(positions[0], 10);
      total = stats.size;
      end = positions[1] ? parseInt(positions[1], 10) : total - 1;
      var chunksize = (end - start) + 1;
      contentRange = true;
      contentLength = chunksize;
    }
    else
    {
      start = 0;
      end = stats.size;
      contentLength = stats.size;
    }

    if(start<=end)
    {
      var responseCode = 200;
      var responseHeader =
      {
        "Accept-Ranges": "bytes",
        "Content-Length": contentLength,
        "Content-Type": "video/mp4"
      };
      if(contentRange)
      {
        responseCode = 206;
        responseHeader["Content-Range"] = "bytes " + start + "-" + end + "/" + total;
      }
      res.writeHead(responseCode, responseHeader);

      var stream = fs.createReadStream(file, { start: start, end: end })
        .on("readable", function() {
          var chunk;
          while (null !== (chunk = stream.read(1024))) {
            res.write(chunk);
          }
        }).on("error", function(err) {
          res.end(err);
        }).on("end", function(err) {
          res.end();
        });
    }
    else
    {
      return res.status(403).send();
    }
  });
};

Let’s analyze this code briefly. There is a function called stream() which is passed the request/response through Node.js. The function looks for a request parameter named fileName, though you can pass around the file identifier, or whatever you please, as long as you have a way to map it to a exact file path on disk.

First we look if the HTTP headers include the Range header. If they do, we can assume the browser requested only a certain range of bytes, so we proceed accordingly. Otherwise if the header is not present, we plan on shipping the whole file back. Node.js’s fs.createReadStream() allows you to create a read stream from a file with specifying what bytes to start and and how many to return (as directed by the browser’s request, or all of it). And then we ship back that stream to the browser.

Note: I did this a while back so I’m forgetting the exact reasons now, but the chunked return is important, otherwise the browser acts funny when playing the video. You may want to adjust the chunk size, but I find 1024 works nicely.

And that’s it!

Dynamic DNS using AWS Route 53 and AWS Java SDK

Route 53 is the Amazon Web Services (AWS) DNS service. Assuming your domain’s DNS is hosted with Route 53, you can create a utility in Java, using the AWS Java SDK, to update a hostname under your domain that points to a dynamic IP address. This may be useful if for example your home’s public IP address changes often, and you want to be able to access it remotely.

To start off, you’ll need to create a hostname in AWS Route 53 that maps to an “A” record pointing to an IP address (doesn’t matter what IP address at this point, since we’ll update it through code later). This can be done manually online, and should be pretty self-explanatory once you open up the Route 53 control panel in the AWS web console.

Let’s say your domain name is domain.com. And you want to dynamically update two hosts: home.domain.com, and dynamic.domain.com, to point to the IP address of a machine that has a dynamically assigned IP.

For this, you can use the following code snippit which I whipped up using the AWS Java SDK documentation for Route 53, and with lots of trial and error:

package utils;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.logging.Logger;

import org.xbill.DNS.ARecord;
import org.xbill.DNS.Lookup;
import org.xbill.DNS.Record;
import org.xbill.DNS.Resolver;
import org.xbill.DNS.SimpleResolver;
import org.xbill.DNS.Type;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.route53.AmazonRoute53;
import com.amazonaws.services.route53.AmazonRoute53ClientBuilder;
import com.amazonaws.services.route53.model.Change;
import com.amazonaws.services.route53.model.ChangeAction;
import com.amazonaws.services.route53.model.ChangeBatch;
import com.amazonaws.services.route53.model.ChangeResourceRecordSetsRequest;
import com.amazonaws.services.route53.model.GetHostedZoneRequest;
import com.amazonaws.services.route53.model.HostedZone;
import com.amazonaws.services.route53.model.ListResourceRecordSetsRequest;
import com.amazonaws.services.route53.model.ListResourceRecordSetsResult;
import com.amazonaws.services.route53.model.ResourceRecord;
import com.amazonaws.services.route53.model.ResourceRecordSet;

public class DynamicDNSUpdater {
	static String AWS_ACCESS_KEY_ID = "xxx";
	static String AWS_SECRET_KEY_ID = "xxx";
	static String ROUT53_HOSTED_ZONE_ID = "Zxxxxxxxxxxxxx";
	static String[] HOSTNAMES_TO_UPDATE = { "home.domain.com", "dynamic.domain.com" };

	static void UpdateIP() throws Exception
	{
		Logger log = ...;

		HashSet<String> hostnamesNeedingUpdate = new HashSet<String>();

		URL awsCheckIpURL = new URL("http://checkip.amazonaws.com");
		HttpURLConnection awsCheckIphttpUrlConnection = (HttpURLConnection) awsCheckIpURL.openConnection();
		BufferedReader awsCheckIpReader = new BufferedReader(new InputStreamReader(awsCheckIphttpUrlConnection.getInputStream()));
		String thisMachinePublicIp = awsCheckIpReader.readLine();
		log.fine("Current public IP of this machine: "+thisMachinePublicIp);
		
	    Resolver resolver = new SimpleResolver("8.8.8.8");
		for(String hostname : HOSTNAMES_TO_UPDATE)
		{
		    Lookup lookup = new Lookup(hostname, Type.A);
		    lookup.setResolver(resolver);
		    Record[] records = lookup.run();
		    String address = ((ARecord) records[0]).getAddress().toString();
		    address = address.substring(address.lastIndexOf("/")+1);
			if(!address.equals(thisMachinePublicIp))
			{
				log.fine("!!! Needs update: "+hostname+". Current IP: "+address+". New public IP: "+thisMachinePublicIp);
				hostnamesNeedingUpdate.add(hostname+".");
			}
		}

		if(hostnamesNeedingUpdate.size()>0)
		{
			BasicAWSCredentials awsCreds = new BasicAWSCredentials(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY_ID);
			AmazonRoute53 route53 = AmazonRoute53ClientBuilder
					.standard()
					.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
					.withRegion(Constants.AWS_REGIONS)
					.build(); 
		    HostedZone hostedZone = route53.getHostedZone(new GetHostedZoneRequest(ROUT53_HOSTED_ZONE_ID)).getHostedZone();

		    ListResourceRecordSetsRequest listResourceRecordSetsRequest = new ListResourceRecordSetsRequest()
		            .withHostedZoneId(hostedZone.getId());
		    ListResourceRecordSetsResult listResourceRecordSetsResult = route53.listResourceRecordSets(listResourceRecordSetsRequest);
		    List<ResourceRecordSet>	resourceRecordSetList = listResourceRecordSetsResult.getResourceRecordSets();
	    	List<Change> changes = new ArrayList<Change>();
		    for(ResourceRecordSet resourceRecordSet : resourceRecordSetList)
		    {
		    	if(resourceRecordSet.getType().equals("A") && hostnamesNeedingUpdate.contains(resourceRecordSet.getName()))
		    	{
			    	List<ResourceRecord> resourceRecords = new ArrayList<ResourceRecord>();
			    	ResourceRecord resourceRecord = new ResourceRecord();
			    	resourceRecord.setValue(thisMachinePublicIp);
			    	resourceRecords.add(resourceRecord);
			    	resourceRecordSet.setResourceRecords(resourceRecords);
			    	Change change = new Change(ChangeAction.UPSERT, resourceRecordSet);
			    	changes.add(change);
			    	log.fine("Updating "+resourceRecordSet.getName()+" to A "+thisMachinePublicIp);
		    	}
		    }
		    if(changes.size()>0)
		    {
		    	ChangeBatch changeBatch = new ChangeBatch(changes);
		    	ChangeResourceRecordSetsRequest changeResourceRecordSetsRequest = new ChangeResourceRecordSetsRequest()
		    			.withHostedZoneId(ROUT53_HOSTED_ZONE_ID)
		    			.withChangeBatch(changeBatch);
		    	route53.changeResourceRecordSets(changeResourceRecordSetsRequest);
		    	log.fine("Done!");
		    }
		    else
		    {
		    	log.fine("None of the specified hostnames found in this zone");
		    }
		}
		else
			log.fine("No updates required!");
	}

	public static void main(String args[]) throws Exception {
		UpdateIP();
	}
}

In order for this to work correctly, you’ll need to set up an AWS API key. This key will need either full access to your AWS account, or at least access to Route53. The documentation for setting it up is available at AWS.

You’ll need to update the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID in the code block above with the key details you get from AWS. And then you’ll need to update ROUT53_HOSTED_ZONE_ID with the Zone ID of your domain hosted in Route 53 (it begins with Z, at least as far as I’ve noticed). And, of course, you’ll need to update HOSTNAMES_TO_UPDATE with the hostname(s) that need to be dynamically updated with the public IP of the machine running this utility.

Here’s a quick breakdown of the code: We start by getting the public IP of the machine this code is running on, and then we look up the IP of the hostnames provided. If these don’t match, that means an update with the new IP is needed. That’s when the com.amazonaws.services.route53.AmazonRoute53 class is used to do the following: using the AWS API access key, it gets a list of all the “A” records for the hosted zone provided. It then loops through the hostnames needing update, and simply posts a com.amazonaws.services.route53.AmazonRoute53.changeResourceRecordSets() with the new public IP of the machine.

And that’s it! There you have it–a Java util that will dynamically update the IP address for the machine it’s running on.

Now in order to run this utility periodically (so it can actually do what it’s meant to, without you manually running it), you can compile the Java code and stick it in a jar, or a simply just copy the .class files in a directory somewhere. (Note: if you’re using Eclipse, it makes it easy to export your project as an executable jar).

Then, if you’re in Linux, you can set up a crontab entry to run every 5 minutes or so and simply run this java utility from the command line.
Granted Java is installed and available in the system path, the command would look something like: java -cp /path/to/MyUtils.jar utils.DynamicDNSUpdater. And if you’re in windows, you can set up a task with the Windows Task Scheduler to run every 5 minutes and run the same command. Pro tip: if using windows, you may want to use “javaw” instead of “java”, if you don’t want a little window to pop up and disappear periodically when you’re in the middle of on the same machine.

Scheduled restarts for your DD-WRT router

DD-WRT is an excellent router OS that can allow you to have complete control over your home or office network. I highly recommend it.

If you want to restart your DD-WRT router every night for whatever reason, there’s an easy way to go about it. You can define bootup/startup scripts for your router to run whenever it starts. These can be defined in Administration -> Commands on your the DD-WRT router’s web interface.

To restart your router every night at 11:45pm, add these commands to startup:

printf "#!/bin/sh\nstartservice run_rc_shutdown; /sbin/reboot" > /tmp/restart_router
chmod a+x /tmp/restart_router
echo "45 23 * * * root /tmp/restart_router" > /tmp/cron.d/restartrouter

And that’s it! (Make sure you either ssh in and run these commands manually when you set this up for the first time, or save the startup script and restart your router once, to kick everything off initially. After that it’ll do it by itself every night at 11:45pm).

Routing destination IPs through OpenVPN on DD-WRT routers

DD-WRT is an excellent router OS. It comes with an OpenVPN client, so you can route all (or selective) outbound traffic through a VPN.

The OpenVPN client in DD-WRT makes it easy to specify the source IP addresses that need to have all their traffic routed through the VPN connection. This is done by specifying the source IP addresses (corresponding to devices on your internal network) in policy based routing.

However, there’s no way to specify what destination IP addresses you want routed through the VPN. For example, if it doesn’t matter what the originating source device is on your network, but you want only a certain set of destination addresses (out on the Internet) routed through the VPN connection.

To do this is pretty straight forward. You can ssh into your router (you’ll need to enable ssh management) and run this command:


ip rule add to 1.2.3.4/32 table 10
ip route flush cache

And that’s it. Table 10 is the routing table for the VPN connection, and thus this command will make any traffic destined to the IP 1.2.3.4 route through the VPN connection.

A couple caveats: the OpenVPN client will need to be already connected before you create this routing rule, the routing rule won’t persist through system reboots, and if the OpenVPN connection is dropped for whatever reason, the client software will reconnect it, but the rule will need to be created again.

A crude way around all this is to create a script that runs at bootup/startup for the router. You can define this startup script Administration -> Commands in the DD-WRT interface. The script can run every 5 minutes and create the rule. So when the router is restarted, or if the OpenVPN client connection drops and reconnects, the routing rule will just get re-created. *Note: this is obviously a very inelegant way of getting this done, but it does the job. I’m sure you can improve on this in many ways.

You can add this to your bootup/startup commands for the router:

echo '#!/bin/sh' > /tmp/update-ip-rules.sh
echo 'while true; do' >> /tmp/update-ip-rules.sh
echo '	ip rule add to 1.2.3.4/32 table 10' >> /tmp/update-ip-rules.sh
echo '	ip route flush cache' >> /tmp/update-ip-rules.sh
echo '	sleep 300' >> /tmp/update-ip-rules.sh
echo 'done' >> /tmp/update-ip-rules.sh
chmod +x /tmp/update-ip-rules.sh
nohup /tmp/update-ip-rules.sh >> /dev/null 2>&1 &

Skip certain file extensions from Morgan HTTP logger for Express.js

Morgan is a useful HTTP request logger middleware for Express.js, which plugs in nicely to Node.js and the MEAN stack. More info on Morgan at https://github.com/expressjs/morgan

One useful feature is to add a filter to skip certain files that you don’t want logged. For example you may not want a log of every single get of an image file.

First you define a filter function that returns a boolean for certain file extensions types:

function skipLog (req, res) {
  var url = req.url;
  if(url.indexOf('?')>0)
    url = url.substr(0,url.indexOf('?'));
  if(url.match(/(js|jpg|png|ico|css|woff|woff2|eot)$/ig)) {
    return true;
  }
  return false;
}

The function above will return true for any files with the extension .js, .jpg, .png, (and so on…). Note: you’ll want to return true for skips because you want to evaluate it to skip=true. Also note that the code extracts out the filename from the URL in case there are request parameters attached to it.

Then to use it, you would initiate Morgan like so when setting it up in express.js:

var morgan = require('morgan');
var express = require('express');
var app = express();
//...
app.use(morgan('combined', {stream: accessLogStream, skip: skipLog}));

And that’s it!

Redirecting all stdout and stderr to Logger in Java

This would seem obvious, but it wasn’t to me, so I thought I’d write about it to help out anyone else attempting to accomplish the same. It’s pretty straight forward actually.

Let’s say you have a java.util.logging.Logger object that you’ve initialized, and you want to redirect all stderr (exceptions, System.err.print()’s) and stdout (System.out.print()) to it. You’ll want to use System.setErr and System.setOut to a custom java.io.PrintStream object which writes to your Logger object.

Let’s first define a class to do this for us, and then I’ll explain how it works:

class CustomOutputStream extends OutputStream 
{
	Logger logger;
	Level level;
	StringBuilder stringBuilder;
	
	public CustomOutputStream(Logger logger, Level level)
	{
		this.logger = logger;
		this.level = level;
		stringBuilder = new StringBuilder();
	}
	
	@Override
	public final void write(int i) throws IOException 
	{
		char c = (char) i;
		if(c == '\r' || c == '\n')
		{
			if(stringBuilder.length()>0)
			{
				logger.log(level,stringBuilder.toString());
				stringBuilder = new StringBuilder();
			}
		}
		else
			stringBuilder.append(c);
	}
}

The way this works is by extending OutputStream and overriding the write() method. But write() only takes one character at a time, so essentially you want to buffer each character into a String Builder, to build up the whole line, until you encounter a \r or \n (carriage return, new line), and then submit it to the logger.

To attach CustomOutputStream to your logger:

Logger logger = Logger.getLogger(...);
//...
System.setErr(
		new PrintStream(
			new CustomOutputStream(logger,Level.SEVERE) //Or whatever logger level you want
			)
		);  
System.setOut(
		new PrintStream(
				new CustomOutputStream(logger,Level.FINE) //Or whatever logger level you
			)
		);

Note: if you’ve configured your logger to always include the class/method with the log message, a side effect of this is that the output will not include your original method that wrote the log message to stderr or stdout , but instead your.package.CustomOutputStream.write().

Happy logging!

Using a SOCKS proxy in Java’s HttpURLConnection

Doing a Google Search on how to get Java’s URLConnection or HttpURLConnection to use a SOCKS proxy yields many results on how to pass in arguments to the JVM to set it up, or to call System.setProperty(), which then sets the SOCKS proxy to be used for all HTTP connections through Java. But what if you want to limit it to only certain connections started from HttpURLConnection, or if the proxy address isn’t available until later on?

Here’s a code snippet on how you’d go about doing that programatically.

String proxyString = "127.0.0.1:12345"; //ip:port
String proxyAddress[] = proxyString.split(":");
Proxy proxy = new Proxy(Proxy.Type.SOCKS, new InetSocketAddress(proxyAddress[0], Integer.parseInt(proxyAddress[1])));
URL url = new URL("http://some.website");
HttpURLConnection httpUrlConnection = (HttpURLConnection) url.openConnection(proxy);
//do whatever with httpUrlConnection, it will be connected through the SOCKS proxy!

And that’s it.