Conditional GANs For Painting and Celebrity Generation

Demo: http://adeel.io/sncgan/
Code: https://github.com/AdeelMufti/SNcGAN
Paper: https://arxiv.org/abs/1903.06259

As a 4-month research project for a course at the University of Edinburgh, my group decided to use Generative Adversarial Nets to generate new paintings using the Painter by Numbers dataset. Additionally, we conditioned the paintings, allowing the generation of paintings with certain attributes (such as gender in the case of paintings of portraits). The project turned out to be a big success, and won 2nd place out of 124 other projects in a competition hosted by IBM!

The deep convolutional GAN architecture we chose for the basis of our model enforced a Lipschitz continuity constraint on the spectral norm of the weights for each layer in the discriminator, as proposed by Miyato et al. (2018). This technique, called SN-GAN by the authors, stabilizes the discriminator, allowing the generator to learn the data distribution more optimally.

We built upon the SN-GAN architecture to add conditioning, which involved introducing a one-hot encoded y vector of supervised labels to the generator and discriminator. The y vector provides some information about the image, such as gender in the case of portrait paintings, allowing the generator to get conditioned on the label to generate the class of image specified during test time. Our contribution was a novel model which we called SN-conditional-GAN or SNcGAN, which is illustrated below.

The SNcGAN architecture. The one-hot encoded y label is introduced to the generator and discriminator.

More details can be found in our paper linked above.

After the publication of the paper, we additionally trained a SNcGAN model on the celebA dataset, allowing the generation of conditioned celebrity images.

Inspecting gradients in Chainer

Chainer is my choice of framework when it comes to implementing Neural Networks. It makes working with and trouble shooting deep learning easy.

Printing out the gradients during back propagation to inspect their values is sometimes useful in deep learning, to see if your gradients are as expected and aren’t either exploding (numbers too large) or vanishing (numbers too small). Fortunately, this is easy to do in Chainer.

Chainer provides access to the parameters in your model, and for each parameter, you can check the gradient during the back propagation step, stored in the optimizer (such as SGD or Adam). To access these, you can extend chainer.training.updaters.StandardUpdater() to additionally output the gradients, by defining your own StandardUpdater like so:

class CustomStandardUpdater(chainer.training.updaters.StandardUpdater):
    def __init__(self, train_iter, optimizer, device):
        super(CustomStandardUpdater, self).__init__(
            train_iter, optimizer, device=device)

    def update_core(self):
        super(CustomStandardUpdater, self).update_core()
        optimizer = self.get_optimizer('main')
        for name, param in optimizer.target.namedparams(include_uninit=False):
            print(name, param.grad)

In lines 9-10 you can see the parameters (weights) of your neural network being accessed through the optimizer, and for each parameter, the name and gradient is being output. This StandardUpdater can be attached to your training module as follows:

model = MyChainerModel()
optimizer.setup(model)
optimizer = chainer.optimizers.Adam()
train_iter = chainer.iterators.SerialIterator(train_dataset, batch_size=32, shuffle=True)
updater = CustomStandardUpdater(train_iter, optimizer, gpu)
trainer = training.Trainer(updater, stop_trigger=(100, 'epoch'))
trainer.run()

Calculate CryptoCurrency cross-exchange arbitrage in Java

CryptoCurrency cross-exchange arbitrage means buying or selling CryptoCurrency at separate exchanges to attempt to profit off the price differences for that currency at the exchanges.

Below I’ve pasted the Java code for a small project I undertook, which I later quickly abandoned. But I thought it might be useful to someone else, to give an idea how it could be done. The code is a hodge-podge and not very well put together. Take it for what it is, a quick experiment and proof of concept.

In summary, it looks at two CryptoCurrencies: Bitcoin and Ethereum. Across 3 exchanges. (Can easily be extended to do more currencies and exchanges). It gets the live books using the exchanges’ RESTful API (most provide these), and bases calculations off real asks and bids currently on the order books of each exchange. It then takes all combinations of buying and selling a crypto at one exchange vs another, and outputs what profits (or losses) could be had at the current moment. And that’s it.

It’s a nice tool for exploring what arbitrage opportunities may exist out there, and that’s all. It does not do any trading. Proceed with caution!

package cryparb;

import java.io.BufferedReader;
import java.io.File;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.TimeZone;

import org.json.JSONArray;
import org.json.JSONObject;

@SuppressWarnings("serial")
public class Calculate 
{
	public static final float INITIAL_TRADE_AMOUNT_ETH = 50f;
	public static final float INITIAL_TRADE_AMOUNT_BTC = 10f;
	
	public static final int SLEEP_TO_SIMULATE_CRYPTOCOIN_TRANSFER_DELAYS_IN_MINS_BTC = 30;
	public static final int SLEEP_TO_SIMULATE_CRYPTOCOIN_TRANSFER_DELAYS_IN_MINS_ETH = 15;
	public static final int WEBSERVICE_GET_RETRY_LIMIT = 3;
	public static final int WEBSERVICE_GET_RETRY_AFTER_SECS = 15;
	public static final int RETAIN_BOOKS_CACHE_SECS = 30;
	
	public static final String QUADRIGACX = "quadrigacx";
	public static final String GEMINI = "gemini";
	public static final String GATECOIN = "gatecoin";
	
	public static final LinkedHashMap<String,Exchange> exchanges = new LinkedHashMap<String,Exchange>()
	{{
		put(QUADRIGACX, 
				new Exchange(
						QUADRIGACX, 
						QUADRIGACX_BOOKS_URL, 
						QUADRIGACX_TRADE_FEE_PERCENT, 
						QUADRIGACX_FIXED_WITHDRAWAL_FEE_BTC,
						QUADRIGACX_FIXED_WITHDRAWAL_FEE_ETH,
						QUADRIGACX_BIDS_KEY,
						QUADRIGACX_ASKS_KEY,
						QUADRIGACX_AMOUNT_KEY,
						QUADRIGACX_RATE_KEY
					)
			);
		put(GEMINI, 
				new Exchange(
						GEMINI, 
						GEMINI_BOOKS_URL, 
						GEMINI_TRADE_FEE_PERCENT, 
						GEMINI_FIXED_WITHDRAWAL_FEE_BTC,
						GEMINI_FIXED_WITHDRAWAL_FEE_ETH,
						GEMINI_BIDS_KEY,
						GEMINI_ASKS_KEY,
						GEMINI_AMOUNT_KEY,
						GEMINI_RATE_KEY
					)
			);
		put(GATECOIN, 
				new Exchange(
						GATECOIN, 
						GATECOIN_BOOKS_URL, 
						GATECOIN_TRADE_FEE_PERCENT, 
						GATECOIN_FIXED_WITHDRAWAL_FEE_BTC,
						GATECOIN_FIXED_WITHDRAWAL_FEE_ETH,
						GATECOIN_BIDS_KEY,
						GATECOIN_ASKS_KEY,
						GATECOIN_AMOUNT_KEY,
						GATECOIN_RATE_KEY
					)
			);
	}};
	
	public static final HashMap<String,JSONObject> booksCache = new HashMap<String,JSONObject>();	

	public static final String QUADRIGACX_BOOKS_URL = "https://api.quadrigacx.com/public/orders?book=eth_btc";
	public static final float QUADRIGACX_TRADE_FEE_PERCENT = 0.2f;
	public static final float QUADRIGACX_FIXED_WITHDRAWAL_FEE_BTC = 0f;
	public static final float QUADRIGACX_FIXED_WITHDRAWAL_FEE_ETH = 0f;
	public static final String QUADRIGACX_BIDS_KEY = "buy";
	public static final String QUADRIGACX_ASKS_KEY = "sell";
	public static final String QUADRIGACX_AMOUNT_KEY = "amount";
	public static final String QUADRIGACX_RATE_KEY = "rate";

	public static final String GEMINI_BOOKS_URL = "https://api.gemini.com/v1/book/ethbtc";
	public static final float GEMINI_TRADE_FEE_PERCENT = 0.25f;
	public static final float GEMINI_FIXED_WITHDRAWAL_FEE_BTC = 0f;
	public static final float GEMINI_FIXED_WITHDRAWAL_FEE_ETH = 0f;
	public static final String GEMINI_BIDS_KEY = "bids";
	public static final String GEMINI_ASKS_KEY = "asks";
	public static final String GEMINI_AMOUNT_KEY = "amount";
	public static final String GEMINI_RATE_KEY = "price";

	public static final String GATECOIN_BOOKS_URL = "https://api.gatecoin.com/Public/MarketDepth/ETHBTC";
	public static final float GATECOIN_TRADE_FEE_PERCENT = 0.35f;
	public static final float GATECOIN_FIXED_WITHDRAWAL_FEE_BTC = 0f;
	public static final float GATECOIN_FIXED_WITHDRAWAL_FEE_ETH = 0f;
	public static final String GATECOIN_BIDS_KEY = "bids";
	public static final String GATECOIN_ASKS_KEY = "asks";
	public static final String GATECOIN_AMOUNT_KEY = "volume";
	public static final String GATECOIN_RATE_KEY = "price";

	public static final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36";
	
    public static class Exchange
    {
    	public String name;
    	public String booksUrl;
    	public float tradeFeePercent;
    	public float fixedWithdrawalFeeBtc;
    	public float fixedWithdrawalFeeEth;
    	public String bidsKey;
    	public String asksKey;
    	public String amountKey;
    	public String rateKey;
    	
    	public Exchange(
	    	    	String name,
	    	    	String booksUrl,
	    	    	float tradeFeePercent,
	    	    	float fixedWithdrawalFeeBtc,
	    	    	float fixedWithdrawalFeeEth,
	    	    	String bidsKey,
	    	    	String asksKey,
	    	    	String amountKey,
	    	    	String rateKey
    			)
    	{
        	this.name = name;
        	this.booksUrl = booksUrl;
        	this.tradeFeePercent = tradeFeePercent;
        	this.fixedWithdrawalFeeBtc = fixedWithdrawalFeeBtc;
        	this.fixedWithdrawalFeeEth = fixedWithdrawalFeeEth;
        	this.bidsKey = bidsKey;
        	this.asksKey = asksKey;
        	this.amountKey = amountKey;
        	this.rateKey = rateKey;
    	}
    }
    
    public static class FullLoop 
    {
    	public String originExchange;
    	public String terminationExchange;
    	public String originCryptoCurrency;
    	public float originAmount;
    	public float profit;
    	public Date timestamp;
    	
    	public FullLoop(String originExchange, String terminationExchange, String originCryptoCurrency, float originAmount, float profit, Date timestamp)
    	{
    		this.originExchange = originExchange;
        	this.terminationExchange = terminationExchange;
        	this.originCryptoCurrency = originCryptoCurrency;
        	this.originAmount = originAmount;
    		this.profit = profit;
    		this.timestamp = timestamp;
    	}
    }

    public static String getUrlAsString(String urlString) throws Exception 
    {
        StringBuilder response = new StringBuilder();
        int retry = 0;
        
        while(true)
        {
            try
        	{
                URL url = new URL(urlString);
                HttpURLConnection httpUrlConnection = (HttpURLConnection) url.openConnection();
        		httpUrlConnection.setRequestMethod("GET");
        		httpUrlConnection.setDoInput(true);
        		httpUrlConnection.setDoOutput(false);
        		httpUrlConnection.setInstanceFollowRedirects(false);
        		httpUrlConnection.setConnectTimeout(60000);
        		httpUrlConnection.setReadTimeout(60000);
        		httpUrlConnection.setRequestProperty("User-Agent", USER_AGENT);
                BufferedReader in = new BufferedReader(
                                        new InputStreamReader(
                                        		httpUrlConnection.getInputStream()));
                String inputLine;
                while ((inputLine = in.readLine()) != null) 
                    response.append(inputLine);
                in.close();
                break;
        	}
        	catch(Exception e)
        	{
        		retry++;
        		if(retry==WEBSERVICE_GET_RETRY_LIMIT)
        			break;
        		System.out.println("Unable to fetch "+urlString+". Will sleep and try again.");
        		try { Thread.sleep(WEBSERVICE_GET_RETRY_AFTER_SECS*1000); } catch(Exception e1) { }
        	}
        }

        return response.toString();
    }

	public static float convertEthToBtcAtExchange(String exchangeName, float amountEthToConvertOriginal) throws Exception
	{
		float btcGotten = 0;
		Exchange exchange = exchanges.get(exchangeName);
		
		float amountEthToConvert = amountEthToConvertOriginal - (exchange.tradeFeePercent/100)*amountEthToConvertOriginal;
		
		System.out.println("---");
		System.out.println("Converting eth->btc on "+exchangeName+": "+amountEthToConvertOriginal+"eth, after fee ("+exchange.tradeFeePercent+"%) amount: "+amountEthToConvert+"eth");

		JSONObject exchangeBooks = booksCache.get(exchangeName);
		if(exchangeBooks==null || (exchangeBooks.has("timestamp") && new Date().getTime()-exchangeBooks.getLong("timestamp")>(RETAIN_BOOKS_CACHE_SECS*1000)))
		{
			System.out.println("Refreshing books");
			exchangeBooks = new JSONObject(getUrlAsString(exchange.booksUrl));
			exchangeBooks.put("timestamp", new Date().getTime());
			booksCache.put(exchangeName, exchangeBooks);
		}
		else
		{
			System.out.println("Using cached books");
		}
		
		JSONArray exchangeBids = exchangeBooks.getJSONArray(exchange.bidsKey);
		float ethRemaining = amountEthToConvert;
		for(int i=0; i<exchangeBids.length(); i++)
		{
			JSONObject thisBid = exchangeBids.getJSONObject(i);
			float amount = Float.parseFloat(thisBid.get(exchange.amountKey).toString());
			float rate = Float.parseFloat(thisBid.get(exchange.rateKey).toString());
			float amountEthSpentOnThisOrder = ethRemaining<amount?ethRemaining:amount;
			float amountBtcGottenFromThisOrder = amountEthSpentOnThisOrder*rate;
			btcGotten += amountBtcGottenFromThisOrder;
			ethRemaining -= amountEthSpentOnThisOrder;
			System.out.println("Amount: "+amount+". Rate: "+rate+". Eth remaining before this order: "+(ethRemaining+amountEthSpentOnThisOrder)+". Selling "+amountEthSpentOnThisOrder+"eth, amounting to "+amountBtcGottenFromThisOrder+"btc. Total btc gotten so far: "+btcGotten+". Eth remaining after order: "+ethRemaining);
			if(ethRemaining==0)
				break;
		}
		System.out.println("Total eth spent: "+amountEthToConvertOriginal+". Total btc gotten: "+btcGotten+".");

		//TODO apply btc withdrawl fee if applicable
		
		System.out.println("---");
		return btcGotten;
	}
	
	public static float convertBtcToEthAtExchange(String exchangeName, float amountBtcToConvertOriginal) throws Exception
	{
		float ethGotten = 0;
		Exchange exchange = exchanges.get(exchangeName);
		
		float amountBtcToConvert = amountBtcToConvertOriginal - (exchange.tradeFeePercent/100)*amountBtcToConvertOriginal;

		System.out.println("---");
		System.out.println("Converting btc->eth on "+exchangeName+": "+amountBtcToConvertOriginal+"btc, after fee ("+exchange.tradeFeePercent+"%) amount: "+amountBtcToConvert+"btc");
		
		JSONObject exchangeBooks = booksCache.get(exchangeName);
		if(exchangeBooks==null || (exchangeBooks.has("timestamp") && new Date().getTime()-exchangeBooks.getLong("timestamp")>15000))
		{
			System.out.println("Refreshing books");
			exchangeBooks = new JSONObject(getUrlAsString(exchange.booksUrl));
			exchangeBooks.put("timestamp", new Date().getTime());
			booksCache.put(exchangeName, exchangeBooks);
		}
		else
		{
			System.out.println("Using cached books");
		}

		JSONArray exchangeAsks = exchangeBooks.getJSONArray(exchange.asksKey);
		float btcRemaining = amountBtcToConvert;
		for(int i=0; i<exchangeAsks.length(); i++)
		{
			JSONObject thisAsk = exchangeAsks.getJSONObject(i);
			float amount = Float.parseFloat(thisAsk.get(exchange.amountKey).toString());
			float rate = Float.parseFloat(thisAsk.get(exchange.rateKey).toString());
			float amountBtcAvailableOnThisOrder = amount*rate;
			float amountBtcSpentToThisOrder = btcRemaining<amountBtcAvailableOnThisOrder?btcRemaining:amountBtcAvailableOnThisOrder;
			float amountEthGottenFromThisOrder = amountBtcSpentToThisOrder/rate;
			ethGotten += amountEthGottenFromThisOrder;
			btcRemaining -= amountBtcSpentToThisOrder;
			System.out.println("Amount: "+amount+". Rate: "+rate+". Btc available on this order: "+amountBtcAvailableOnThisOrder+". Btc remaining before this order: "+(btcRemaining+amountBtcSpentToThisOrder)+". Selling "+amountBtcSpentToThisOrder+"btc, amounting to "+amountEthGottenFromThisOrder+"eth. Total eth gotten so far: "+ethGotten+". Btc remaining after order: "+btcRemaining);
			if(btcRemaining==0)
				break;
		}
		System.out.println("Total btc spent: "+amountBtcToConvertOriginal+". Total eth gotten: "+ethGotten+".");

		//TODO apply eth withdrawl fee if applicable

		System.out.println("---");
		return ethGotten;
	}
	
	public static String printProfits(ArrayList<FullLoop> fullLoopList)
	{
		StringBuilder profitsStr = new StringBuilder();
		profitsStr.append("---\n");
		for(int i=0; i<fullLoopList.size(); i++)
		{
			FullLoop fullLoop = fullLoopList.get(i);
			profitsStr.append(fullLoop.timestamp+" - Profit if started with "+fullLoop.originAmount+fullLoop.originCryptoCurrency+" at origin exchange '"+fullLoop.originExchange+"' and ended at termination exchange '"+fullLoop.terminationExchange+"': "+fullLoop.profit+fullLoop.originCryptoCurrency+"\n");
		}
		profitsStr.append("---");
		return profitsStr.toString();
	}

	public static void main(String args[]) throws Exception
	{
		TimeZone.setDefault(TimeZone.getTimeZone("UTC")); 
		
		File logFile = new File("log.txt");
		ArrayList<String> exchangeNames = new ArrayList<String>(exchanges.keySet());
		LinkedHashMap<Date,ArrayList<FullLoop>> fullLoops = new LinkedHashMap<Date,ArrayList<FullLoop>>();
		
		while(true)
		{
			Date currentTimestamp = new Date();
			
			ArrayList<FullLoop> fullLoopList = new ArrayList<FullLoop>();
			fullLoops.put(currentTimestamp, fullLoopList);
			
			for(int i=0; i<exchangeNames.size(); i++)
			{
				for(int j=0; j<exchangeNames.size(); j++)
				{
					if(i==j)
						continue;
					
					String originExchangeName = exchangeNames.get(i);
					String terminationExchangeName = exchangeNames.get(j);
					
					System.out.println("Origin '"+originExchangeName+"'. Termination '"+terminationExchangeName+"'");
					
					float btcGottenOriginExchange = convertEthToBtcAtExchange(originExchangeName, INITIAL_TRADE_AMOUNT_ETH);
					float ethGottenOriginExchange = convertBtcToEthAtExchange(originExchangeName, INITIAL_TRADE_AMOUNT_BTC);
					
					float ethRevertedTerminationExchange = convertBtcToEthAtExchange(terminationExchangeName, btcGottenOriginExchange);
					float btcRevertedTerminationExchange = convertEthToBtcAtExchange(terminationExchangeName, ethGottenOriginExchange);
					
					float profitEth = ethRevertedTerminationExchange-INITIAL_TRADE_AMOUNT_ETH;
					float profitBtc = btcRevertedTerminationExchange-INITIAL_TRADE_AMOUNT_BTC;
					
					fullLoopList.add(new FullLoop(originExchangeName, terminationExchangeName, "eth", INITIAL_TRADE_AMOUNT_ETH, profitEth, currentTimestamp));
					fullLoopList.add(new FullLoop(originExchangeName, terminationExchangeName, "btc", INITIAL_TRADE_AMOUNT_BTC, profitBtc, currentTimestamp));
				}
			}

			Collections.sort(fullLoopList, new Comparator<FullLoop>() {
			    @Override
			    public int compare(FullLoop o1, FullLoop o2) {
			        return new Float(o2.profit).compareTo(o1.profit);
			    }
			});
			
			String profitsStr = printProfits(fullLoopList);
			System.out.println(profitsStr);
			
			Files.write(logFile.toPath(), (profitsStr+"\n").getBytes(), StandardOpenOption.CREATE, StandardOpenOption.APPEND);
			
			Thread.sleep(30000);
		}
		
	}
}

Resuming a HTTP download in Java

Let’s say you’re downloading a large file using Java, like so:

File file = new File("/downloads/some.file.ext");
URL url = new URL("http://some.url/some.file.ext");
HttpURLConnection httpUrlConnection = (HttpURLConnection) url.openConnection();
//...any other httpUrlConnection setup, such as setting headers
BufferedInputStream in = new BufferedInputStream(httpUrlConnection.getInputStream());
FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bout = new BufferedOutputStream(fos, 1024);
try
{
    byte[] data = new byte[1024];
    int x = 0;
    while ((x = in.read(data, 0, 1024)) >= 0) 
    {
    	bout.write(data, 0, x);
    }
}
catch(Exception e)
{
	throw e;
}
finally
{
	if(bout!=null)
	{
		bout.flush();
		bout.close();
	}
	if(fos!=null)
	{
		fos.flush();
		fos.close();
	}
}

Now let’s say the file already exists locally and you need to resume it. Assuming the HTTP server on the other end supports file resume using the HTTP Range header, you can simply do the following:

//Add this right after you initialize httpUrlConnection but before beginning download
if(file.exists())
	httpUrlConnection.setRequestProperty("Range", "bytes="+file.length()+"-");

//And then you'd initialize the file output stream like so:
if(file.exists())
	fos = new FileOutputStream(file, true); //resume download, append to existing file
else
	fos = new FileOutputStream(file);

And that’s it! The Range header you we set for httpUrlConnection does the magic.

Baby Announcement Photo Ideas

This post is a change from your usual Software Development related posts.

My baby girl was born a few weeks ago, and we were playing around with fun ideas on how to announce the pregnancy and birth. We came up with the following.

For pregnancy announcement:
(click for full size)
Pregnancy Announcement

For birth announcement:
Birth Announcement

And I thought I’d share just another photo for good measure:
Lion King

Enjoy!

Creating a rotating proxy in AWS using the Java SDK

AWS EC2 instances can be used to create a HTTP proxy server, so when a client browser using the proxy browses the internet, the AWS EC2 instance’s public IP address effectively becomes their IP address. This may be useful for anonymity, for example if you’re browsing the Internet from home but want to mask your IP address.

Furthermore, you can even have the IP address of your AWS EC2 instance change, by releasing and attaching a new AWS Elastic IP to it, thus “rotating” the public IP of the HTTP proxy. This way you can achieve even more anonymity by using an ever changing IP address.

This is a guide on how to use an AWS EC2 instance (particularly Linux) to create a rotating HTTP proxy. We’ll achieve this using the AWS Java SDK.

To get started start, install tinyproxy on your EC2 instance. SSH into it, and run the following command:

sudo yum -y install tinyproxy –enablerepo=epel

Then edit /etc/tinyproxy/tinyproxy.conf. Note the port, which should be 8888 by default. Make sure the following options are set:

BindSame yes
Allow 0.0.0.0/0
#Listen 192.168.0.1 (make sure this is commented out, meaning line starts with #)
#Bind 192.168.0.1 (make sure this is commented out, meaning line starts with #)

Fire up the tinyproxy by running:

sudo service tinyproxy start

You may also want to add the same command (without the sudo) to /etc/rc.local so tinyproxy is started whenever the EC2 instance is restarted. There’s a proper way to indicate in Linux what services to start on system startup, but I’m forgetting how, and being too lazy to look it up right now :). Adding this command to /etc/rc.local will certainly do the trick.

Now set your web browser (or at the OS level) to use an HTTP proxy by pointing the settings to the public IP address of the EC2 instance. If you don’t know the IP already, you can get it using the AWS EC2 web console. Or by typing the following command on the EC2 server shell:

wget http://ipinfo.io/ip -qO –

You can now go to Google and type in “What is my IP address”. Google will show you, and you’ll notice that it’s not your real IP, but the public IP of the EC2 instance you’re using as a proxy.

Before we move on, let’s set up some security group settings for the EC2 instance to prevent access. This is necessary so not everyone on the Internet can use your proxy server. The best way to go about this is to use the AWS EC2 web console. Navigate to the security group of the EC2 instance, and note the “Group Name” of the security group (we’ll use that later). Add a custom inbound TCP rule to allow traffic from your IP address to port 8888 (or whatever you configured the proxy to run on).

Next what you need to do is to attach new network interfaces to your EC2 instance (one or multiple). This is so that you can have additional network interfaces that you can map an elastic IP address to, as you don’t want to mess with the main network interface so you can have at least one static IP so you can connect to your EC2 instance for whatever reason. The other network interfaces will rotate their public IPs by attaching and releasing to Elastic IPs (AWS seems to have an endless pool of Elastic IPs, you get a new random one every time you release an Elastic IP and reallocate a new one… this works in our favor so we get new IPs every time).

To attach an Elastic Network Interface to your EC2 instance, check out this documentation: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html. Also note that depending on the type of EC2 instance, you only get to allocate a certain number of network interfaces (for t2.micro, I believe the limit is 1 default and 2 additional (so 3 total)). Lastly, take note of the Elastic Network Interface IDs and their corresponding private IP addresses, once you create them. We’ll use them in our java code.

Now, below is a Java code segment that can be used to assign and rotate Elastic IPs to your EC2 instance, which then become the IPs used as proxy. Note at the top of the code there are a number of configuration parameters (static class level variables) that you’ll need to fill out. And of course you’ll need to have the AWS Java SDK in your classpath.

The method associateAll() will associate the Elastic Network Interfaces provided with new Elastic IPs. And the method releaseAll() will detach the Elastic IPs from the Elastic Network Interfaces and release them to the wild (and thus a subsequent associateAll() will then return new IPs). associateAll() will return an ArrayList of Strings corresponding to the new Elastic IPs attached to the EC2 instance. And these IPs can then be used as the HTTP proxy (tinyproxy will automatically bind itself to the proxy port (8888) on the new public IP addresses, so you can connect to them from your client/browser).

Also note that associateAll() will authorize the public IP of the machine running this code by adding it to the EC2 security group to allow connection to TCP port 8888 (or whatever you configured your HTTP proxy port to be) going into the EC2 instance.

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.ArrayList;
import java.util.HashSet;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.ec2.AmazonEC2;
import com.amazonaws.services.ec2.AmazonEC2ClientBuilder;
import com.amazonaws.services.ec2.model.Address;
import com.amazonaws.services.ec2.model.AllocateAddressRequest;
import com.amazonaws.services.ec2.model.AllocateAddressResult;
import com.amazonaws.services.ec2.model.AmazonEC2Exception;
import com.amazonaws.services.ec2.model.AssociateAddressRequest;
import com.amazonaws.services.ec2.model.AssociateAddressResult;
import com.amazonaws.services.ec2.model.AuthorizeSecurityGroupIngressRequest;
import com.amazonaws.services.ec2.model.AuthorizeSecurityGroupIngressResult;
import com.amazonaws.services.ec2.model.DescribeAddressesResult;
import com.amazonaws.services.ec2.model.DomainType;
import com.amazonaws.services.ec2.model.IpPermission;
import com.amazonaws.services.ec2.model.IpRange;
import com.amazonaws.services.ec2.model.ReleaseAddressRequest;
import com.amazonaws.services.ec2.model.ReleaseAddressResult;

public class AWSProxyUtil 
{
	static String SECURITY_GROUP = "security-group-name-of-your-ec2-instance";
	static int DEFAULT_PROXY_PORT_TO_ASSIGN = 8888;
	static String PUBLIC_IP_TO_IGNORE = "1.2.3.4"; 	//This is the IP you want to remain static,
							//so you can connect to your EC2 instance.

	@SuppressWarnings("serial")
	static HashSet<String> NETWORK_ID_PRIVATE_IPs_TO_ASSOCIATE_WITH = new HashSet<String>()
	{{
		//These are the network interface IDs and their private IPs
		//that will be used to attach Elastic IPs to. Format is <ID>:<IP>.
		add("eni-xxxxxxxx:1.2.3.4");
		add("eni-xxxxxxxx:1.2.3.4");
		add("eni-xxxxxxxx:1.2.3.4");
	}};
	
	public static String AWS_ACCESS_KEY_ID = "xxx"; //Your AWS API key info
	public static String AWS_SECRET_KEY_ID = "xxx";

	public static Regions AWS_REGIONS = Regions.US_WEST_2;

	public static void releaseAll() throws Exception
	{
		debugSOP("Relasing elastic IPs");
		
		BasicAWSCredentials awsCreds = new BasicAWSCredentials(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY_ID);
		final AmazonEC2 ec2 = 
				AmazonEC2ClientBuilder
					.standard()
					.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
					.withRegion(AWS_REGIONS)
					.build(); 

		DescribeAddressesResult response = ec2.describeAddresses();

		for(Address address : response.getAddresses()) 
		{
			if(address.getPublicIp().equals(PUBLIC_IP_TO_IGNORE))
			{
				debugSOP(" * Keeping "+address.getPublicIp());
				continue;
			}
			debugSOP(" * Releasing "+address.getPublicIp());
			ReleaseAddressRequest releaseAddressRequest = new ReleaseAddressRequest().withAllocationId(address.getAllocationId());
			ReleaseAddressResult releaseAddressResult = ec2.releaseAddress(releaseAddressRequest);
			debugSOP("   * Result "+releaseAddressResult.toString());
		}
	}
	
	public static ArrayList<String> associateAll() throws Exception
	{
		ArrayList<String> result = new ArrayList<String>();
		
		debugSOP("Associating elastic IPs");
		
		BasicAWSCredentials awsCreds = new BasicAWSCredentials(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY_ID);
		final AmazonEC2 ec2 = 
				AmazonEC2ClientBuilder
					.standard()
					.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
					.withRegion(AWS_REGIONS)
					.build(); 

		DescribeAddressesResult response = ec2.describeAddresses();

		HashSet<String> alreadyAssociated = new HashSet<String>();
		for(Address address : response.getAddresses()) 
		{
			if(address.getPublicIp().equals(PUBLIC_IP_TO_IGNORE))
			{
				continue;
			}
			debugSOP(" * Already associated - Private IP: "+address.getPrivateIpAddress()+", Public IP: "+address.getPublicIp());
			result.add(address.getPublicIp()+":"+DEFAULT_PROXY_PORT_TO_ASSIGN);
			alreadyAssociated.add(address.getNetworkInterfaceId()+":"+address.getPrivateIpAddress());
		}
		
		for(String networkIdPrivateId : NETWORK_ID_PRIVATE_IPs_TO_ASSOCIATE_WITH)
		{
			if(alreadyAssociated.contains(networkIdPrivateId))
				continue;
			
			String fields[] = networkIdPrivateId.split(":");
			String networkId = fields[0];
			String privateIp = fields[1];

			AllocateAddressRequest allocate_request = new AllocateAddressRequest()
				    .withDomain(DomainType.Vpc);

			AllocateAddressResult allocate_response =
			    ec2.allocateAddress(allocate_request);

			String publicIp = allocate_response.getPublicIp();
			String allocation_id = allocate_response.getAllocationId();

			debugSOP(" * Associating Public IP "+publicIp+" to "+networkIdPrivateId);

			AssociateAddressRequest associate_request =
			    new AssociateAddressRequest()
			    	.withNetworkInterfaceId(networkId)
			    	.withPrivateIpAddress(privateIp)
			        .withAllocationId(allocation_id);
			
			AssociateAddressResult associate_response =
				    ec2.associateAddress(associate_request);
			
			debugSOP("   * Result "+associate_response.toString());
			
			result.add(publicIp+":"+DEFAULT_PROXY_PORT_TO_ASSIGN);
		}
		
		debugSOP("Getting public IP address of this machine");
		URL awsCheckIpURL = new URL("http://checkip.amazonaws.com");
		HttpURLConnection awsCheckIphttpUrlConnection = (HttpURLConnection) awsCheckIpURL.openConnection();
		BufferedReader awsCheckIpReader = new BufferedReader(new InputStreamReader(awsCheckIphttpUrlConnection.getInputStream()));
		String thisMachinePublicIp = awsCheckIpReader.readLine();
		
		debugSOP("Authorizing public IP for this machine "+thisMachinePublicIp+" to security group "+SECURITY_GROUP+" for incoming tcp port "+DEFAULT_PROXY_PORT_TO_ASSIGN);
		IpRange ip_range = new IpRange()
			    .withCidrIp(thisMachinePublicIp+"/32");
		IpPermission ip_perm = new IpPermission()
		    .withIpProtocol("tcp")
		    .withToPort(DEFAULT_PROXY_PORT_TO_ASSIGN)
		    .withFromPort(DEFAULT_PROXY_PORT_TO_ASSIGN)
		    .withIpv4Ranges(ip_range);
		AuthorizeSecurityGroupIngressRequest auth_request = new
		    AuthorizeSecurityGroupIngressRequest()
		        .withGroupName(SECURITY_GROUP)
		        .withIpPermissions(ip_perm);
		try
		{
			AuthorizeSecurityGroupIngressResult auth_response =
			    ec2.authorizeSecurityGroupIngress(auth_request);
			debugSOP(" * Result "+auth_response.toString());
		}
		catch(AmazonEC2Exception e)
		{
			if(e.getMessage().contains("already exists"))
				debugSOP(" * Already associated");
			else
			{
				throw e;
			}
		}
		
		debugSOP("Sleeping for 120 seconds to allow EC2 instance(s) to get up to speed.");
		Thread.sleep(120000);

		return result;
	}

	public static void debugSOP(String str)
	{
		System.out.println("[AWSProxyUtil] "+str);
	}
}

An important note on cost! If you allocate and release Elastic IPs too many times, AWS starts charging you (I think the first couple hundred(?) are free, but after that they start charging and it can add up!). And there is also a cost for leaving an Elastic IP address allocated.

Java: Workaround for Array.sort() slowness when sorting on File.lastModified()

Let’s say you have a File[] array gotten using File.listFiles() (or any other means). Now you want to sort that array based on the last modified date of the files. You could whip up the following code:

File directory = new File("/SomeDirectory");
File[] filesList = directory.listFiles();
Arrays.sort(filesList, new Comparator<File>() {
    public int compare(File file1, File file2)
    {
    	return Long.valueOf(file1.lastModified()).compareTo(file2.lastModified());
    } 
});

Note: this will sort them with the latest modified files first.

So this is all well and good, but let’s say your directory has 5 million files in it. Turns out the code above will be extremely slow in sorting the array on such a large list of files (also depending on the speed of your disk drive). The reason for that is because File.lastModified() is called on each file, every time a comparison is made during the sort. Arrays.sort() is an O(n log(n)) operation, so you do the math to see how many times File.lastModified() will be called on each individual file repeatedly in the worst case. (The issue with the repeated File.lastModified() calls is that the method does not cache the last modified timestamp; the call ventures out to the OS and the disk in real time to get the information every time.)

The way around this is simple. Cache the File.lastModified() timestamp. Here’s a code snippet on how to go about that:

public class FileLastModifiedWrapper implements Comparable<FileLastModifiedWrapper> 
{
	public final File file;
	public final long lastModified;

	public FileLastModifiedWrapper(File file) 
	{
		this.file = file;
		lastModified = file.lastModified();
	}

	public int compareTo(FileLastModifiedWrapper other) 
	{
		return Long.compare(this.lastModified, other.lastModified);
	}
}

//...somewhere else:

File directory = new File("/SomeDirectory");
File[] filesList = directory.listFiles();
FileLastModifiedWrapper[] wrappedFilesList = new FileLastModifiedWrapper[filesList.length];
for(int i=0; i<filesList.length; i++)
	wrappedFilesList[i] = new FileLastModifiedWrapper(filesList[i]);
Arrays.sort(wrappedFilesList);
for(int i=0; i<filesList.length; i++)
	filesList[i] = wrappedFilesList[i].file;

And voila! This will sort immensely faster. I noted that on around 100k files, it took just a few seconds, whereas the original code took up to two minutes.

As you see, FileLastModifiedWrapper caches the lastModified timestamp locally. Then we instantiate an array of FileLastModifiedWrapper objects with each file in our filesList. We then sort this new array, and use it to rearrange the original array.

Streaming HTML5 video through Node.js

The server side to support streaming HTML5 video needs to be able to handle headers sent in from the browser. So unfortunately you can’t simply just read the video bytes and send everything back in the response.

The main thing to note here is that the browser may send in a Range HTTP header, which will specify what byte range from the video the browser is requesting. If the range is missing, we can send the whole video starting from byte 0. If it’s there, we’ll want to send only the range of bytes requested. The range header will be in this format:

Range: bytes=0-

…meaning it’s requesting the whole video (or at least it’s the initial request so the size of the video can be determined by the browser from the Content-Length header in the response).

Or:

Range: bytes=5000-10000

…meaning the browser is requesting the video starting from 5000 bytes to 10000 bytes (the user may have skipped ahead).

Also important to note the response headers sent back from the server. These should include:

Accept-Ranges: bytes
Content-Type: video/html
Content-Length: (length)
Content-Range: bytes (start)-(end)/(total)

The Accept-Ranges tells the browser that the server side supports HTML5 video streaming and can take byte ranges. Content-Length sends the total length of the file in bytes. And Content-Range sends the range of the content being returned, in bytes.

So in our Node.js API that handles the HTML5 video streaming requests, we need to be able to handle these headers, and ship the video file back accordingly in ranges (if so requested).

Here’s what the code would look like:

exports.stream = function(req, res) {
  var fileName = req.params.fileName ? req.params.fileName : null;
  if(!fileName)
    return res.status(404).send();

  fs.stat(fileName, function(err, stats) {
    if (err) {
      if (err.code === 'ENOENT') {
        return res.status(404).send();
      }
    }

    var start;
    var end;
    var total = 0;
    var contentRange = false;
    var contentLength = 0;

    var range = req.headers.range;
    if (range)
    {
      var positions = range.replace(/bytes=/, "").split("-");
      start = parseInt(positions[0], 10);
      total = stats.size;
      end = positions[1] ? parseInt(positions[1], 10) : total - 1;
      var chunksize = (end - start) + 1;
      contentRange = true;
      contentLength = chunksize;
    }
    else
    {
      start = 0;
      end = stats.size;
      contentLength = stats.size;
    }

    if(start<=end)
    {
      var responseCode = 200;
      var responseHeader =
      {
        "Accept-Ranges": "bytes",
        "Content-Length": contentLength,
        "Content-Type": "video/mp4"
      };
      if(contentRange)
      {
        responseCode = 206;
        responseHeader["Content-Range"] = "bytes " + start + "-" + end + "/" + total;
      }
      res.writeHead(responseCode, responseHeader);

      var stream = fs.createReadStream(file, { start: start, end: end })
        .on("readable", function() {
          var chunk;
          while (null !== (chunk = stream.read(1024))) {
            res.write(chunk);
          }
        }).on("error", function(err) {
          res.end(err);
        }).on("end", function(err) {
          res.end();
        });
    }
    else
    {
      return res.status(403).send();
    }
  });
};

Let’s analyze this code briefly. There is a function called stream() which is passed the request/response through Node.js. The function looks for a request parameter named fileName, though you can pass around the file identifier, or whatever you please, as long as you have a way to map it to a exact file path on disk.

First we look if the HTTP headers include the Range header. If they do, we can assume the browser requested only a certain range of bytes, so we proceed accordingly. Otherwise if the header is not present, we plan on shipping the whole file back. Node.js’s fs.createReadStream() allows you to create a read stream from a file with specifying what bytes to start and and how many to return (as directed by the browser’s request, or all of it). And then we ship back that stream to the browser.

Note: I did this a while back so I’m forgetting the exact reasons now, but the chunked return is important, otherwise the browser acts funny when playing the video. You may want to adjust the chunk size, but I find 1024 works nicely.

And that’s it!

Dynamic DNS using AWS Route 53 and AWS Java SDK

Route 53 is the Amazon Web Services (AWS) DNS service. Assuming your domain’s DNS is hosted with Route 53, you can create a utility in Java, using the AWS Java SDK, to update a hostname under your domain that points to a dynamic IP address. This may be useful if for example your home’s public IP address changes often, and you want to be able to access it remotely.

To start off, you’ll need to create a hostname in AWS Route 53 that maps to an “A” record pointing to an IP address (doesn’t matter what IP address at this point, since we’ll update it through code later). This can be done manually online, and should be pretty self-explanatory once you open up the Route 53 control panel in the AWS web console.

Let’s say your domain name is domain.com. And you want to dynamically update two hosts: home.domain.com, and dynamic.domain.com, to point to the IP address of a machine that has a dynamically assigned IP.

For this, you can use the following code snippit which I whipped up using the AWS Java SDK documentation for Route 53, and with lots of trial and error:

package utils;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.logging.Logger;

import org.xbill.DNS.ARecord;
import org.xbill.DNS.Lookup;
import org.xbill.DNS.Record;
import org.xbill.DNS.Resolver;
import org.xbill.DNS.SimpleResolver;
import org.xbill.DNS.Type;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.route53.AmazonRoute53;
import com.amazonaws.services.route53.AmazonRoute53ClientBuilder;
import com.amazonaws.services.route53.model.Change;
import com.amazonaws.services.route53.model.ChangeAction;
import com.amazonaws.services.route53.model.ChangeBatch;
import com.amazonaws.services.route53.model.ChangeResourceRecordSetsRequest;
import com.amazonaws.services.route53.model.GetHostedZoneRequest;
import com.amazonaws.services.route53.model.HostedZone;
import com.amazonaws.services.route53.model.ListResourceRecordSetsRequest;
import com.amazonaws.services.route53.model.ListResourceRecordSetsResult;
import com.amazonaws.services.route53.model.ResourceRecord;
import com.amazonaws.services.route53.model.ResourceRecordSet;

public class DynamicDNSUpdater {
	static String AWS_ACCESS_KEY_ID = "xxx";
	static String AWS_SECRET_KEY_ID = "xxx";
	static String ROUT53_HOSTED_ZONE_ID = "Zxxxxxxxxxxxxx";
	static String[] HOSTNAMES_TO_UPDATE = { "home.domain.com", "dynamic.domain.com" };

	static void UpdateIP() throws Exception
	{
		Logger log = ...;

		HashSet<String> hostnamesNeedingUpdate = new HashSet<String>();

		URL awsCheckIpURL = new URL("http://checkip.amazonaws.com");
		HttpURLConnection awsCheckIphttpUrlConnection = (HttpURLConnection) awsCheckIpURL.openConnection();
		BufferedReader awsCheckIpReader = new BufferedReader(new InputStreamReader(awsCheckIphttpUrlConnection.getInputStream()));
		String thisMachinePublicIp = awsCheckIpReader.readLine();
		log.fine("Current public IP of this machine: "+thisMachinePublicIp);
		
	    Resolver resolver = new SimpleResolver("8.8.8.8");
		for(String hostname : HOSTNAMES_TO_UPDATE)
		{
		    Lookup lookup = new Lookup(hostname, Type.A);
		    lookup.setResolver(resolver);
		    Record[] records = lookup.run();
		    String address = ((ARecord) records[0]).getAddress().toString();
		    address = address.substring(address.lastIndexOf("/")+1);
			if(!address.equals(thisMachinePublicIp))
			{
				log.fine("!!! Needs update: "+hostname+". Current IP: "+address+". New public IP: "+thisMachinePublicIp);
				hostnamesNeedingUpdate.add(hostname+".");
			}
		}

		if(hostnamesNeedingUpdate.size()>0)
		{
			BasicAWSCredentials awsCreds = new BasicAWSCredentials(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY_ID);
			AmazonRoute53 route53 = AmazonRoute53ClientBuilder
					.standard()
					.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
					.withRegion(Constants.AWS_REGIONS)
					.build(); 
		    HostedZone hostedZone = route53.getHostedZone(new GetHostedZoneRequest(ROUT53_HOSTED_ZONE_ID)).getHostedZone();

		    ListResourceRecordSetsRequest listResourceRecordSetsRequest = new ListResourceRecordSetsRequest()
		            .withHostedZoneId(hostedZone.getId());
		    ListResourceRecordSetsResult listResourceRecordSetsResult = route53.listResourceRecordSets(listResourceRecordSetsRequest);
		    List<ResourceRecordSet>	resourceRecordSetList = listResourceRecordSetsResult.getResourceRecordSets();
	    	List<Change> changes = new ArrayList<Change>();
		    for(ResourceRecordSet resourceRecordSet : resourceRecordSetList)
		    {
		    	if(resourceRecordSet.getType().equals("A") && hostnamesNeedingUpdate.contains(resourceRecordSet.getName()))
		    	{
			    	List<ResourceRecord> resourceRecords = new ArrayList<ResourceRecord>();
			    	ResourceRecord resourceRecord = new ResourceRecord();
			    	resourceRecord.setValue(thisMachinePublicIp);
			    	resourceRecords.add(resourceRecord);
			    	resourceRecordSet.setResourceRecords(resourceRecords);
			    	Change change = new Change(ChangeAction.UPSERT, resourceRecordSet);
			    	changes.add(change);
			    	log.fine("Updating "+resourceRecordSet.getName()+" to A "+thisMachinePublicIp);
		    	}
		    }
		    if(changes.size()>0)
		    {
		    	ChangeBatch changeBatch = new ChangeBatch(changes);
		    	ChangeResourceRecordSetsRequest changeResourceRecordSetsRequest = new ChangeResourceRecordSetsRequest()
		    			.withHostedZoneId(ROUT53_HOSTED_ZONE_ID)
		    			.withChangeBatch(changeBatch);
		    	route53.changeResourceRecordSets(changeResourceRecordSetsRequest);
		    	log.fine("Done!");
		    }
		    else
		    {
		    	log.fine("None of the specified hostnames found in this zone");
		    }
		}
		else
			log.fine("No updates required!");
	}

	public static void main(String args[]) throws Exception {
		UpdateIP();
	}
}

In order for this to work correctly, you’ll need to set up an AWS API key. This key will need either full access to your AWS account, or at least access to Route53. The documentation for setting it up is available at AWS.

You’ll need to update the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID in the code block above with the key details you get from AWS. And then you’ll need to update ROUT53_HOSTED_ZONE_ID with the Zone ID of your domain hosted in Route 53 (it begins with Z, at least as far as I’ve noticed). And, of course, you’ll need to update HOSTNAMES_TO_UPDATE with the hostname(s) that need to be dynamically updated with the public IP of the machine running this utility.

Here’s a quick breakdown of the code: We start by getting the public IP of the machine this code is running on, and then we look up the IP of the hostnames provided. If these don’t match, that means an update with the new IP is needed. That’s when the com.amazonaws.services.route53.AmazonRoute53 class is used to do the following: using the AWS API access key, it gets a list of all the “A” records for the hosted zone provided. It then loops through the hostnames needing update, and simply posts a com.amazonaws.services.route53.AmazonRoute53.changeResourceRecordSets() with the new public IP of the machine.

And that’s it! There you have it–a Java util that will dynamically update the IP address for the machine it’s running on.

Now in order to run this utility periodically (so it can actually do what it’s meant to, without you manually running it), you can compile the Java code and stick it in a jar, or a simply just copy the .class files in a directory somewhere. (Note: if you’re using Eclipse, it makes it easy to export your project as an executable jar).

Then, if you’re in Linux, you can set up a crontab entry to run every 5 minutes or so and simply run this java utility from the command line.
Granted Java is installed and available in the system path, the command would look something like: java -cp /path/to/MyUtils.jar utils.DynamicDNSUpdater. And if you’re in windows, you can set up a task with the Windows Task Scheduler to run every 5 minutes and run the same command. Pro tip: if using windows, you may want to use “javaw” instead of “java”, if you don’t want a little window to pop up and disappear periodically when you’re in the middle of on the same machine.

Scheduled restarts for your DD-WRT router

DD-WRT is an excellent router OS that can allow you to have complete control over your home or office network. I highly recommend it.

If you want to restart your DD-WRT router every night for whatever reason, there’s an easy way to go about it. You can define bootup/startup scripts for your router to run whenever it starts. These can be defined in Administration -> Commands on your the DD-WRT router’s web interface.

To restart your router every night at 11:45pm, add these commands to startup:

printf "#!/bin/sh\nstartservice run_rc_shutdown; /sbin/reboot" > /tmp/restart_router
chmod a+x /tmp/restart_router
echo "45 23 * * * root /tmp/restart_router" > /tmp/cron.d/restartrouter

And that’s it! (Make sure you either ssh in and run these commands manually when you set this up for the first time, or save the startup script and restart your router once, to kick everything off initially. After that it’ll do it by itself every night at 11:45pm).