Tag: java

Sorting a JSON Array in Java

There are a number of approaches you can take. But a simple one is to first convert it to a Java Collection, and use Collections.sort() with a custom comparator.

The example I’ll follow consists of an org.json.JSONArray which is has (and only has) org.json.JSONObject’s. So a json array of json objects, which is pretty common. Say you want to sort the JSONObjects in the JSONArray, based on a key in the JSONObject.

Let’s start by converting a JSONArray to a Collection of JSONObjects, using the java List type:

List<JSONObject> myJsonArrayAsList = new ArrayList<JSONObject>();
for (int i = 0; i < myJsonArray.length(); i++)
    myJsonArrayAsList.add(myJsonArray.getJSONObject(i));

Now you can use Collections.sort() with a custom comparator. Let’s say you have a key named “key” in each json object, which maps to an int, and you want to sort on int value. You would use the following code:

Collections.sort(myJsonArrayAsList, new Comparator<JSONObject>() {
    @Override
    public int compare(JSONObject jsonObjectA, JSONObject jsonObjectB) {
    	int compare = 0;
    	try
    	{
    		int keyA = jsonObjectA.getInt("key");
    		int keyB = jsonObjectB.getInt("key");
    		compare = Integer.compare(keyA, keyB);
    	}
    	catch(JSONException e)
    	{
    		e.printStackTrace();
    	}
    	return compare;
    }
});

That’ll do. Now, let’s take it a step further. Let’s say the values in the key field of each json object are Strings, and you want to sort based on the Strings, but in a particular order. You want the string “oranges” to come first, “bananas” to come second, “pineapples” to come third, and “apples” to come last.

An easy way to go about this is to create a HashMap, and assign these strings integer values. Then use those integer values to compare. Here’s what the code would look like for that:

Collections.sort(myJsonArrayAsList, new Comparator<JSONObject>() {
    @Override
    public int compare(JSONObject jsonObjectA, JSONObject jsonObjectB) {
    	int compare = 0;
    	try
    	{
			HashMap<String,Integer> fruitTypeSorts = new HashMap<String,Integer>();
			fruitTypeSorts.put("orange", 1);
			fruitTypeSorts.put("bananas", 2);
			fruitTypeSorts.put("pineapples", 3);
			fruitTypeSorts.put("apples", 4);
			int valueA=fruitTypeSorts.get(jsonObjectA.getString("key"));
			int valueB=fruitTypeSorts.get(jsonObjectB.getString("key"));
			return Integer.compare(valueA, valueB);
    	}
    	catch(JSONException e)
    	{
    		e.printStackTrace();
    	}
    	return compare;
    }
});

And voila! Now you have the List sorted based on the key value, with objects with key values oranges being first, then bananas, and so on.

To tie it all together, you want to convert it back to a JSONArray. To put the sorted JSONObjects back into your original array, simply:

myJsonArray = new JSONArray();
for (int i = 0; i < myJsonArrayAsList.size(); i++) {
	myJsonArray.put(myJsonArrayAsList.get(i));
}

Recovering Java keystore password

So I hadn’t updated my Android app in 5 years. I had worked on an update for the past couple months, and was finally ready to upload it. I knew I had my Java keystore backed up, and the password written down. But come time to deploy the updated .apk, I couldn’t get into the keystore! The password I had written down 5 years ago was not it.

I was very frustrated, and disheartened after all the recent effort. Should I delete my old application, and upload the update as a completely new one? But then I’d lose all my current users.

I knew there are only a handful of passwords that I use, sometimes with slight variations, and sometimes multiple of my standard passwords put together. So I thought that I’d try to crack it.

Enter the Java based Android Keystore Password Recovery tool: http://maxcamillo.github.io/android-keystore-password-recover/. It’s a pretty configurable brute forcer. And it worked for me. Though it took several tries to get it just right so it could crack it, and I’ll explore how to best use it in this post.

The best way to use Android Keystore Password Recovery tool is with the Smart Dictionary mode, which tries variations of the words in a dictionary. It appends the words in the dictionary together in all orders, and adds numbers to it. It can also try upper case/lower case for the first letter of each word when trying variations. This is useful when you know roughly what your password was, but you don’t remember the case, or the numbers you added around it. And it’s especially useful if you concatenate two or more of your passwords together, but don’t remember in what order or what variation of your password (as was my case).

To run it, download the .jar (in my case I imported the source into Eclipse and ran it that way). Make sure you have java installed, and you can run it from the command line.

First you’ll want to create a dictionary of passwords that you usually use. Try to think of all variations and break them up into as little of pieces as possible. For example, if your password is CatManDu57, but you don’t remember if you variated it to 57DuManCat or something else, then break the dictionary into the following words:

Cat

Man

Du

57

Try to think of all the variations of cases that you use. A limitation in Android Keystore Password Recovery is that it is only capable of trying the first letter of the word with uppercase and lower case. So you may want to put in different combinations of your word, i.e. cAt, CaT, cAT (all added to the dictionary) should do it. And Android Keystore Password Recovery will try all the variations.

To run it, use the following command:

java -jar AndroidKeystoreBrute_v1.06.jar -m 3 -k “/path/to/mykeystore” -d “/path/to/wordlist.txt”

And voila! It will try all combinations in your wordlist, individually, or concatenated together.

Note that there’s a very useful and configurable feature called “pieces”, which isn’t very obvious from the documentation. You can define the number of pieces from the wordlist that can be concatenated together to form your password. This can narrow down the search for the password a lot so that the tool can go only up to a certain number of combinations. For example if we go back to the CatManDu57 example, that’s 4 total pieces. And since you know that there are at least 4 pieces together, and a maximum of 4 pieces, you’d use between “4 and 4” and it will try all combinations that way. I believe the default number of pieces it goes to is 1 to 64, and that will take much longer to run, so you’re better off limiting this if you can remember the number of pieces. Note: that a piece can be repeated within the same password combination it tries, so for our example with 4 pieces, one of the combinations will be CatCatCatCat, and one will be 57575757, and every combination in between, in every order. So you can see this can be very useful if you have some idea what your password was, especially if you remember pieces of it.

Some other notable parameters are as follow: -l sets the minimum password length. So if you knew it’s at least a certain number of characters, you should use it to narrow it down. -firstchars sets the first few characters of your password, and should be used if you remember for sure what the first few characters of the password were. -p uses common replacements such as “@” or “a”. And -onlylower makes it not mess with the case of the first letter of each word in your dictionary… very useful if you know exactly the case and that’s what you should use in the dictionary for each piece (which was my case).

The better and more precise the parameters you provide based on what password you use, obviously, the more likelihood of result. Try different combinations and try to narrow it down. I had to go through lots of iterations of wordlists and parameters (especially pieces) to get it just right.

Another mode that I found useful was the brute forcing method, but limited to a certain set of characters. This one will require editing the source code. If you knew you used some variation of CatManDu57, and want to simply brute force with just those characters, you can edit BrutePasswd.java in the source, and limit it to the characters:

‘C’,’c’,’A’,’a’,’T’,’t’,’M’,’m’,’N’,’n’,’D’,’d’,’U’,’u’,’5′,’7′.

And this will do a brute of all combinations of all permutations starting from whatever length you specify, and will most certainly crack your password eventually if you’re sure those were the only characters in your password. Of course this becomes a lot slower, and may take weeks or months, the more the characters you have and the longer the length of the password.

Another notable mention for a cracker is John The Ripper (http://www.openwall.com/john/). Most importantly, it has support for Java keystores. Download the community enhanced version, and use the “keystore2john” utility to convert it to John the Rippper (JtR) format. I tried this but unfortunately didn’t quite figure out how to configure it with my own dictionary. Nor could I get it to repeat pieces of the dictionary which I knew was important in my password. I did try it on a bogus keystore I created with a simple password (“test57”), just to test, and it did crack that very quickly, since it does have a built in dictionary.

Performance note: while Android Keystore Password Recovery works pretty fast (I was getting 200,000 combinations of passwords per second), John The Ripper beat that in several orders of magnitude. JtR was guessing over a million combinations of passwords per second! Holy batman.

Last notable mention is Hashcat (https://hashcat.net/hashcat/), which is supposed to be a very fast password cracker, and can also use GPU to do the hashing which will greatly speed it up even more than JtR. With this, you can set up a cluster of GPU-enabled EC2 machines in AWS (or anywhere else), and run it and it will eventually crack. Unfortunately, at the time of writing this post, hashcat does not support Java keystores, though I did see in a few places (such as a poll on what feature to add) that it’s a commonly requested feature in Hashcat. Maybe if I have time, I’ll contribute to that. But for now I’m happy because I have my lost keystore password recovered!

Good luck!

P.S. check out my Android app at: https://play.google.com/store/apps/details?id=com.synappze.stoptimer and https://github.com/AdeelMufti/StopTimer

Throttling against Egnyte RESTful API from Java

Last week I wrote about how to interface with the Egnyte RESTful API using a custom Java class, so that you can use methods like uploading, downloading, or listing files from anywhere in your code (https://adeelscorner.wordpress.com/2016/07/24/creating-java-helper-class-to-interact-with-egnyte-restful-api/).

It turns out that Egnyte throttles requests to their API. According to their documentation, it is throttled at two transactions per second, so one per 500ms. And if your network connection is fast enough, you will most certainly hit that limit and get denial of service from the API, if you make several calls to API back to back. So, you need to throttle the requests. I’ll talk on how to go about doing that in this post.

Recall the EgnyteHelper class that I created in the aforementioned blog post. A few modifications to this class and methods, with the use of a Timer, can easily overcome this issue. And I’ll show you how you can retry your call recursively if it still gets blocked.

First let’s define some class level constants, variables, a constructor, and two methods to assist. And then I’ll show you how you can use that in one of the Egnyte API helper methods (such as listFileOrFolder()) to prevent against throttling, and assist in retrying.

 private static final int EGNYTE_API_WAIT_MS_BETWEEN_TRANSACTIONS = 501;
 private static final int RETRY_API_CALL_TIMES = 3;

 private Timer timer;
 private Object lock;
 private HashMap < String, Integer > recurseHelperHashMap;

 public EgnyteHelper() {
  lock = new Object();
  recurseHelperHashMap = new HashMap < String, Integer > ();
 }

 private void setTimer() {
  timer = new Timer();
  timer.schedule(new TimerTask() {
   @Override
   public void run() {
    timer.cancel();
    timer = null;
    synchronized(lock) {
     lock.notify();
    }
   }
  }, EGNYTE_API_WAIT_MS_BETWEEN_TRANSACTIONS);
 }

 private void waitForTimerOrSetTimer() {
  if (timer == null)
   setTimer();
  else {
   synchronized(lock) {
    try {
     lock.wait();
    } catch (InterruptedException e) {}
    setTimer();
   }
  }
 }

Starting from the top, we define a static variable called EGNYTE_API_WAIT_MS_BETWEEN_TRANSACTIONS. You’ll notice that I’ve set it to 501ms instead of the 500ms limit, so there’s no chance a boundary case is hit and you get blocked. The RETRY_API_CALL_TIMES static variable define show many times you want your method to retry before giving up. The timer variable is used to do the actual waiting, lock variable is used when a method needs to wait for the previous call’s 500ms to elapse, and recurseHelperHashMap is used to assist in recursion (more on that later).

In the constructor we simply just initialize the lock and recurseHelperHashMap. The next two methods are where the magic happens. setTimer() initializes a Timer to wait 501ms, and then calls notify on the lock object, which we catch later. So if a subsequent method is waiting for the lock, it can move on, and do the Egnyte API RESTful call. waitForTimerOrSetTimer() is the method that is actually used in our helper methods. It checks if the timer has already been set, and if not, it simply creates it and moves on. But if the timer is set, it waits on the lock object, until the timer from the previous call releases it, and then it moves on, thus preventing your helper method from moving too fast and hitting the throttle limit.

Next let’s see how we can put this all together and use it in a helper method such as listFileOrFolder(). Here is a modified version of listFileOrFolder() that we explored last time:

 public JSONObject listFileOrFolder(String fullFileOrFolderPathWithSpaces) throws Exception {
  String thisMethodName = Thread.currentThread().getStackTrace()[1].getMethodName();

  waitForTimerOrSetTimer();

  JSONObject listing;

  URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs/" + fullFileOrFolderPathWithSpaces).build().toURL();
  URLConnection urlConnection = url.openConnection();
  HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
  httpUrlConnection.setRequestMethod("GET");
  httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

  httpUrlConnection.connect();

  if (httpUrlConnection.getResponseCode() != HttpURLConnection.HTTP_OK && httpUrlConnection.getResponseCode() != HttpURLConnection.HTTP_CREATED) {
   Map < String, List < String >> hdrs = httpUrlConnection.getHeaderFields();
   if (hdrs.containsKey("Retry-After")) {
    if (!recurseHelperHashMap.containsKey(thisMethodName))
     recurseHelperHashMap.put(thisMethodName, 1);
    else
     recurseHelperHashMap.put(thisMethodName, recurseHelperHashMap.get(thisMethodName) + 1);

    if (recurseHelperHashMap.get(thisMethodName).intValue() <= RETRY_API_CALL_TIMES) {
     Thread.sleep(Integer.parseInt(hdrs.get("Retry-After").get(0)) * 1000);
     return listFileOrFolder(fullFileOrFolderPathWithSpaces);
    }
   }

   listing = new JSONObject();
  } else {
   Scanner scanner = new Scanner(httpUrlConnection.getInputStream());
   scanner.useDelimiter("\\A");
   String returnJsonString = (scanner.hasNext() ? scanner.next() : "{}");
   scanner.close();
   listing = new JSONObject(returnJsonString);
  }

  httpUrlConnection.disconnect();

  if (recurseHelperHashMap.containsKey(thisMethodName))
   recurseHelperHashMap.remove(thisMethodName);

  return listing;
 }

Starting from the top, we enter the method, get the method name (I used reflection instead of hardcoding the method name in a string, since I can copy/paste that code in any other helper method), and call waitForOrSetTimer(). This makes the current method call either wait first if a previous call was made within the last 500ms, or it sets the timer if no previous call’s throttle limit was waiting. Next you see that we make the RESTful API call to Egnyte, and then we detect if there was a non-normal HTTP code returned. Fortunately Egnyte sends back an HTTP header in the response if your request does get throttled, called “Retry-After”, which defines the number of seconds we need to wait before retrying. If the header is detected, the code starts with the recursive magic. First it sleeps for the amount of time Egnyte API wants us to sleep (you may want to put EgnyteHelper’s helper method calls on a background thread so your main thread doesn’t get blocked). Then it recursively calls the helper method, up to “RETRY_API_CALL_TIMES” times, before giving up. It does so with the help of the recurseHelperHashMap class level variable, which maps the method name to the number of times the recursion has already happened. Finally when exiting the method, we clean up the entry for this method from recurseHelperHashMap.

And that’s all!

Note that you should have one instance of EgnyteHelper shared per your whole Java application. So if different parts of the application need to access the Egnyte API at the same time, EgnyteHelper can throttle all those requests appropriately using the same timer, application-wide.

Creating Java “Helper” class to interact with Egnyte RESTful API

Egnyte is a HIPAA compliant secure cloud file service. If you’re using Egnyte, and have Java applications that need to access your files on Egnyte, you can create a “Helper” class to interact with the RESTful API Egnyte provides. You could create methods such as “downloadFile()”, “uploadFile()”, “listFileOrFolder()” to use anywhere in your Java code.

As of when this blog post was written, Egnyte does not provide a native Java API. But it does provide a pretty rich RESTful API using web services. The documentation for this API is here: https://developers.egnyte.com/docs

The API requires an authentication token sent in with the HTTP headers. So first you’ll need to obtain this token using your Egnyte account username and password. More details on obtaining this token is in the documentation referenced above.

Once you have your authorization token, you can get started with making RESTful calls.

Let’s define two variables in a Constants class, called EGNYTEAPI_BASE_URL and EGNYTEAPI_AUTH_TOKEN. The base URL should be “https://(your-domain).egnyte.com/pubapi/v1/”, and the auth token should be the simple alphanumeric string you obtained from Egnyte. You’ll need to use these in all the methods that interact with the Egnyte RESTful API.

Here’s a snippet of what a downloadFile() method would look like:

public File downloadFile(String fullPathWithSpaces) throws Exception {
 File file = null;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs-content/" + fullPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestMethod("GET");
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

 httpUrlConnection.connect();

 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED) {
  String fileName = fullPathWithSpaces.substring(fullPathWithSpaces.lastIndexOf('/') + 1);
  File tempDir = new File(System.getProperty("java.io.tmpdir") + "/" + Long.toString(System.nanoTime()));
  tempDir.mkdir();
  file = new File(tempDir.getAbsolutePath() + "/" + fileName);
  InputStream is = httpUrlConnection.getInputStream();
  Files.copy(is, file.toPath());
 }

 httpUrlConnection.disconnect();

 return file;
}

As simple as that! The method takes a full path to a file, and returns a java File object, saved in the system temp folder. If the file is not found, or there’s another error, the method simply returns a null File object. Note that the method will handle any URL-sensitive characters by percent-encoding them (https://en.wikipedia.org/wiki/Percent-encoding), so you won’t need to worry about spaces or other characters in the file name.

Another useful method is one that returns the listing of a file or folder on your Egnyte file server account:

public JSONObject listFileOrFolder(String fullFileOrFolderPathWithSpaces) throws Exception {
 JSONObject listing;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs/" + fullFileOrFolderPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestMethod("GET");
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

 httpUrlConnection.connect();

 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED) {
  Scanner scanner = new Scanner(httpUrlConnection.getInputStream());
  scanner.useDelimiter("\\A");
  String returnJsonString = (scanner.hasNext() ? scanner.next() : "{}");
  scanner.close();
  listing = new JSONObject(returnJsonString);
 } else
  listing = new JSONObject();

 httpUrlConnection.disconnect();

 return listing;
}

This method will also take a full path to a folder or file (can include URL-sensitive characters, such as spaces) and will return either a null JSONObject if there was an error or if the file or folder was not found, or it will return a JSONObject of the listing returned by Egnyte.

The upload file method is a bit more complex, and took me a while to get right. Here it is:

public boolean uploadFile(String fullPathWithSpaces, File file) throws Exception {
 boolean returnValue = false;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs-content/" + fullPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestMethod("POST");
 httpUrlConnection.setDoOutput(true);
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

 String crlf = "\r\n", twoHyphens = "--", boundary = UUID.randomUUID().toString();

 httpUrlConnection.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);

 DataOutputStream request = new DataOutputStream(httpUrlConnection.getOutputStream());

 request.writeBytes(twoHyphens + boundary + crlf);
 request.writeBytes("Content-Disposition: form-data; name=\"file\";filename=\"" + file.getName() + "\"" + crlf);
 request.writeBytes(crlf);

 InputStream fileInputStream = new FileInputStream(file);

 int bytesRead, bytesAvailable, bufferSize;
 byte[] buffer;
 int maxBufferSize = 1 * 1024 * 1024;

 bytesAvailable = fileInputStream.available();
 bufferSize = Math.min(bytesAvailable, maxBufferSize);
 buffer = new byte[bufferSize];
 bytesRead = fileInputStream.read(buffer, 0, bufferSize);
 while (bytesRead > 0) {
  request.write(buffer, 0, bufferSize);
  bytesAvailable = fileInputStream.available();
  bufferSize = Math.min(bytesAvailable, maxBufferSize);
  bytesRead = fileInputStream.read(buffer, 0, bufferSize);
 }
 fileInputStream.close();

 request.writeBytes(crlf);
 request.writeBytes(twoHyphens + boundary + twoHyphens + crlf);

 request.flush();
 request.close();

 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED)
  returnValue = true;

 httpUrlConnection.disconnect();

 return returnValue;
}

The uploadFile() method takes a java File object, and a full path for where you want the file uploaded to on your Egnyte account. It then reads the file, and uploads it using HTTP multipart data.

Essentially using the methods above you have the basics down: listing a file or folder, uploading a file, and downloading a file. The same code from these can be used to do other actions too such as making a directory, or anything else that the Egnyte RESTful API allows.

Enjoy!

Encrypting already existing files in AWS S3 using the AWS Java API

In my last post I covered how to server-side encrypt files in S3 using the AWS Java API. Unfortunately, if you didn’t turn on encryption from the very first day when uploading to S3, you may have some files that are not encrypted. This post will cover an easy block of Java code which you can use to server-side encrypt any existing files that aren’t already, using the AWS Java API.

In summary, you need to loop through all existing files in a bucket, and see which one is not encrypted. And if not encrypted, you set the metadata to turn on server-side encryption, and have to save the file again in S3. Note: this may change the timestamps on your files, but this is essentially the only way through the API to save the metadata for a file to turn on encryption.

Here is the code:

public S3EncryptionMigrator(String bucketName) {
 Logger.getLogger("com.amazonaws.http.AmazonHttpClient").setLevel(Level.OFF); //AWS API outputs too much information, totally flodding the console. Turn it off

 AmazonS3Client amazonS3Client = new AmazonS3Client(...);

 ObjectListing objectListing = amazonS3Client.listObjects(bucketName);
 List s3ObjectSummaries = objectListing.getObjectSummaries();
 while (objectListing.isTruncated()) {
  objectListing = amazonS3Client.listNextBatchOfObjects(objectListing);
  s3ObjectSummaries.addAll(objectListing.getObjectSummaries());
 }

 for (S3ObjectSummary s3ObjectSummary: s3ObjectSummaries) {
  String s3ObjectKey = s3ObjectSummary.getKey();
  S3Object unecryptedS3Object = amazonS3Client.getObject(bucketName, s3ObjectKey);
  ObjectMetadata meta = unecryptedS3Object.getObjectMetadata();
  String currentSSEAlgorithm = meta.getSSEAlgorithm();
  unecryptedS3Object.close();
  if (currentSSEAlgorithm != null && currentSSEAlgorithm.equals(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION))
   continue; //Already encrypted, skip
  meta.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION); //set encryption
  CopyObjectRequest copyObjectRequest = new CopyObjectRequest(bucketName, s3ObjectKey, bucketName, s3ObjectKey);
  copyObjectRequest.setNewObjectMetadata(meta);
  amazonS3Client.copyObject(copyObjectRequest); //Save the file
  System.out.println(">> '" + s3ObjectKey + "' encrypted.");
 }
}

Let’s examine the code. First you instantiate AmazonS3Client with the correct credentials. This should be tailored to your S3 authentication setup.  You start by getting a list of all files in a bucket. Note that you have to loop through objectListing.getObjectSummaries() because only 1000 results are returned at a time. In case you have more than 1000 files, you’ll need to loop through the rest until you get all of them.

Then you loop through the list of files. For each file you check if server-side encryption is already turned on by reading the existing metadata of the file. If not, you set the flag for encryption, and then essentially copy the file onto itself. This will save the new metadata, and will turn on server-side encryption.

Encrypting files in AWS S3 using Java API

If you use AWS S3 Java API, and would like to see how you can encrypt files on S3, this post is for you.

First of all, there are two ways you can encrypt files in S3. One is to encrypt files on the server side, and one is to encrypt files on the client side. With using the server side option, you don’t have to worry about too much. S3 encrypts the files for you when they are written to disk, and decrypts them when they are read, seamlessly. With the client side option, the client (your application) has to encrypt files before transmitting them to S3, and decrypt them after receiving the file from S3.

In this post I’ll cover server side encryption. We opted to use this one because it’s just simpler, and seamless. You don’t have to worry about encrypting/decrypting files yourself, nor do you have to worry about the key.

I’m assuming that you’re already familiar with the AWS Java API. For most things related to S3, AWS provides a class called AmazonS3Client. Once you have AmazonS3Client instantiated with your configuration, you will need to enable encryption in the matadata for each file you upload.

Example:

File fileForUpload = new File(...);
AmazonS3Client amazonS3Client = new AmazonS3Client(...);
ObjectMetadata meta = new ObjectMetadata();
meta.setContentType(URLConnection.guessContentTypeFromName(fileForUpload.getName()));
meta.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
amazonS3Client.putObject(s3Bucket, s3FullDestinationPath, new FileInputStream(fileForUpload), meta);

Let’s examine. First you instantiate the File you want to upload, and AmazonS3Client. Next you set the metadata on the file. This includes setting the content type of the file (important because having the wrong content-type can cause issues down the line), and sets the encryption flag for the file. Then when you upload the file using AmazonS3Client.putObject(…), the file will be encrypted by S3 before it is stored, and automatically decrypted when it is retrieved, all by S3’s servers. And that’s it!

Note that according to AWS Java API documentation, AmazonS3Client uses SSL under the hood so you don’t have to worry about transmitting unencrypted files over the network.

Getting the number of active sessions in Vaadin

Last week I explored how to get the number of active sessions that Tomcat reports: https://adeelscorner.wordpress.com/2016/03/13/getting-the-number-of-active-sessions-in-tomcat/

The problem with that metric is that some of those sessions, or all, may not actually be active users. What if someone is connected to your java web application (Tomcat in this case), but has just been sitting idle, for hours or even days. They’re not really using your application per se. However, Tomcat will report them as active users, especially if your application has a heartbeat, or a long polling push connection established.

So an additional metric that is useful in determining if your java application is actually in use is to track the number of active sessions within your java application. In our case we’re using Vaadin, so that’s what this post is geared towards. However, you’ll see the general idea here and can apply this to any user based application.

Before I continue, I want to point out a useful Vaadin add-on to measure idle time: https://vaadin.com/directory#!addon/idle. This is pretty handy in checking the actual idle time for a user.

The other important thing to explore first is as follows. Each Vaadin Session is based on a cookie, so if a browser has multiple tabs open to the same application server running Vaadin, the Vaadin Session will be shared among those tabs, because the cookie is shared. Hence, when a Vaadin session ends (perhaps when the user logs out, or times out), all user browser tabs open corresponding to that session are disconnected.

Let’s get started. At the core of how this will work is a public static HashMap which we can use to track a browser tab that belongs to a particular Vaadin session, and how long it’s been active or idle. Since this is a static data structure, it will be shared among all sessions at the Tomcat level. So it’s pretty handy in tracking all sessions on that particular Tomcat server. We’ll be able to track not only how many browser tabs a user has open to your application, but how long they’ve been idle in each tab.

I created a class called ApplicationLevelTracker to assist, which is pretty simple really. It’s as follows:

public class ApplicationLevelTracker {
 public static HashMap < String, Pair < Boolean, Long >> browserTabUniqueIDToIdleTimeHashMap = new HashMap < String, Pair < Boolean, Long >> ();

 private String thisBrowserTabUniqueID;

 public ApplicationLevelTracker(int userID, String vaadinSessionID) {
  thisBrowserTabUniqueID = userID + ":" + vaadinSessionID + ":" + UUID.randomUUID().toString();
  registerIdle(false);
 }

 public void registerIdle(boolean idle) {
  Date date = new Date();
  browserTabUniqueIDToIdleTimeHashMap.put(thisBrowserTabUniqueID, new Pair < Boolean, Long > (idle, date.getTime()));
 }

 public void deRegisterThisTab() {
  browserTabUniqueIDToIdleTimeHashMap.remove(thisBrowserTabUniqueID);
 }

 public void deRegisterAllTabsForThisVaadinSession() {
  String userID = thisBrowserTabUniqueID.split(":")[0];
  String vaadinSessionID = thisBrowserTabUniqueID.split(":")[1];
  Set < String > keySet = ApplicationLevelTracker.browserTabUniqueIDToIdleTimeHashMap.keySet();
  List < String > browserTabUniqueIDsToRemove = new ArrayList < String > ();
  for (String key: keySet) {
   String _userID = key.split(":")[0];
   String _vaadinSessionID = key.split(":")[1];
   if (_userID.equals(userID) && _vaadinSessionID.equals(vaadinSessionID))
    browserTabUniqueIDsToRemove.add(key);
  }
  for (String browserTabUniqueID: browserTabUniqueIDsToRemove) {
   browserTabUniqueIDToIdleTimeHashMap.remove(browserTabUniqueID);
  }
 }
}

Here’s a breakdown of the class, and the methods: The constructor takes the user’s ID (any identifier you have for your user, in our case an integer), and the Vaadin Session ID. Using these keys, and a random ID, each browser tab that a user opens to your application will get its own unique ID created in the constructor.

When a user goes idle, registerIdle(true) is called. When a user is active, registerIdle(false) is called. This way we can track whether they are active or idle in the corresponding browser tab. These methods will be used by the Vaadin Idle add-on (more on that soon).

When a browser tab is closed, deRegisterThisTab() is called. When the Vaadin session is being ended (for example, you may do that when a user logs out from your application, or when Vaadin determines the user has timed out), deRegisterAllTabsForThisVaadinSession() is meant to be called, to clean them out of the tracking HashMap.

Next, to actually make use of the ApplicationLevelTracker, you’ll have to set it up and use it properly in your Vaadin UI class, where your application is created and each user’s instance is initiated. So in your UI class that creates your Vaadin application (the class usually extends Vaadin’s “UI”), you’ll need to set up the following code for all this to come together:

First, define this variable in your UI class:

private ApplicationLevelTracker applicationLevelTracker = null;

Next, when the views are being created, after the User is logged in (note: you should set it up so this code is called on every new creation of the UI class, and not just the creation of a new Vaadin Session (the same Vaadin session can have multiple UIs attached to it)), you’ll want to set up ApplicationLevelTracker:

applicationLevelTracker = new ApplicationLevelTracker(user.USER_ID, VaadinSession.getCurrent().getSession().getId());
Idle.track(UI.getCurrent(), 120000, new Idle.Listener() {
 @Override
 public void userInactive() {
  applicationLevelTracker.registerIdle(true);
 }
 @Override
 public void userActive() {
  applicationLevelTracker.registerIdle(false);
 }
});

Notice that the Vaadin Idle add-on is used here. We create an instance of ApplicationLevelTracker with the user’s ID, and the Vaadin session ID, which registers this browser tab in static (shared) memory. And then when the user goes idle, or when the user goes active, as reported by the Idle add-on, we register that with ApplicationLevelTracker.

Now that things are setup, we need to be sure they are cleaned up when the user goes away.

You can use this neat trick to register whenever that particular browser tab is closed, or when the user navigates away from your application. This is where you’ll want to call applicationLevelTracker.deRegisterThisTab():

JavaScript.getCurrent().addFunction("browserIsLeaving", new JavaScriptFunction() {
 @Override
 public void call(JsonArray arguments) {
  if (applicationLevelTracker != null)
   applicationLevelTracker.deRegisterThisTab();
 }
});
Page.getCurrent().getJavaScript().execute("window.onbeforeunload = function (e) { var e = e || window.event; browserIsLeaving(); return; };");

You’ll also want to use applicationLevelTracker.deRegisterThisTab() when the Vaadin detach listener is fired. A detach listener corresponds to the UI, and not the whole Vaadin session. So it will be fired when one particular instance of UI belonging to a singular browser tab (which may one of several tabs belonging to the same Vaadin session) is detected to have gone away, this code will be called to clean up that particular browser tab. You can add the following block of code in the init() method in your Vaadin UI class:

addDetachListener(new DetachListener() {
 @Override
 public void detach(DetachEvent event) {
  if (applicationLevelTracker != null)
   applicationLevelTracker.deRegisterThisTab();
 }
});

Lastly, wherever in your code the user logs out, and you end the Vaadin session, you’ll want to call applicationLevelTracker.deRegisterAllTabsForThisVaadinSession():

if (applicationLevelTracker != null)
 applicationLevelTracker.deRegisterAllTabsForThisVaadinSession();

And that’s it! Now you have an application wide HashMap which fully tracks all individual browser tabs open to your application, and information on how long the user has been active or idle in each tab. In my next blog post I’ll cover how to actually pull this information and use it in a meaningful way to make a determination of how many users are using your application at any given moment.

Before wrapping up I want to point out one final thing on how to interpret this tracking information after you’ve started collecting it. If a user is connected, and is idle for too long (say 2 hours, or several days), you can reliably know that they are not using the application. They are either connected and left their machine running and browser open, just sitting idle. Or they’re gone (no longer connected) and their session was never cleaned up for whatever reason. If the user has been active for a long time (say, several days), that means something is wrong too, unless they really have been working every minute in your application for the last several days without a break. In this case you can assume the user is not really there, that their browser connection went away and their tracking didn’t get cleaned up for whatever reason (for example, Vaadin didn’t fire the detach listener reliably, which I’ve noticed happens sometimes though it may be fixed in new Vaadin versions). The only case you can be pretty certain that a user is using your application at the present moment is if they’ve just been idle for a short period (say within a 30 minute window), or if they’ve been active within a short period (say 30 minutes again).

Getting the number of active sessions in Tomcat

This is really useful when you’re trying to determine if your Java application running on a Tomcat server is currently serving any clients. For example, if your server is under use, you can avoid taking it offline for maintenance or upgrade. Or, you can use it to determine the load on a server.

Essentially Tomcat has some internal statistics that can be accessed using JMX (Java Management Extension). More information and details for that are available here: https://tomcat.apache.org/tomcat-8.0-doc/monitoring.html

The Tomcat JMX stats can be accessed in several ways. This article will cover how to access them from within the tomcat serer, which was our requirement. Since the application deployed to the tomcat server will be running locally on the JVM running tomcat, JMX can be accessed locally from the Java code, and no remote setup is required.

The code is as simple as this:

MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
ObjectName objectName = new ObjectName("Catalina:type=Manager,context=/,host=localhost");
int activeSessions = (Integer) mBeanServer.getAttribute(objectName, "activeSessions");

And that’s it! You’ll get the number of active sessions tomcat is reporting. And since this is running within the application deployed to tomcat, no remote setup or custom JVM arguments for tomcat are required.

Vaadin upgrade from 7.4.5 to 7.6.1

I was recently tasked with upgrading our Vaadin applications at work from version 7.4.5 to version 7.6.1. I came across an issue that took me a long time to figure out, so I thought I’d share in case anyone else is running into the same.

The process went pretty smoothly at first. We use Ivy for our projects, and simply updating the Vaadin version in ivy.xml allowed all the new library files to be downloaded. Then I re-compiled the widgetset, and took one of our applications out for a spin.

Lo and behold I got an error stating: “Failed to load the bootstrap javascript: ./VAADIN/vaadinBootstrap.js?v=7.6.1”. I did some investigating, and noticed that the vaadinBootstrap.js served to the browser looked like a binary file, whereas the browser was trying to interpret it as a JavaScript file.

I eventually realized that the vaadinBootstrap.js is compressed. I did everything I could to verify that the compression filters were correct, but the browser for some reason seemed like it was not uncompressing vaadinBootstrap.js before trying to interpret it.

Turns out that in prior versions of Vaadin, with the Vaadin demo applications, Vaadin had stuck in a CompressionFilter using net.sf.ehcache.constructs.web.filter.GzipFilter to web.xml. It compressed all *.js, *.html, *.css files (which only works if the browser/client announces it supports it in the HTTP headers) before serving it. HOWEVER, in the new version of Vaadin (7.6.1) it seems that there are some new compression filters already built in! Hence, the content (in this case vaadinBootstrap.js) was compressed twice! So the browser would uncompress it as it should, but that would yield a second compressed file! Sheesh. Had almost pulled my hair out before I figured this out.

The solution was to get rid of all the custom compression filters using net.sf.ehcache.constructs.web.filter.GzipFilter from our web.xml. And then everything started working smoothly.

Vaadin: Executing custom JavaScript from a thread, or loading custom JavaScript functions into global scope

As you may know already, in Vaadin you can use com.vaadin.ui.JavaScript.getCurrent().execute(…) function to execute some custom JavaScript on the client browser.

  1. Executing custom JavaScript from a thread:
    The above works well as long as the JavaScript execute() method is being called on the main UI thread. However, if JavaScript.getCurrent().execute() is called from a background thread, the JavaScript won’t get executed until there is a periodic refresh of the UI, or there’s a UI event (triggered by the user, such as a mouse click somewhere). This can seem to cause erratic behavior, with the JavaScript executing at unpredictable times. (Side note: any Vaadin UI access/manipulation from a background thread needs to be done inside com.vaadin.UI.getCurrent().access(new Runnable() { … });, and also note that you want to do your time-consuming heavy lifting first (such as retrieving data from the back end) and then go into UI.getCurrent().access(…) to manipulate the UI).To get around this problem, simply use Vaadin Push. You’ll need to enable push if not already (see Vaadin documentation on how). Then depending on the push mode you’ve used (manual or automatic) you’ll either need to call com.vaadin.ui.UI.getCurrent().push() or not (for manual mode you’ll need to call the push() method, for automatic mode it will be called after the runnable you send to the UI.access(…) method finishes executing). So call get JavaScript.execute(…), and then UI.push() last. Example:

    new Thread() {
    public void run() {
        //some long running background work
             UI.getCurrent.access(new Runnable() {
            public void run() {
                JavaScript.getCurrent().execute("alert('Background Task Complete!');");
                            UI.getCurent().push();
            }
        });
    }
}.start();
  2. Loading custom JavaScript functions into global scope:
    This is extremely useful so you can define a JavaScript function which you can use later from JavaScript.getCurrent().execute(…), such as inside an event (a button click, for example). However, the JavaScript function will need to be in the global scope by injecting it into thetag for the HTML page served by Vaadin. To do this, use the following code while your Vaadin view is being created, or is being enter()’ed.

    StringBuilder script = new StringBuilder(); script .append("var head = document.getElementsByTagName('head')[0];") .append("var script = document.createElement('script');") .append("script.content='function sayHello() { alert(\"Hello!\");}';") .append("head.appendChild(script);"); JavaScript.getCurrent().execute(script.toString());

    Note: as I mentioned, a JavaScript function can only be loaded this way when the view is being created, or in the view’s enter() method. To create a function this way AFTER the page is already loaded (such as through some event), you’ll need to use Vaadin Push, and call UI.getCurrent().push() after the JavaScript.getCurrent().execute() even though you’re not on a background thread.

  3. You can define a function in the script tag being created above, which can then be called later on through JavaScript.getCurrent().execute(“sayHello();”); perhaps inside a Vaadin button click listener.

Enjoy!