Throttling against Egnyte RESTful API from Java

Last week I wrote about how to interface with the Egnyte RESTful API using a custom Java class, so that you can use methods like uploading, downloading, or listing files from anywhere in your code (

It turns out that Egnyte throttles requests to their API. According to their documentation, it is throttled at two transactions per second, so one per 500ms. And if your network connection is fast enough, you will most certainly hit that limit and get denial of service from the API, if you make several calls to API back to back. So, you need to throttle the requests. I’ll talk on how to go about doing that in this post.

Recall the EgnyteHelper class that I created in the aforementioned blog post. A few modifications to this class and methods, with the use of a Timer, can easily overcome this issue. And I’ll show you how you can retry your call recursively if it still gets blocked.

First let’s define some class level constants, variables, a constructor, and two methods to assist. And then I’ll show you how you can use that in one of the Egnyte API helper methods (such as listFileOrFolder()) to prevent against throttling, and assist in retrying.

 private static final int EGNYTE_API_WAIT_MS_BETWEEN_TRANSACTIONS = 501;
 private static final int RETRY_API_CALL_TIMES = 3;

 private Timer timer;
 private Object lock;
 private HashMap < String, Integer > recurseHelperHashMap;

 public EgnyteHelper() {
  lock = new Object();
  recurseHelperHashMap = new HashMap < String, Integer > ();

 private void setTimer() {
  timer = new Timer();
  timer.schedule(new TimerTask() {
   public void run() {
    timer = null;
    synchronized(lock) {

 private void waitForTimerOrSetTimer() {
  if (timer == null)
  else {
   synchronized(lock) {
    try {
    } catch (InterruptedException e) {}

Starting from the top, we define a static variable called EGNYTE_API_WAIT_MS_BETWEEN_TRANSACTIONS. You’ll notice that I’ve set it to 501ms instead of the 500ms limit, so there’s no chance a boundary case is hit and you get blocked. The RETRY_API_CALL_TIMES static variable define show many times you want your method to retry before giving up. The timer variable is used to do the actual waiting, lock variable is used when a method needs to wait for the previous call’s 500ms to elapse, and recurseHelperHashMap is used to assist in recursion (more on that later).

In the constructor we simply just initialize the lock and recurseHelperHashMap. The next two methods are where the magic happens. setTimer() initializes a Timer to wait 501ms, and then calls notify on the lock object, which we catch later. So if a subsequent method is waiting for the lock, it can move on, and do the Egnyte API RESTful call. waitForTimerOrSetTimer() is the method that is actually used in our helper methods. It checks if the timer has already been set, and if not, it simply creates it and moves on. But if the timer is set, it waits on the lock object, until the timer from the previous call releases it, and then it moves on, thus preventing your helper method from moving too fast and hitting the throttle limit.

Next let’s see how we can put this all together and use it in a helper method such as listFileOrFolder(). Here is a modified version of listFileOrFolder() that we explored last time:

 public JSONObject listFileOrFolder(String fullFileOrFolderPathWithSpaces) throws Exception {
  String thisMethodName = Thread.currentThread().getStackTrace()[1].getMethodName();


  JSONObject listing;

  URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs/" + fullFileOrFolderPathWithSpaces).build().toURL();
  URLConnection urlConnection = url.openConnection();
  HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
  httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);


  if (httpUrlConnection.getResponseCode() != HttpURLConnection.HTTP_OK && httpUrlConnection.getResponseCode() != HttpURLConnection.HTTP_CREATED) {
   Map < String, List < String >> hdrs = httpUrlConnection.getHeaderFields();
   if (hdrs.containsKey("Retry-After")) {
    if (!recurseHelperHashMap.containsKey(thisMethodName))
     recurseHelperHashMap.put(thisMethodName, 1);
     recurseHelperHashMap.put(thisMethodName, recurseHelperHashMap.get(thisMethodName) + 1);

    if (recurseHelperHashMap.get(thisMethodName).intValue() <= RETRY_API_CALL_TIMES) {
     Thread.sleep(Integer.parseInt(hdrs.get("Retry-After").get(0)) * 1000);
     return listFileOrFolder(fullFileOrFolderPathWithSpaces);

   listing = new JSONObject();
  } else {
   Scanner scanner = new Scanner(httpUrlConnection.getInputStream());
   String returnJsonString = (scanner.hasNext() ? : "{}");
   listing = new JSONObject(returnJsonString);


  if (recurseHelperHashMap.containsKey(thisMethodName))

  return listing;

Starting from the top, we enter the method, get the method name (I used reflection instead of hardcoding the method name in a string, since I can copy/paste that code in any other helper method), and call waitForOrSetTimer(). This makes the current method call either wait first if a previous call was made within the last 500ms, or it sets the timer if no previous call’s throttle limit was waiting. Next you see that we make the RESTful API call to Egnyte, and then we detect if there was a non-normal HTTP code returned. Fortunately Egnyte sends back an HTTP header in the response if your request does get throttled, called “Retry-After”, which defines the number of seconds we need to wait before retrying. If the header is detected, the code starts with the recursive magic. First it sleeps for the amount of time Egnyte API wants us to sleep (you may want to put EgnyteHelper’s helper method calls on a background thread so your main thread doesn’t get blocked). Then it recursively calls the helper method, up to “RETRY_API_CALL_TIMES” times, before giving up. It does so with the help of the recurseHelperHashMap class level variable, which maps the method name to the number of times the recursion has already happened. Finally when exiting the method, we clean up the entry for this method from recurseHelperHashMap.

And that’s all!

Note that you should have one instance of EgnyteHelper shared per your whole Java application. So if different parts of the application need to access the Egnyte API at the same time, EgnyteHelper can throttle all those requests appropriately using the same timer, application-wide.

Creating Java “Helper” class to interact with Egnyte RESTful API

Egnyte is a HIPAA compliant secure cloud file service. If you’re using Egnyte, and have Java applications that need to access your files on Egnyte, you can create a “Helper” class to interact with the RESTful API Egnyte provides. You could create methods such as “downloadFile()”, “uploadFile()”, “listFileOrFolder()” to use anywhere in your Java code.

As of when this blog post was written, Egnyte does not provide a native Java API. But it does provide a pretty rich RESTful API using web services. The documentation for this API is here:

The API requires an authentication token sent in with the HTTP headers. So first you’ll need to obtain this token using your Egnyte account username and password. More details on obtaining this token is in the documentation referenced above.

Once you have your authorization token, you can get started with making RESTful calls.

Let’s define two variables in a Constants class, called EGNYTEAPI_BASE_URL and EGNYTEAPI_AUTH_TOKEN. The base URL should be “https://(your-domain)”, and the auth token should be the simple alphanumeric string you obtained from Egnyte. You’ll need to use these in all the methods that interact with the Egnyte RESTful API.

Here’s a snippet of what a downloadFile() method would look like:

public File downloadFile(String fullPathWithSpaces) throws Exception {
 File file = null;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs-content/" + fullPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);


 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED) {
  String fileName = fullPathWithSpaces.substring(fullPathWithSpaces.lastIndexOf('/') + 1);
  File tempDir = new File(System.getProperty("") + "/" + Long.toString(System.nanoTime()));
  file = new File(tempDir.getAbsolutePath() + "/" + fileName);
  InputStream is = httpUrlConnection.getInputStream();
  Files.copy(is, file.toPath());


 return file;

As simple as that! The method takes a full path to a file, and returns a java File object, saved in the system temp folder. If the file is not found, or there’s another error, the method simply returns a null File object. Note that the method will handle any URL-sensitive characters by percent-encoding them (, so you won’t need to worry about spaces or other characters in the file name.

Another useful method is one that returns the listing of a file or folder on your Egnyte file server account:

public JSONObject listFileOrFolder(String fullFileOrFolderPathWithSpaces) throws Exception {
 JSONObject listing;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs/" + fullFileOrFolderPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);


 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED) {
  Scanner scanner = new Scanner(httpUrlConnection.getInputStream());
  String returnJsonString = (scanner.hasNext() ? : "{}");
  listing = new JSONObject(returnJsonString);
 } else
  listing = new JSONObject();


 return listing;

This method will also take a full path to a folder or file (can include URL-sensitive characters, such as spaces) and will return either a null JSONObject if there was an error or if the file or folder was not found, or it will return a JSONObject of the listing returned by Egnyte.

The upload file method is a bit more complex, and took me a while to get right. Here it is:

public boolean uploadFile(String fullPathWithSpaces, File file) throws Exception {
 boolean returnValue = false;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs-content/" + fullPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

 String crlf = "\r\n", twoHyphens = "--", boundary = UUID.randomUUID().toString();

 httpUrlConnection.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);

 DataOutputStream request = new DataOutputStream(httpUrlConnection.getOutputStream());

 request.writeBytes(twoHyphens + boundary + crlf);
 request.writeBytes("Content-Disposition: form-data; name=\"file\";filename=\"" + file.getName() + "\"" + crlf);

 InputStream fileInputStream = new FileInputStream(file);

 int bytesRead, bytesAvailable, bufferSize;
 byte[] buffer;
 int maxBufferSize = 1 * 1024 * 1024;

 bytesAvailable = fileInputStream.available();
 bufferSize = Math.min(bytesAvailable, maxBufferSize);
 buffer = new byte[bufferSize];
 bytesRead =, 0, bufferSize);
 while (bytesRead > 0) {
  request.write(buffer, 0, bufferSize);
  bytesAvailable = fileInputStream.available();
  bufferSize = Math.min(bytesAvailable, maxBufferSize);
  bytesRead =, 0, bufferSize);

 request.writeBytes(twoHyphens + boundary + twoHyphens + crlf);


 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED)
  returnValue = true;


 return returnValue;

The uploadFile() method takes a java File object, and a full path for where you want the file uploaded to on your Egnyte account. It then reads the file, and uploads it using HTTP multipart data.

Essentially using the methods above you have the basics down: listing a file or folder, uploading a file, and downloading a file. The same code from these can be used to do other actions too such as making a directory, or anything else that the Egnyte RESTful API allows.


Using MFA to authenticate a user in a Java application

In this post I’ll explore how to set up MFA (Multi Factor Authentication) in a user application written in Java. Be it a stand alone java application, or a web portal. And I’ll examine how the Google Authenticator smartphone app can be used by the user to generate an MFA code.

First, I’ll assume that you already have a data back end which stores user’s details, and a scheme to authenticate usernames with password. MFA will augment the already existing user auth scheme, and not replace it, by adding an added layer of protection in addition to the password.

MFA works with a seed key assigned to a user. The seed key is used along with the current timestamp to generate a numeric code. And this is the code that the user must submit, in order for the server to authenticate. So this seed key must be stored, along with the user’s password, in the database. Hence your user data table may look like:

Table USERS:

To generate the MFA code to store in your USERS table/datastore, you can use the Java code below:

public static class MFACredentials {
 public String user;
 public String encodedKey;
 public String QRBarcodeURL;
 public MFACredentials(String user, String encodedKey, String QRBarcodeURL) {
  this.user = user;
  this.encodedKey = encodedKey;
  this.QRBarcodeURL = QRBarcodeURL;

public static MFACredentials generateNewKeyAndGetQRBarcodeURL(String user, String hostname) {
 int secretSize = 20;

 byte[] buffer = new byte[secretSize];

 new SecureRandom().nextBytes(buffer);

 Base32 codec = new Base32();
 byte[] secretKey = Arrays.copyOf(buffer, secretSize);
 byte[] bEncodedKey = codec.encode(secretKey);
 String encodedKey = new String(bEncodedKey);

 String QRBarcodeURL = getQRBarcodeURL(user, hostname, encodedKey);

 MFACredentials mfaCredentials = new MFACredentials(user, encodedKey, QRBarcodeURL);
 return (mfaCredentials);

private static String getQRBarcodeURL(String user, String host, String secret) {
+ "auth://totp/%s@%s%%3Fsecret%%3D%s"
 return String.format(format, user, host, secret);

Let’s examine. The generateNewKeyAndGetQRBarcodeURL() method is what generates the MFA key, and returns an instance of the MFACredentials class. The returned values include the username, a URL that can be used to generate a QR barcode that can be scanned into Google Authenticator, and the user’s MFA key. The MFA key should then be stored in MFA_KEY field in your USERS data table. Note that the hostname passed in is not important and you can make it whatever you’d like, at least for the way we use MFA in this example.

The QR barcode URL returned by generateNewKeyAndGetQRBarcodeURL() should be sent to the user, with instructions to visit the URL in a browser on a device, and then scan the QR barcode into the Google Authenticator app on a smartphone.

Next, in your UI you’ll want to add a field for a 6 digit numeric code, call it “MFA code” or whatever you please, in addition to the existing username and password fields.

And then lastly, when the MFA code is passed in to your server side for authentication, you’ll want to use the Java code below to verify it:

public static boolean checkCode(String secret, long code)
throws NoSuchAlgorithmException, InvalidKeyException {
 Base32 codec = new Base32();
 byte[] decodedKey = codec.decode(secret);
 long t = new Date().getTime() / TimeUnit.SECONDS.toMillis(30);

 if (secret == null || secret.replaceAll("\\s+", "").equals("") || code == 0)
  return false;

 // Window is used to check codes generated in the near past.
 // You can use this value to tune how far you're willing to go.
 int window = 3;
 for (int i = -window; i <= window; ++i) { long hash = verifyCode(decodedKey, t + i); if (hash == code) { return true; } } return false; } private static int verifyCode(byte[] key, long t) throws NoSuchAlgorithmException, InvalidKeyException { byte[] data = new byte[8]; long value = t; for (int i = 8; i--> 0; value >>>= 8) {
  data[i] = (byte) value;

 SecretKeySpec signKey = new SecretKeySpec(key, "HmacSHA1");
 Mac mac = Mac.getInstance("HmacSHA1");
 byte[] hash = mac.doFinal(data);

 int offset = hash[20 - 1] & 0xF;

 long truncatedHash = 0;
 for (int i = 0; i < 4; ++i) {
  truncatedHash <<= 8;
  truncatedHash |= (hash[offset + i] & 0xFF);

 truncatedHash &= 0x7FFFFFFF;
 truncatedHash %= 1000000;

 return (int) truncatedHash;

The checkCode() method takes the user’s MFA seed key (taken from the MFA_KEY field in the USERS table in our example), and the numeric MFA code that the user passes in. It then authenticates the code against the user’s MFA seed key. You can call the checkCode() method in your server side method that takes the username/password/mfa-code input from the user for authentication. If checkCode() returns true, all is good. If false, that means the MFA code input was incorrect.

And voila, you now have a way to generate an MFA seed key, send a user a URL that generates a QR code that they can use to scan into Google Authenticator, and take the user’s MFA code, to authenticate it. Google Authenticator will generate the MFA code for the user to input when logging in, along with their password.

Disclaimer: The code above was originally not my own, and at this point I’m not sure where I got it from. Though I augmented it a bit, I cannot take credit for it.

Creating a simple ping servlet for AWS Elastic Load Balancer (ELB) Health Check

If you use the AWS Elastic Load Balancer (ELB) you’ll need to decide what to use as an endpoint on your application server for the health checker. This is how the ELB will determine if an instance is healthy, or not.

If you use the default “/” path, this may mean that a session in your application is kicked off every time the health checker connects, which could translate to unnecessary added load on your server (though perhaps it may be negligible).

Furthermore, if you create a static .html file and map it, and point the health checker to that, it could turn out that though your application server is hung, the simple static .html is still getting served. This would not make for an accurate health check, and happened to me in my experience.

The best way to ensure that your application server is online and not hung, without adding extra load to your server, will be to create a simple program that runs on the application server. In the case of a Java application server, you can create a simple servlet, as follows:



import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class Ping extends HttpServlet {
 private String message;

 public void init() throws ServletException {
  message = "Pong";

 public void doGet(HttpServletRequest request,
  HttpServletResponse response)
 throws ServletException, IOException {

  PrintWriter out = response.getWriter();
  out.println(" < h1 > " + message + " < /h1>

And map the servlet to a path in web.xml:


Now, you can configure the ELB health checker to connect to the “/ping” path on your instance. If it times out, or returns an error, that means the application server is not healthy. If it returns a normal HTTP code, then all is good.

Encrypting already existing files in AWS S3 using the AWS Java API

In my last post I covered how to server-side encrypt files in S3 using the AWS Java API. Unfortunately, if you didn’t turn on encryption from the very first day when uploading to S3, you may have some files that are not encrypted. This post will cover an easy block of Java code which you can use to server-side encrypt any existing files that aren’t already, using the AWS Java API.

In summary, you need to loop through all existing files in a bucket, and see which one is not encrypted. And if not encrypted, you set the metadata to turn on server-side encryption, and have to save the file again in S3. Note: this may change the timestamps on your files, but this is essentially the only way through the API to save the metadata for a file to turn on encryption.

Here is the code:

public S3EncryptionMigrator(String bucketName) {
 Logger.getLogger("com.amazonaws.http.AmazonHttpClient").setLevel(Level.OFF); //AWS API outputs too much information, totally flodding the console. Turn it off

 AmazonS3Client amazonS3Client = new AmazonS3Client(...);

 ObjectListing objectListing = amazonS3Client.listObjects(bucketName);
 List s3ObjectSummaries = objectListing.getObjectSummaries();
 while (objectListing.isTruncated()) {
  objectListing = amazonS3Client.listNextBatchOfObjects(objectListing);

 for (S3ObjectSummary s3ObjectSummary: s3ObjectSummaries) {
  String s3ObjectKey = s3ObjectSummary.getKey();
  S3Object unecryptedS3Object = amazonS3Client.getObject(bucketName, s3ObjectKey);
  ObjectMetadata meta = unecryptedS3Object.getObjectMetadata();
  String currentSSEAlgorithm = meta.getSSEAlgorithm();
  if (currentSSEAlgorithm != null && currentSSEAlgorithm.equals(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION))
   continue; //Already encrypted, skip
  meta.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION); //set encryption
  CopyObjectRequest copyObjectRequest = new CopyObjectRequest(bucketName, s3ObjectKey, bucketName, s3ObjectKey);
  amazonS3Client.copyObject(copyObjectRequest); //Save the file
  System.out.println(">> '" + s3ObjectKey + "' encrypted.");

Let’s examine the code. First you instantiate AmazonS3Client with the correct credentials. This should be tailored to your S3 authentication setup.  You start by getting a list of all files in a bucket. Note that you have to loop through objectListing.getObjectSummaries() because only 1000 results are returned at a time. In case you have more than 1000 files, you’ll need to loop through the rest until you get all of them.

Then you loop through the list of files. For each file you check if server-side encryption is already turned on by reading the existing metadata of the file. If not, you set the flag for encryption, and then essentially copy the file onto itself. This will save the new metadata, and will turn on server-side encryption.

Encrypting files in AWS S3 using Java API

If you use AWS S3 Java API, and would like to see how you can encrypt files on S3, this post is for you.

First of all, there are two ways you can encrypt files in S3. One is to encrypt files on the server side, and one is to encrypt files on the client side. With using the server side option, you don’t have to worry about too much. S3 encrypts the files for you when they are written to disk, and decrypts them when they are read, seamlessly. With the client side option, the client (your application) has to encrypt files before transmitting them to S3, and decrypt them after receiving the file from S3.

In this post I’ll cover server side encryption. We opted to use this one because it’s just simpler, and seamless. You don’t have to worry about encrypting/decrypting files yourself, nor do you have to worry about the key.

I’m assuming that you’re already familiar with the AWS Java API. For most things related to S3, AWS provides a class called AmazonS3Client. Once you have AmazonS3Client instantiated with your configuration, you will need to enable encryption in the matadata for each file you upload.


File fileForUpload = new File(...);
AmazonS3Client amazonS3Client = new AmazonS3Client(...);
ObjectMetadata meta = new ObjectMetadata();
amazonS3Client.putObject(s3Bucket, s3FullDestinationPath, new FileInputStream(fileForUpload), meta);

Let’s examine. First you instantiate the File you want to upload, and AmazonS3Client. Next you set the metadata on the file. This includes setting the content type of the file (important because having the wrong content-type can cause issues down the line), and sets the encryption flag for the file. Then when you upload the file using AmazonS3Client.putObject(…), the file will be encrypted by S3 before it is stored, and automatically decrypted when it is retrieved, all by S3’s servers. And that’s it!

Note that according to AWS Java API documentation, AmazonS3Client uses SSL under the hood so you don’t have to worry about transmitting unencrypted files over the network.