Throttling against Egnyte RESTful API from Java

Last week I wrote about how to interface with the Egnyte RESTful API using a custom Java class, so that you can use methods like uploading, downloading, or listing files from anywhere in your code (https://adeelscorner.wordpress.com/2016/07/24/creating-java-helper-class-to-interact-with-egnyte-restful-api/).

It turns out that Egnyte throttles requests to their API. According to their documentation, it is throttled at two transactions per second, so one per 500ms. And if your network connection is fast enough, you will most certainly hit that limit and get denial of service from the API, if you make several calls to API back to back. So, you need to throttle the requests. I’ll talk on how to go about doing that in this post.

Recall the EgnyteHelper class that I created in the aforementioned blog post. A few modifications to this class and methods, with the use of a Timer, can easily overcome this issue. And I’ll show you how you can retry your call recursively if it still gets blocked.

First let’s define some class level constants, variables, a constructor, and two methods to assist. And then I’ll show you how you can use that in one of the Egnyte API helper methods (such as listFileOrFolder()) to prevent against throttling, and assist in retrying.

 private static final int EGNYTE_API_WAIT_MS_BETWEEN_TRANSACTIONS = 501;
 private static final int RETRY_API_CALL_TIMES = 3;

 private Timer timer;
 private Object lock;
 private HashMap < String, Integer > recurseHelperHashMap;

 public EgnyteHelper() {
  lock = new Object();
  recurseHelperHashMap = new HashMap < String, Integer > ();
 }

 private void setTimer() {
  timer = new Timer();
  timer.schedule(new TimerTask() {
   @Override
   public void run() {
    timer.cancel();
    timer = null;
    synchronized(lock) {
     lock.notify();
    }
   }
  }, EGNYTE_API_WAIT_MS_BETWEEN_TRANSACTIONS);
 }

 private void waitForTimerOrSetTimer() {
  if (timer == null)
   setTimer();
  else {
   synchronized(lock) {
    try {
     lock.wait();
    } catch (InterruptedException e) {}
    setTimer();
   }
  }
 }

Starting from the top, we define a static variable called EGNYTE_API_WAIT_MS_BETWEEN_TRANSACTIONS. You’ll notice that I’ve set it to 501ms instead of the 500ms limit, so there’s no chance a boundary case is hit and you get blocked. The RETRY_API_CALL_TIMES static variable define show many times you want your method to retry before giving up. The timer variable is used to do the actual waiting, lock variable is used when a method needs to wait for the previous call’s 500ms to elapse, and recurseHelperHashMap is used to assist in recursion (more on that later).

In the constructor we simply just initialize the lock and recurseHelperHashMap. The next two methods are where the magic happens. setTimer() initializes a Timer to wait 501ms, and then calls notify on the lock object, which we catch later. So if a subsequent method is waiting for the lock, it can move on, and do the Egnyte API RESTful call. waitForTimerOrSetTimer() is the method that is actually used in our helper methods. It checks if the timer has already been set, and if not, it simply creates it and moves on. But if the timer is set, it waits on the lock object, until the timer from the previous call releases it, and then it moves on, thus preventing your helper method from moving too fast and hitting the throttle limit.

Next let’s see how we can put this all together and use it in a helper method such as listFileOrFolder(). Here is a modified version of listFileOrFolder() that we explored last time:

 public JSONObject listFileOrFolder(String fullFileOrFolderPathWithSpaces) throws Exception {
  String thisMethodName = Thread.currentThread().getStackTrace()[1].getMethodName();

  waitForTimerOrSetTimer();

  JSONObject listing;

  URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs/" + fullFileOrFolderPathWithSpaces).build().toURL();
  URLConnection urlConnection = url.openConnection();
  HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
  httpUrlConnection.setRequestMethod("GET");
  httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

  httpUrlConnection.connect();

  if (httpUrlConnection.getResponseCode() != HttpURLConnection.HTTP_OK && httpUrlConnection.getResponseCode() != HttpURLConnection.HTTP_CREATED) {
   Map < String, List < String >> hdrs = httpUrlConnection.getHeaderFields();
   if (hdrs.containsKey("Retry-After")) {
    if (!recurseHelperHashMap.containsKey(thisMethodName))
     recurseHelperHashMap.put(thisMethodName, 1);
    else
     recurseHelperHashMap.put(thisMethodName, recurseHelperHashMap.get(thisMethodName) + 1);

    if (recurseHelperHashMap.get(thisMethodName).intValue() <= RETRY_API_CALL_TIMES) {
     Thread.sleep(Integer.parseInt(hdrs.get("Retry-After").get(0)) * 1000);
     return listFileOrFolder(fullFileOrFolderPathWithSpaces);
    }
   }

   listing = new JSONObject();
  } else {
   Scanner scanner = new Scanner(httpUrlConnection.getInputStream());
   scanner.useDelimiter("\\A");
   String returnJsonString = (scanner.hasNext() ? scanner.next() : "{}");
   scanner.close();
   listing = new JSONObject(returnJsonString);
  }

  httpUrlConnection.disconnect();

  if (recurseHelperHashMap.containsKey(thisMethodName))
   recurseHelperHashMap.remove(thisMethodName);

  return listing;
 }

Starting from the top, we enter the method, get the method name (I used reflection instead of hardcoding the method name in a string, since I can copy/paste that code in any other helper method), and call waitForOrSetTimer(). This makes the current method call either wait first if a previous call was made within the last 500ms, or it sets the timer if no previous call’s throttle limit was waiting. Next you see that we make the RESTful API call to Egnyte, and then we detect if there was a non-normal HTTP code returned. Fortunately Egnyte sends back an HTTP header in the response if your request does get throttled, called “Retry-After”, which defines the number of seconds we need to wait before retrying. If the header is detected, the code starts with the recursive magic. First it sleeps for the amount of time Egnyte API wants us to sleep (you may want to put EgnyteHelper’s helper method calls on a background thread so your main thread doesn’t get blocked). Then it recursively calls the helper method, up to “RETRY_API_CALL_TIMES” times, before giving up. It does so with the help of the recurseHelperHashMap class level variable, which maps the method name to the number of times the recursion has already happened. Finally when exiting the method, we clean up the entry for this method from recurseHelperHashMap.

And that’s all!

Note that you should have one instance of EgnyteHelper shared per your whole Java application. So if different parts of the application need to access the Egnyte API at the same time, EgnyteHelper can throttle all those requests appropriately using the same timer, application-wide.

Creating Java “Helper” class to interact with Egnyte RESTful API

Egnyte is a HIPAA compliant secure cloud file service. If you’re using Egnyte, and have Java applications that need to access your files on Egnyte, you can create a “Helper” class to interact with the RESTful API Egnyte provides. You could create methods such as “downloadFile()”, “uploadFile()”, “listFileOrFolder()” to use anywhere in your Java code.

As of when this blog post was written, Egnyte does not provide a native Java API. But it does provide a pretty rich RESTful API using web services. The documentation for this API is here: https://developers.egnyte.com/docs

The API requires an authentication token sent in with the HTTP headers. So first you’ll need to obtain this token using your Egnyte account username and password. More details on obtaining this token is in the documentation referenced above.

Once you have your authorization token, you can get started with making RESTful calls.

Let’s define two variables in a Constants class, called EGNYTEAPI_BASE_URL and EGNYTEAPI_AUTH_TOKEN. The base URL should be “https://(your-domain).egnyte.com/pubapi/v1/”, and the auth token should be the simple alphanumeric string you obtained from Egnyte. You’ll need to use these in all the methods that interact with the Egnyte RESTful API.

Here’s a snippet of what a downloadFile() method would look like:

public File downloadFile(String fullPathWithSpaces) throws Exception {
 File file = null;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs-content/" + fullPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestMethod("GET");
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

 httpUrlConnection.connect();

 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED) {
  String fileName = fullPathWithSpaces.substring(fullPathWithSpaces.lastIndexOf('/') + 1);
  File tempDir = new File(System.getProperty("java.io.tmpdir") + "/" + Long.toString(System.nanoTime()));
  tempDir.mkdir();
  file = new File(tempDir.getAbsolutePath() + "/" + fileName);
  InputStream is = httpUrlConnection.getInputStream();
  Files.copy(is, file.toPath());
 }

 httpUrlConnection.disconnect();

 return file;
}

As simple as that! The method takes a full path to a file, and returns a java File object, saved in the system temp folder. If the file is not found, or there’s another error, the method simply returns a null File object. Note that the method will handle any URL-sensitive characters by percent-encoding them (https://en.wikipedia.org/wiki/Percent-encoding), so you won’t need to worry about spaces or other characters in the file name.

Another useful method is one that returns the listing of a file or folder on your Egnyte file server account:

public JSONObject listFileOrFolder(String fullFileOrFolderPathWithSpaces) throws Exception {
 JSONObject listing;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs/" + fullFileOrFolderPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestMethod("GET");
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

 httpUrlConnection.connect();

 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED) {
  Scanner scanner = new Scanner(httpUrlConnection.getInputStream());
  scanner.useDelimiter("\\A");
  String returnJsonString = (scanner.hasNext() ? scanner.next() : "{}");
  scanner.close();
  listing = new JSONObject(returnJsonString);
 } else
  listing = new JSONObject();

 httpUrlConnection.disconnect();

 return listing;
}

This method will also take a full path to a folder or file (can include URL-sensitive characters, such as spaces) and will return either a null JSONObject if there was an error or if the file or folder was not found, or it will return a JSONObject of the listing returned by Egnyte.

The upload file method is a bit more complex, and took me a while to get right. Here it is:

public boolean uploadFile(String fullPathWithSpaces, File file) throws Exception {
 boolean returnValue = false;

 URL url = UriBuilder.fromPath(Constants.EGNYTEAPI_BASE_URL + "fs-content/" + fullPathWithSpaces).build().toURL();
 URLConnection urlConnection = url.openConnection();
 HttpURLConnection httpUrlConnection = (HttpURLConnection) urlConnection;
 httpUrlConnection.setRequestMethod("POST");
 httpUrlConnection.setDoOutput(true);
 httpUrlConnection.setRequestProperty("Authorization", "Bearer " + Constants.EGNYTEAPI_AUTH_TOKEN);

 String crlf = "\r\n", twoHyphens = "--", boundary = UUID.randomUUID().toString();

 httpUrlConnection.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);

 DataOutputStream request = new DataOutputStream(httpUrlConnection.getOutputStream());

 request.writeBytes(twoHyphens + boundary + crlf);
 request.writeBytes("Content-Disposition: form-data; name=\"file\";filename=\"" + file.getName() + "\"" + crlf);
 request.writeBytes(crlf);

 InputStream fileInputStream = new FileInputStream(file);

 int bytesRead, bytesAvailable, bufferSize;
 byte[] buffer;
 int maxBufferSize = 1 * 1024 * 1024;

 bytesAvailable = fileInputStream.available();
 bufferSize = Math.min(bytesAvailable, maxBufferSize);
 buffer = new byte[bufferSize];
 bytesRead = fileInputStream.read(buffer, 0, bufferSize);
 while (bytesRead > 0) {
  request.write(buffer, 0, bufferSize);
  bytesAvailable = fileInputStream.available();
  bufferSize = Math.min(bytesAvailable, maxBufferSize);
  bytesRead = fileInputStream.read(buffer, 0, bufferSize);
 }
 fileInputStream.close();

 request.writeBytes(crlf);
 request.writeBytes(twoHyphens + boundary + twoHyphens + crlf);

 request.flush();
 request.close();

 if (httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_OK || httpUrlConnection.getResponseCode() == HttpURLConnection.HTTP_CREATED)
  returnValue = true;

 httpUrlConnection.disconnect();

 return returnValue;
}

The uploadFile() method takes a java File object, and a full path for where you want the file uploaded to on your Egnyte account. It then reads the file, and uploads it using HTTP multipart data.

Essentially using the methods above you have the basics down: listing a file or folder, uploading a file, and downloading a file. The same code from these can be used to do other actions too such as making a directory, or anything else that the Egnyte RESTful API allows.

Enjoy!