[HADOOP-14563] LoadBalancingKMSClientProvider#warmUpEncryptedKeys swallows IOException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
• Target Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
• Hadoop Flags: Reviewed
• Assignee: Rushabh Shah
• Reporter: Rushabh Shah
• Created: 21/Jun/17 14:17
• Updated: 02/Oct/19 17:14
• Resolved: 07/Jul/17 13:18

Description

TestAclsEndToEnd is failing consistently in HADOOP-14521.
The reason behind it is LoadBalancingKMSClientProvider#warmUpEncryptedKeys swallows IOException while KMSClientProvider#warmUpEncryptedKeys throws all the way back to createEncryptionZone and creation of EZ fails.
Following are the relevant piece of code snippets.

KMSClientProvider.java

  @Override
  public void warmUpEncryptedKeys(String... keyNames)
      throws IOException {
    try {
      encKeyVersionQueue.initializeQueuesForKeys(keyNames);
    } catch (ExecutionException e) {
      throw new IOException(e);
    }
  }

LoadBalancingKMSClientProvider.java

   // This request is sent to all providers in the load-balancing group
  @Override
  public void warmUpEncryptedKeys(String... keyNames) throws IOException {
    for (KMSClientProvider provider : providers) {
      try {
        provider.warmUpEncryptedKeys(keyNames);
      } catch (IOException ioe) {
        LOG.error(
            "Error warming up keys for provider with url"
            + "[" + provider.getKMSUrl() + "]", ioe);
      }
    }
  }

In HADOOP-14521, I intend to always instantiate LoadBalancingKMSClientProvider even if there is only one provider so that the retries can applied at only one place.
We need to decide whether we want to fail in both the case or continue.