Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15609

Retry KMS calls when SSLHandshakeException occurs

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.2.0, 3.0.4, 3.1.2
    • Component/s: common, kms
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      KMS call should retry when javax.net.ssl.SSLHandshakeException occurs and FailoverOnNetworkExceptionRetry policy is used.

      For example in the following stack trace, we can see that the KMS Provider's connection is lost, an SSLHandshakeException is thrown and the operation is not retried:

      W0711 18:19:50.213472  1508 LoadBalancingKMSClientProvider.java:132] KMS provider at [https://example.com:16000/kms/v1/] threw an IOException:
      Java exception follows:
      javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
              at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
              at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
              at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
              at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
              at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
              at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
              at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
              at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)
              at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:512)
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:502)
              at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:791)
              at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:288)
              at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:284)
              at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:124)
              at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:284)
              at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
              at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
              at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
              at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
              at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
              at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
              at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
      Caused by: java.io.EOFException: SSL peer shut down incorrectly
              at sun.security.ssl.InputRecord.read(InputRecord.java:505)
              at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
              ... 22 more
      W0711 18:19:50.239328  1508 LoadBalancingKMSClientProvider.java:149] Aborting since the Request has failed with all KMS providers(depending on hadoop.security.kms.client.failover.max.retries=1 setting and numProviders=1) in the group OR the exception is not recoverable
      

        Attachments

        1. HADOOP-15609.004.patch
          8 kB
          Kitti Nanasi
        2. HADOOP-15609.003.patch
          8 kB
          Kitti Nanasi
        3. HADOOP-15609.002.patch
          7 kB
          Kitti Nanasi
        4. HADOOP-15609.001.patch
          1 kB
          Kitti Nanasi

          Activity

            People

            • Assignee:
              knanasi Kitti Nanasi
              Reporter:
              knanasi Kitti Nanasi
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: