Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15414

Job submit not work well on HDFS Federation with Transparent Encryption feature

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      When submit sample MapReduce job WordCount which read/write path under encryption zone on HDFS Federation in security mode to YARN, task throws exception as below:

      18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, Status : FAILED
      Error: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
          at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489)
          at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
          at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
          at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468)
          at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538)
          at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306)
          at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
          at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
          at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300)
          at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
          at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258)
          at org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424)
          at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793)
          at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
          at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
          at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
          at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
      Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
          at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332)
          at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
          at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
          at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
          at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
          at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483)
          at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
          at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478)
          ... 21 more
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
          at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
          at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
          at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
          at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
          at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
          at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
          at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:311)
          at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:287)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:287)
          ... 31 more
      

      The main reason is before submit job in security mode, we need to collect delegation tokens include delegation token for NameNode and KMS firstly. IF on HDFS Federation, all delegation tokens for NameNode can collection correctly BUT delegation token for KMS not collect reference FileSystem#addDelegationTokens -> FileSystem#collectDelegationTokens, so when launch task it fails because KMS token not pass to through ResourceManager as exception shows GSSException: No valid credentials provided.

        Attachments

        1. HADOOP-15414-trunk.001.patch
          0.8 kB
          Xiaoqiao He
        2. HADOOP-15414-trunk.002.patch
          0.8 kB
          Xiaoqiao He

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hexiaoqiao Xiaoqiao He
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: