Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10760

DataXceiver#run() should not log InvalidToken exception as an error

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Log InvalidTokenException at trace level in DataXceiver#run().

      Description

      DataXceiver#run() just log InvalidToken exception as an error.
      When client has an expired token and just refetch a new token, the DN log will has an error like below:

      2016-08-11 02:41:09,817 ERROR datanode.DataNode (DataXceiver.java:run(269)) - XXXXXXX:50010:DataXceiver error processing READ_BLOCK operation  src: /10.17.1.5:38844 dst: /10.17.1.5:50010
      org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1470850746803, keyId=-2093956963, userId=hbase, blockPoolId=BP-641703426-10.17.1.2-1468517918886, blockId=1077120201, access modes=[READ]) is expired.
              at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
              at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:301)
              at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:97)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.java:1236)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:481)
              at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
              at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:242)
              at java.lang.Thread.run(Thread.java:745)
      

      This is not a server error and the DataXceiver#checkAccess() has already loged the InvalidToken as a warning.
      A simple fix by catching the InvalidToken exception in DataXceiver#run(), only keeping the warning logged by DataXceiver#checkAccess() in the DN log.

      1. HADOOP-13492.patch
        0.9 kB
        Pan Yuxuan
      2. HDFS-10760-1.patch
        0.9 kB
        Pan Yuxuan

        Issue Links

          Activity

          Hide
          panyuxuan Pan Yuxuan added a comment -

          Attachment is a patch for this.

          Show
          panyuxuan Pan Yuxuan added a comment - Attachment is a patch for this.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 27s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 53s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 40s the patch passed
          +1 javac 0m 40s the patch passed
          -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 58 unchanged - 0 fixed = 59 total (was 58)
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 52s the patch passed
          +1 unit 58m 42s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          90m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823430/HADOOP-13492.patch
          JIRA Issue HDFS-10760
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e56684d45254 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 23c6e3c
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16415/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16415/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16415/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 27s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 40s the patch passed +1 javac 0m 40s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 58 unchanged - 0 fixed = 59 total (was 58) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 58m 42s hadoop-hdfs in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 90m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823430/HADOOP-13492.patch JIRA Issue HDFS-10760 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e56684d45254 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23c6e3c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16415/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16415/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16415/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          panyuxuan Pan Yuxuan added a comment -

          Uploaded a new patch with checkstyle fixes.

          Show
          panyuxuan Pan Yuxuan added a comment - Uploaded a new patch with checkstyle fixes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 2s trunk passed
          +1 compile 0m 50s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 59s the patch passed
          -1 unit 77m 28s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          98m 14s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.TestFileAppendRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823640/HDFS-10760-1.patch
          JIRA Issue HDFS-10760
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d806d63e0609 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d677b68
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16418/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16418/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 2s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 59s the patch passed -1 unit 77m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 98m 14s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.TestFileAppendRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823640/HDFS-10760-1.patch JIRA Issue HDFS-10760 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d806d63e0609 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d677b68 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16418/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16418/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16418/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          panyuxuan Pan Yuxuan added a comment -

          Test Failure is not related to this patch

          Show
          panyuxuan Pan Yuxuan added a comment - Test Failure is not related to this patch
          Hide
          panyuxuan Pan Yuxuan added a comment -

          Brahma Reddy Battula Akira Ajisaka Rushabh S Shah
          Could anyone please help to review this simple patch? Thanks!

          Show
          panyuxuan Pan Yuxuan added a comment - Brahma Reddy Battula Akira Ajisaka Rushabh S Shah Could anyone please help to review this simple patch? Thanks!
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Pan Yuxuan thanks for reporting this issue.. Patch looks good to me....Akira Ajisaka and Rushabh S Shah how do you think.?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Pan Yuxuan thanks for reporting this issue.. Patch looks good to me.... Akira Ajisaka and Rushabh S Shah how do you think.?
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Pan Yuxuan: Thanks for reporting the issue.
          Patch looks good to me as well.
          +1 non-binding.

          Show
          shahrs87 Rushabh S Shah added a comment - Pan Yuxuan : Thanks for reporting the issue. Patch looks good to me as well. +1 non-binding.
          Hide
          jojochuang Wei-Chiu Chuang added a comment - - edited

          Pan Yuxuan thanks for reporting the issue and creating the patch.
          In fact, DataXceiver#checkAccess() is used by multiple methods and I think it would be beneficial if you can also improve other callers as well.

          For example, I can see DataXceiver#writeBlock calls checkAccess, and if an InvalidTokenException is thrown, it also logs a message at error level.

          try {
          ...
          } catch (IOException ioe) {
          if (isClient) {
                      LOG.error(datanode + ":Exception transfering block " +
                                block + " to mirror " + mirrorNode + ": " + e);
                      throw e;
                    }
          }
          
          Show
          jojochuang Wei-Chiu Chuang added a comment - - edited Pan Yuxuan thanks for reporting the issue and creating the patch. In fact, DataXceiver#checkAccess() is used by multiple methods and I think it would be beneficial if you can also improve other callers as well. For example, I can see DataXceiver#writeBlock calls checkAccess, and if an InvalidTokenException is thrown, it also logs a message at error level. try { ... } catch (IOException ioe) { if (isClient) { LOG.error(datanode + ":Exception transfering block " + block + " to mirror " + mirrorNode + ": " + e); throw e; } }
          Hide
          shahrs87 Rushabh S Shah added a comment -

          I also checked for the same thing when I was reviewing.
          But it turns out that checkAccess method is not in the try block, so it will just throw the InvlaidToken Exception all the way back to DataXceiver#run.
          So I think the patch should be good.
          Correct me if I am wrong.

          Show
          shahrs87 Rushabh S Shah added a comment - I also checked for the same thing when I was reviewing. But it turns out that checkAccess method is not in the try block, so it will just throw the InvlaidToken Exception all the way back to DataXceiver#run. So I think the patch should be good. Correct me if I am wrong.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks for correcting me. You're right about that.
          +1.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks for correcting me. You're right about that. +1.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed it to trunk, branch-2 and branch-2.8.
          Thanks Pan Yuxuan for contributing the patch and Brahma Reddy Battula and Rushabh S Shah for comments!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed it to trunk, branch-2 and branch-2.8. Thanks Pan Yuxuan for contributing the patch and Brahma Reddy Battula and Rushabh S Shah for comments!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10375 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10375/)
          HDFS-10760. DataXceiver#run() should not log InvalidToken exception as (weichiu: rev c4ee6915a14e00342755d7cdcbf2d61518f306aa)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10375 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10375/ ) HDFS-10760 . DataXceiver#run() should not log InvalidToken exception as (weichiu: rev c4ee6915a14e00342755d7cdcbf2d61518f306aa) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java

            People

            • Assignee:
              panyuxuan Pan Yuxuan
              Reporter:
              panyuxuan Pan Yuxuan
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development