Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12300

Audit-log delegation token related operations

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 3.0.0-beta1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      <!-- markdown -->

      NameNode now audit-logs `getDelegationToken`, `renewDelegationToken`, `cancelDelegationToken`.
      Show
      <!-- markdown --> NameNode now audit-logs `getDelegationToken`, `renewDelegationToken`, `cancelDelegationToken`.

      Description

      When inspecting the code, I found that the following methods in FSNamesystem are not audit logged:

      • getDelegationToken
      • renewDelegationToken
      • cancelDelegationToken

      The audit log itself does have a logTokenTrackingId field to additionally log some details when a token is used for authentication.

      After emailing the community, we should add that.

      1. HDFS-12300.01.patch
        10 kB
        Xiao Chen
      2. HDFS-12300.02.patch
        10 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Finally find some cycles to get this rolling.

          Patch 1 attached, example audit logs could be seen by running the added unit test.

          Tried to keep existing behavior in FSN - namely do not use Token#decodeIdentifier due to (I guess) the overhead of reflection. Extracted the existing code to DFSUtil, and fixed a surprising stream leakage which I'm not sure why hasn't caused any issues thus far.

          Show
          xiaochen Xiao Chen added a comment - Finally find some cycles to get this rolling. Patch 1 attached, example audit logs could be seen by running the added unit test. Tried to keep existing behavior in FSN - namely do not use Token#decodeIdentifier due to (I guess) the overhead of reflection. Extracted the existing code to DFSUtil , and fixed a surprising stream leakage which I'm not sure why hasn't caused any issues thus far.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 13m 45s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 39s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 250 unchanged - 1 fixed = 251 total (was 251)
          +1 mvnsite 0m 51s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 0m 38s the patch passed
                Other Tests
          -1 unit 85m 33s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          111m 11s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestLeaseRecoveryStriped
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12300
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883991/HDFS-12300.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1649c7fb5a43 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ae7abad
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20887/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20887/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20887/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20887/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 45s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 0m 54s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 39s trunk passed       Patch Compile Tests +1 mvninstall 0m 49s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 250 unchanged - 1 fixed = 251 total (was 251) +1 mvnsite 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 38s the patch passed       Other Tests -1 unit 85m 33s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 111m 11s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12300 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883991/HDFS-12300.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1649c7fb5a43 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ae7abad Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20887/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20887/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20887/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20887/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Any watchers have cycle to review? (Will address checkstyle with any comments together in the next rev.)
          Thank you.

          Show
          xiaochen Xiao Chen added a comment - Any watchers have cycle to review? (Will address checkstyle with any comments together in the next rev.) Thank you.
          Hide
          raviprak Ravi Prakash added a comment -

          Hi Xiao! Thanks a lot for your effort on this.

          Is the performance overhead of reflection appreciable over here? Should different modules really be decoding tokens? I'm not sure I understand why we don't audit log when some exceptions are thrown and not others. But that is not related to this JIRA.

          Otherwise patch looks good to me.

          Show
          raviprak Ravi Prakash added a comment - Hi Xiao! Thanks a lot for your effort on this. Is the performance overhead of reflection appreciable over here? Should different modules really be decoding tokens? I'm not sure I understand why we don't audit log when some exceptions are thrown and not others. But that is not related to this JIRA. Otherwise patch looks good to me.
          Hide
          xiaochen Xiao Chen added a comment -

          Hey Ravi,

          Thanks a lot for reviewing!

          I was only deducting about the reasoning why FSN writes is own code instead of calling the general helper method, so I don't have numbers. Considering reflections are usually resource-heavy, and we tend to optimize NN within the namespace lock, it seems plausible.
          The overhead I was referring to are specific to these lines. IMHO it's up to each module to decide whether this is costly enough for optimization, likely from the result of stress tests.

          why we don't audit log when some exceptions are thrown and not others

          I think HDFS-10776 (and its first comment) is the best answer available - we only log AccessControlExceptions, and don't care about others.

          Patch 2 to fix the checkstyle.

          Show
          xiaochen Xiao Chen added a comment - Hey Ravi, Thanks a lot for reviewing! I was only deducting about the reasoning why FSN writes is own code instead of calling the general helper method, so I don't have numbers. Considering reflections are usually resource-heavy, and we tend to optimize NN within the namespace lock, it seems plausible. The overhead I was referring to are specific to these lines . IMHO it's up to each module to decide whether this is costly enough for optimization, likely from the result of stress tests. why we don't audit log when some exceptions are thrown and not others I think HDFS-10776 (and its first comment) is the best answer available - we only log AccessControlExceptions, and don't care about others. Patch 2 to fix the checkstyle.
          Hide
          raviprak Ravi Prakash added a comment -

          Sounds good to me. Patch looks good to me. +1.

          Show
          raviprak Ravi Prakash added a comment - Sounds good to me. Patch looks good to me. +1.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 16m 6s trunk passed
          +1 compile 1m 3s trunk passed
          +1 checkstyle 0m 50s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 findbugs 2m 13s trunk passed
          +1 javadoc 0m 42s trunk passed
                Patch Compile Tests
          -1 mvninstall 0m 57s hadoop-hdfs in the patch failed.
          +1 compile 0m 52s the patch passed
          +1 javac 0m 52s the patch passed
          +1 checkstyle 0m 40s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 250 unchanged - 1 fixed = 250 total (was 251)
          +1 mvnsite 0m 57s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 1s the patch passed
          +1 javadoc 0m 41s the patch passed
                Other Tests
          -1 unit 95m 9s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          125m 16s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
            hadoop.hdfs.TestFileAppendRestart
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
            hadoop.hdfs.TestLeaseRecoveryStriped
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HDFS-12300
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884824/HDFS-12300.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3eae518577a8 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 91cc070
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/20949/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20949/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20949/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 6s trunk passed +1 compile 1m 3s trunk passed +1 checkstyle 0m 50s trunk passed +1 mvnsite 1m 5s trunk passed +1 findbugs 2m 13s trunk passed +1 javadoc 0m 42s trunk passed       Patch Compile Tests -1 mvninstall 0m 57s hadoop-hdfs in the patch failed. +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed +1 checkstyle 0m 40s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 250 unchanged - 1 fixed = 250 total (was 251) +1 mvnsite 0m 57s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 1s the patch passed +1 javadoc 0m 41s the patch passed       Other Tests -1 unit 95m 9s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 125m 16s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12300 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884824/HDFS-12300.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3eae518577a8 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 91cc070 Default Java 1.8.0_144 findbugs v3.1.0-RC1 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/20949/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20949/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20949/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment - - edited

          Thanks again Ravi!

          Last pre-commit mvn install seems to have failed with a race condition, retriggerred.

          Show
          xiaochen Xiao Chen added a comment - - edited Thanks again Ravi! Last pre-commit mvn install seems to have failed with a race condition, retriggerred.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 13m 49s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 42s trunk passed
                Patch Compile Tests
          +1 mvninstall 1m 4s the patch passed
          +1 compile 0m 57s the patch passed
          +1 javac 0m 57s the patch passed
          +1 checkstyle 0m 41s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 250 unchanged - 1 fixed = 250 total (was 251)
          +1 mvnsite 1m 4s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 2s the patch passed
          +1 javadoc 0m 44s the patch passed
                Other Tests
          -1 unit 93m 54s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          121m 12s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.TestLeaseRecoveryStriped
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          Timed out junit tests org.apache.hadoop.hdfs.TestReplication
            org.apache.hadoop.hdfs.TestWriteReadStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HDFS-12300
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884824/HDFS-12300.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 09f0dbcae0c4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27359b7
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20962/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20962/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20962/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 49s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 0m 53s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 42s trunk passed       Patch Compile Tests +1 mvninstall 1m 4s the patch passed +1 compile 0m 57s the patch passed +1 javac 0m 57s the patch passed +1 checkstyle 0m 41s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 250 unchanged - 1 fixed = 250 total (was 251) +1 mvnsite 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 0m 44s the patch passed       Other Tests -1 unit 93m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 121m 12s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 Timed out junit tests org.apache.hadoop.hdfs.TestReplication   org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12300 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884824/HDFS-12300.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 09f0dbcae0c4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27359b7 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20962/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20962/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20962/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Committed this to trunk. Thanks Ravi for the review.

          What do people think about branch-2? The cherry pick isn't clean, but I can put up a branch-2 patch if that's considered valuable.

          Show
          xiaochen Xiao Chen added a comment - Committed this to trunk. Thanks Ravi for the review. What do people think about branch-2? The cherry pick isn't clean, but I can put up a branch-2 patch if that's considered valuable.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12295 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12295/)
          HDFS-12300. Audit-log delegation token related operations. (xiao: rev 1b3b9938cf663c71d2e5d9032fdfb1460bae0d3f)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLoggerWithCommands.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12295 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12295/ ) HDFS-12300 . Audit-log delegation token related operations. (xiao: rev 1b3b9938cf663c71d2e5d9032fdfb1460bae0d3f) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLoggerWithCommands.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          raviprak Ravi Prakash added a comment -

          Thanks for the work Xiao!

          Up to you to push into branch-2. I'm supportive of it.

          Show
          raviprak Ravi Prakash added a comment - Thanks for the work Xiao! Up to you to push into branch-2. I'm supportive of it.
          Hide
          xiaochen Xiao Chen added a comment -

          I tried to backport to branch-2, which isn't too hard. But DelegationTokenIdentifier#toStringStable was added by HDFS-9732, which is very messy to get backported, according to Yongjun's comment. (Also roughly tried myself, HDFS-5570 is a source of headache, where DelegationTokenFetcher was largely modified)

          So for this reason I will resolve this as branch-3 only, and link to HDFS-9732 as a dependent.
          Thanks again Ravi!

          Show
          xiaochen Xiao Chen added a comment - I tried to backport to branch-2, which isn't too hard. But DelegationTokenIdentifier#toStringStable was added by HDFS-9732 , which is very messy to get backported, according to Yongjun's comment . (Also roughly tried myself, HDFS-5570 is a source of headache, where DelegationTokenFetcher was largely modified) So for this reason I will resolve this as branch-3 only, and link to HDFS-9732 as a dependent. Thanks again Ravi!

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development