Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: hdfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The INodeAttributeProvider and AccessControlEnforcer features degrade performance and generate excessive garbage even when neither is used. Main issues:

      1. A byte[][] of components is unnecessarily created. Each path component lookup converts a subrange of the byte[][] to a new String[] - then not used by default attribute provider.
      2. Subaccess checks are insanely expensive. The full path of every subdir is created by walking up the inode tree, creating a INode[], building a string by converting each inode's byte[] name to a string, etc. Which will only be used if there's an exception.

      The expensive of #1 should only be incurred when using the provider/enforcer feature. For #2, paths should be created on-demand for exceptions.

      1. HDFS-10673.1.patch
        21 kB
        Daryn Sharp
      2. HDFS-10673.2.patch
        20 kB
        Daryn Sharp
      3. HDFS-10673.patch
        18 kB
        Daryn Sharp
      4. HDFS-10673-branch-2.7.00.patch
        21 kB
        Zhe Zhang

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 32s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 48s trunk passed
          +1 javadoc 0m 57s trunk passed
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 114 unchanged - 14 fixed = 115 total (was 128)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 56s the patch passed
          +1 javadoc 0m 53s the patch passed
          -1 unit 60m 47s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          81m 38s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestReconstructStripedBlocks



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819448/HDFS-10673.patch
          JIRA Issue HDFS-10673
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e234f5ee63a3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ecff7d0
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16141/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16141/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16141/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16141/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 32s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 114 unchanged - 14 fixed = 115 total (was 128) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 60m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 81m 38s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestReconstructStripedBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819448/HDFS-10673.patch JIRA Issue HDFS-10673 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e234f5ee63a3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ecff7d0 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16141/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16141/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16141/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16141/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          Test passes for me. Build artifacts are missing, not sure how/why it failed.

          Show
          daryn Daryn Sharp added a comment - Test passes for me. Build artifacts are missing, not sure how/why it failed.
          Hide
          daryn Daryn Sharp added a comment -

          Zhe Zhang, do you have time to review? This removes many unnecessary allocations of objects that are unused in the common case.

          Show
          daryn Daryn Sharp added a comment - Zhe Zhang , do you have time to review? This removes many unnecessary allocations of objects that are unused in the common case.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the work, Daryn. The patch looks good to me. Only one comment: in the following code, when checking subaccess, the inodeAttr is no longer retrieved from the attribute provider, but directly got from the default d.getSnapshotINode(snapshotId). In case we have an external attribute provider (which may still use FSPermissonChecker as its enforcer), this changes the semantic.

          -  private void checkSubAccess(byte[][] pathByNameArr, int pathIdx, INode inode,
          +  private void checkSubAccess(INode inode,
                 int snapshotId, FsAction access, boolean ignoreEmptyDir)
                 throws AccessControlException {
               if (inode == null || !inode.isDirectory()) {
          @@ -299,8 +291,11 @@ private void checkSubAccess(byte[][] pathByNameArr, int pathIdx, INode inode,
                 ReadOnlyList<INode> cList = d.getChildrenList(snapshotId);
                 if (!(cList.isEmpty() && ignoreEmptyDir)) {
                   //TODO have to figure this out with inodeattribute provider
          -        check(getINodeAttrs(pathByNameArr, pathIdx, d, snapshotId),
          -            inode.getFullPathName(), access);
          +        INodeAttributes inodeAttr = d.getSnapshotINode(snapshotId);
          +        if (!hasPermission(inodeAttr, access)) {
          +          throw new AccessControlException(
          +              toAccessControlString(inodeAttr, d.getFullPathName(), access));
          +        }
                 }
          

          +1 after addressing the comment.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the work, Daryn. The patch looks good to me. Only one comment: in the following code, when checking subaccess, the inodeAttr is no longer retrieved from the attribute provider, but directly got from the default d.getSnapshotINode(snapshotId) . In case we have an external attribute provider (which may still use FSPermissonChecker as its enforcer), this changes the semantic. - private void checkSubAccess( byte [][] pathByNameArr, int pathIdx, INode inode, + private void checkSubAccess(INode inode, int snapshotId, FsAction access, boolean ignoreEmptyDir) throws AccessControlException { if (inode == null || !inode.isDirectory()) { @@ -299,8 +291,11 @@ private void checkSubAccess( byte [][] pathByNameArr, int pathIdx, INode inode, ReadOnlyList<INode> cList = d.getChildrenList(snapshotId); if (!(cList.isEmpty() && ignoreEmptyDir)) { //TODO have to figure this out with inodeattribute provider - check(getINodeAttrs(pathByNameArr, pathIdx, d, snapshotId), - inode.getFullPathName(), access); + INodeAttributes inodeAttr = d.getSnapshotINode(snapshotId); + if (!hasPermission(inodeAttr, access)) { + throw new AccessControlException( + toAccessControlString(inodeAttr, d.getFullPathName(), access)); + } } +1 after addressing the comment.
          Hide
          daryn Daryn Sharp added a comment -

          Ok, thanks. Do you want it to work correctly or be as broken as it is today? Currently the provider is called with the path elements for a directory, and that directory's inode attrs. Then the provider is called with the same original path elements for every descendent of said directory. Ex. do a subaccess check on "/a/b". Provider is called with ["","a","b"] and b's inode attrs. Provider is called with ["","a","b"] even when passing inode attrs for "/a/b/c/d".

          Show
          daryn Daryn Sharp added a comment - Ok, thanks. Do you want it to work correctly or be as broken as it is today? Currently the provider is called with the path elements for a directory, and that directory's inode attrs. Then the provider is called with the same original path elements for every descendent of said directory. Ex. do a subaccess check on "/a/b". Provider is called with ["","a","b"] and b's inode attrs. Provider is called with ["","a","b"] even when passing inode attrs for "/a/b/c/d".
          Hide
          jingzhao Jing Zhao added a comment -

          So what's your patch's behavior? When doing a subaccess check on "/a/b", your patch uses provider to get attrs for ["/","a","b"], and then fall back to use default way (getSnapshotINode) to get attributes for c and d. And you think this is a correct way?

          Show
          jingzhao Jing Zhao added a comment - So what's your patch's behavior? When doing a subaccess check on "/a/b", your patch uses provider to get attrs for ["/","a","b"] , and then fall back to use default way ( getSnapshotINode ) to get attributes for c and d. And you think this is a correct way?
          Hide
          daryn Daryn Sharp added a comment -

          The current subcheck access behavior (before my patch) is passing ["","a","b"] with the inode for "/a/b/c/d". Current patch doesn't call the provider. I'll post a patch that passes ["","a","b","c","d"] with the inode for "/a/b/c/d". That addresses the existing "TODO" in the code.

          Show
          daryn Daryn Sharp added a comment - The current subcheck access behavior (before my patch) is passing ["","a","b"] with the inode for "/a/b/c/d". Current patch doesn't call the provider. I'll post a patch that passes ["","a","b","c","d"] with the inode for "/a/b/c/d". That addresses the existing "TODO" in the code.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 20s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 169 unchanged - 14 fixed = 171 total (was 183)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 51s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 79m 45s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          99m 54s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820969/HDFS-10673.1.patch
          JIRA Issue HDFS-10673
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8ab951bf65d9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 204a205
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16254/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16254/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16254/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16254/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 20s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 169 unchanged - 14 fixed = 171 total (was 183) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 79m 45s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 99m 54s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820969/HDFS-10673.1.patch JIRA Issue HDFS-10673 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8ab951bf65d9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 204a205 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16254/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16254/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16254/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16254/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          Jing Zhao, please let me know if latest patch is ok, or if I should revert checking subdir access back to calling the inode attr provider with just the components of the original subdir.

          Show
          daryn Daryn Sharp added a comment - Jing Zhao , please let me know if latest patch is ok, or if I should revert checking subdir access back to calling the inode attr provider with just the components of the original subdir.
          Hide
          daryn Daryn Sharp added a comment -
          Show
          daryn Daryn Sharp added a comment - Jing Zhao ?
          Hide
          kihwal Kihwal Lee added a comment -

          Let's preserve the old behavior for now and discuss potential improvement in a separate jira.

          Show
          kihwal Kihwal Lee added a comment - Let's preserve the old behavior for now and discuss potential improvement in a separate jira.
          Hide
          daryn Daryn Sharp added a comment -

          To expedite this jira, I reverted the subaccess check to do the prior questionably broken attributes provider behavior. This should meet the criteria for Jing's prior +1.

          Show
          daryn Daryn Sharp added a comment - To expedite this jira, I reverted the subaccess check to do the prior questionably broken attributes provider behavior. This should meet the criteria for Jing's prior +1.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 34s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 51s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 59s the patch passed
          +1 compile 1m 0s the patch passed
          +1 javac 1m 0s the patch passed
          -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 165 unchanged - 14 fixed = 166 total (was 179)
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 59s the patch passed
          +1 javadoc 0m 59s the patch passed
          -1 unit 56m 51s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          78m 13s



          Reason Tests
          Failed junit tests hadoop.tracing.TestTracing



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822077/HDFS-10673.2.patch
          JIRA Issue HDFS-10673
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 55f3a182e40f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6ae3919
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16320/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16320/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16320/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 34s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 59s the patch passed +1 compile 1m 0s the patch passed +1 javac 1m 0s the patch passed -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 165 unchanged - 14 fixed = 166 total (was 179) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 59s the patch passed -1 unit 56m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 78m 13s Reason Tests Failed junit tests hadoop.tracing.TestTracing Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822077/HDFS-10673.2.patch JIRA Issue HDFS-10673 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 55f3a182e40f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6ae3919 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16320/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16320/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16320/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Yes, I agree with Kihwal Lee, we should make sure we really understand the old behavior and potential improvement and we can do that in a separate jira.

          +1 on the latest patch.

          Show
          jingzhao Jing Zhao added a comment - Yes, I agree with Kihwal Lee , we should make sure we really understand the old behavior and potential improvement and we can do that in a separate jira. +1 on the latest patch.
          Hide
          kihwal Kihwal Lee added a comment -

          Committed this to trunk and branch-2. Thanks for the reviews and feedback, Jing Zhao.

          Show
          kihwal Kihwal Lee added a comment - Committed this to trunk and branch-2. Thanks for the reviews and feedback, Jing Zhao .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10219 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10219/)
          HDFS-10673. Optimize FSPermissionChecker's internal path usage. (kihwal: rev 438a9f047eb6af2a4b916a4f6ef6f68adeab8068)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSPermission.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributeProvider.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10219 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10219/ ) HDFS-10673 . Optimize FSPermissionChecker's internal path usage. (kihwal: rev 438a9f047eb6af2a4b916a4f6ef6f68adeab8068) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSPermission.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributeProvider.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          Hide
          zhz Zhe Zhang added a comment -

          Sorry for responding late. I was on vacation.

          Thanks Daryn and Jing. I'll catch up with the subtasks and try to backport to 2.7/2.6.

          Show
          zhz Zhe Zhang added a comment - Sorry for responding late. I was on vacation. Thanks Daryn and Jing. I'll catch up with the subtasks and try to backport to 2.7/2.6.
          Hide
          zhz Zhe Zhang added a comment -

          Just backported this one to branch-2.8. Working on branch-2.7 backport.

          Show
          zhz Zhe Zhang added a comment - Just backported this one to branch-2.8. Working on branch-2.7 backport.
          Hide
          zhz Zhe Zhang added a comment -

          Sorry to reopen the JIRA. I want to test the branch-2.7 patch on Jenkins.

          Show
          zhz Zhe Zhang added a comment - Sorry to reopen the JIRA. I want to test the branch-2.7 patch on Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 1s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 14s branch-2.7 passed
          +1 compile 1m 11s branch-2.7 passed with JDK v1.8.0_101
          +1 compile 1m 9s branch-2.7 passed with JDK v1.7.0_111
          +1 checkstyle 0m 29s branch-2.7 passed
          +1 mvnsite 1m 10s branch-2.7 passed
          +1 mvneclipse 0m 16s branch-2.7 passed
          +1 findbugs 3m 20s branch-2.7 passed
          +1 javadoc 1m 4s branch-2.7 passed with JDK v1.8.0_101
          +1 javadoc 1m 58s branch-2.7 passed with JDK v1.7.0_111
          +1 mvninstall 0m 53s the patch passed
          +1 compile 0m 56s the patch passed with JDK v1.8.0_101
          +1 javac 0m 56s the patch passed
          +1 compile 0m 57s the patch passed with JDK v1.7.0_111
          +1 javac 0m 57s the patch passed
          +1 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 142 unchanged - 7 fixed = 142 total (was 149)
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 2202 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 54s The patch 83 line(s) with tabs.
          +1 findbugs 3m 4s the patch passed
          +1 javadoc 1m 0s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 45s the patch passed with JDK v1.7.0_111
          -1 unit 48m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          -1 asflicense 0m 27s The patch generated 3 ASF License warnings.
          143m 56s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          JDK v1.7.0_111 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-10673
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828098/HDFS-10673-branch-2.7.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5400e682c37d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / d7da703
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16715/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16715/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16715/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16715/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16715/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16715/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 14s branch-2.7 passed +1 compile 1m 11s branch-2.7 passed with JDK v1.8.0_101 +1 compile 1m 9s branch-2.7 passed with JDK v1.7.0_111 +1 checkstyle 0m 29s branch-2.7 passed +1 mvnsite 1m 10s branch-2.7 passed +1 mvneclipse 0m 16s branch-2.7 passed +1 findbugs 3m 20s branch-2.7 passed +1 javadoc 1m 4s branch-2.7 passed with JDK v1.8.0_101 +1 javadoc 1m 58s branch-2.7 passed with JDK v1.7.0_111 +1 mvninstall 0m 53s the patch passed +1 compile 0m 56s the patch passed with JDK v1.8.0_101 +1 javac 0m 56s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_111 +1 javac 0m 57s the patch passed +1 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 142 unchanged - 7 fixed = 142 total (was 149) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2202 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 54s The patch 83 line(s) with tabs. +1 findbugs 3m 4s the patch passed +1 javadoc 1m 0s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_111 -1 unit 48m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_111. -1 asflicense 0m 27s The patch generated 3 ASF License warnings. 143m 56s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure JDK v1.7.0_111 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-10673 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828098/HDFS-10673-branch-2.7.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5400e682c37d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / d7da703 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16715/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16715/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16715/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16715/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16715/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16715/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 9s branch-2.7 passed
          +1 compile 1m 7s branch-2.7 passed with JDK v1.8.0_101
          +1 compile 1m 6s branch-2.7 passed with JDK v1.7.0_111
          +1 checkstyle 0m 25s branch-2.7 passed
          +1 mvnsite 1m 0s branch-2.7 passed
          +1 mvneclipse 0m 15s branch-2.7 passed
          +1 findbugs 2m 54s branch-2.7 passed
          +1 javadoc 1m 2s branch-2.7 passed with JDK v1.8.0_101
          +1 javadoc 1m 48s branch-2.7 passed with JDK v1.7.0_111
          +1 mvninstall 0m 58s the patch passed
          +1 compile 1m 4s the patch passed with JDK v1.8.0_101
          +1 javac 1m 4s the patch passed
          +1 compile 1m 4s the patch passed with JDK v1.7.0_111
          +1 javac 1m 4s the patch passed
          +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 142 unchanged - 7 fixed = 142 total (was 149)
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 2202 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 54s The patch 83 line(s) with tabs.
          +1 findbugs 3m 8s the patch passed
          +1 javadoc 0m 55s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 41s the patch passed with JDK v1.7.0_111
          -1 unit 56m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          -1 asflicense 0m 20s The patch generated 3 ASF License warnings.
          163m 2s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.TestFileTruncate
          JDK v1.7.0_111 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-10673
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828098/HDFS-10673-branch-2.7.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 86849dc40a66 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / d7da703
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16714/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16714/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16714/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16714/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16714/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 9s branch-2.7 passed +1 compile 1m 7s branch-2.7 passed with JDK v1.8.0_101 +1 compile 1m 6s branch-2.7 passed with JDK v1.7.0_111 +1 checkstyle 0m 25s branch-2.7 passed +1 mvnsite 1m 0s branch-2.7 passed +1 mvneclipse 0m 15s branch-2.7 passed +1 findbugs 2m 54s branch-2.7 passed +1 javadoc 1m 2s branch-2.7 passed with JDK v1.8.0_101 +1 javadoc 1m 48s branch-2.7 passed with JDK v1.7.0_111 +1 mvninstall 0m 58s the patch passed +1 compile 1m 4s the patch passed with JDK v1.8.0_101 +1 javac 1m 4s the patch passed +1 compile 1m 4s the patch passed with JDK v1.7.0_111 +1 javac 1m 4s the patch passed +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 142 unchanged - 7 fixed = 142 total (was 149) +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2202 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 54s The patch 83 line(s) with tabs. +1 findbugs 3m 8s the patch passed +1 javadoc 0m 55s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_111 -1 unit 56m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_111. -1 asflicense 0m 20s The patch generated 3 ASF License warnings. 163m 2s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.TestFileTruncate JDK v1.7.0_111 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-10673 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828098/HDFS-10673-branch-2.7.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 86849dc40a66 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / d7da703 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16714/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16714/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16714/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16714/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16714/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16714/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Verified all reported failures pass locally. Pushed to branch-2.7.

          Show
          zhz Zhe Zhang added a comment - Verified all reported failures pass locally. Pushed to branch-2.7.

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development