Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15684

EC: Call recoverLease on DFSStripedOutputStream close exception

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • dfsclient, ec
    • None
    • Reviewed

    Description

      HDFS-14694 add a feature that call recoverLease operation automatically when DFSOutputSteam close encounters exception. When we wanted to apply this feature to our cluster, we found that it does not support EC files. 

      I think this feature should take effect whether replica files or EC files. This Jira proposes to make it effective when in the case of EC files.

      Attachments

        1. HDFS-15684.001.patch
          2 kB
          Hongbing Wang
        2. HDFS-15684.002.patch
          7 kB
          Hongbing Wang
        3. HDFS-15684.003.patch
          7 kB
          Hongbing Wang

        Issue Links

          Activity

            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 0m 52s   Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s   No case conflicting files found.
            +1 @author 0m 0s   The patch does not contain any @author tags.
            -1 test4tests 0m 0s   The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                  trunk Compile Tests  
            +1 mvninstall 26m 52s   trunk passed
            +1 compile 1m 7s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 compile 0m 55s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 checkstyle 0m 23s   trunk passed
            +1 mvnsite 0m 58s   trunk passed
            +1 shadedclient 19m 5s   branch has no errors when building and testing our client artifacts.
            +1 javadoc 0m 42s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javadoc 0m 35s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            0 spotbugs 2m 52s   Used deprecated FindBugs config; considering switching to SpotBugs.
            +1 findbugs 2m 48s   trunk passed
                  Patch Compile Tests  
            +1 mvninstall 0m 56s   the patch passed
            +1 compile 1m 2s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javac 1m 2s   the patch passed
            +1 compile 0m 48s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 javac 0m 48s   the patch passed
            +1 blanks 0m 0s   The patch has no blanks issues.
            +1 checkstyle 0m 19s   the patch passed
            +1 mvnsite 0m 54s   the patch passed
            +1 shadedclient 18m 26s   patch has no errors when building and testing our client artifacts.
            +1 javadoc 0m 33s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javadoc 0m 30s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 findbugs 2m 37s   the patch passed
                  Other Tests  
            +1 unit 2m 21s   hadoop-hdfs-client in the patch passed.
            +1 asflicense 0m 37s   The patch does not generate ASF License warnings.
            83m 16s  



            Subsystem Report/Notes
            Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/301/artifact/out/Dockerfile
            JIRA Issue HDFS-15684
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13015309/HDFS-15684.001.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux ff8cb1d573fb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / ebe1d1fbf7a7549078e5a468080513db09b6416f
            Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            Test Results https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/301/testReport/
            Max. process+thread count 511 (vs. ulimit of 5500)
            modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
            Console output https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/301/console
            versions git=2.17.1 maven=3.6.0 findbugs=4.0.6
            Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 0m 52s   Docker mode activated.       Prechecks   +1 dupname 0m 0s   No case conflicting files found. +1 @author 0m 0s   The patch does not contain any @author tags. -1 test4tests 0m 0s   The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests   +1 mvninstall 26m 52s   trunk passed +1 compile 1m 7s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 compile 0m 55s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 checkstyle 0m 23s   trunk passed +1 mvnsite 0m 58s   trunk passed +1 shadedclient 19m 5s   branch has no errors when building and testing our client artifacts. +1 javadoc 0m 42s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javadoc 0m 35s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 0 spotbugs 2m 52s   Used deprecated FindBugs config; considering switching to SpotBugs. +1 findbugs 2m 48s   trunk passed       Patch Compile Tests   +1 mvninstall 0m 56s   the patch passed +1 compile 1m 2s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javac 1m 2s   the patch passed +1 compile 0m 48s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 javac 0m 48s   the patch passed +1 blanks 0m 0s   The patch has no blanks issues. +1 checkstyle 0m 19s   the patch passed +1 mvnsite 0m 54s   the patch passed +1 shadedclient 18m 26s   patch has no errors when building and testing our client artifacts. +1 javadoc 0m 33s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javadoc 0m 30s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 findbugs 2m 37s   the patch passed       Other Tests   +1 unit 2m 21s   hadoop-hdfs-client in the patch passed. +1 asflicense 0m 37s   The patch does not generate ASF License warnings. 83m 16s   Subsystem Report/Notes Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/301/artifact/out/Dockerfile JIRA Issue HDFS-15684 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13015309/HDFS-15684.001.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux ff8cb1d573fb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / ebe1d1fbf7a7549078e5a468080513db09b6416f Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 Test Results https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/301/testReport/ Max. process+thread count 511 (vs. ulimit of 5500) modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Console output https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/301/console versions git=2.17.1 maven=3.6.0 findbugs=4.0.6 Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            wanghongbing Hongbing Wang added a comment -

            add Tests in v2 patch.

            wanghongbing Hongbing Wang added a comment - add Tests in v2 patch.
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 37m 6s   Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s   No case conflicting files found.
            +1 @author 0m 0s   The patch does not contain any @author tags.
            +1 test4tests 0m 0s   The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 2m 7s   Maven dependency ordering for branch
            +1 mvninstall 28m 3s   trunk passed
            +1 compile 5m 20s   trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
            +1 compile 5m 2s   trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
            +1 checkstyle 1m 11s   trunk passed
            +1 mvnsite 2m 35s   trunk passed
            +1 shadedclient 20m 33s   branch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 27s   trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
            +1 javadoc 1m 55s   trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
            0 spotbugs 3m 19s   Used deprecated FindBugs config; considering switching to SpotBugs.
            +1 findbugs 5m 49s   trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 23s   Maven dependency ordering for patch
            +1 mvninstall 2m 15s   the patch passed
            +1 compile 5m 3s   the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
            +1 javac 5m 3s   the patch passed
            +1 compile 5m 4s   the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
            +1 javac 5m 4s   the patch passed
            +1 blanks 0m 0s   The patch has no blanks issues.
            -0 checkstyle 0m 52s /results-checkstyle-hadoop-hdfs-project.txt hadoop-hdfs-project: The patch generated 4 new + 33 unchanged - 0 fixed = 37 total (was 33)
            +1 mvnsite 2m 0s   the patch passed
            +1 shadedclient 20m 56s   patch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 25s   the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
            +1 javadoc 1m 49s   the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
            +1 findbugs 5m 53s   the patch passed
                  Other Tests  
            +1 unit 2m 18s   hadoop-hdfs-client in the patch passed.
            -1 unit 131m 40s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
            +1 asflicense 0m 41s   The patch does not generate ASF License warnings.
            286m 50s  



            Reason Tests
            Failed junit tests hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
              hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
              hadoop.hdfs.server.namenode.TestFileTruncate
              hadoop.hdfs.TestCrcCorruption



            Subsystem Report/Notes
            Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/artifact/out/Dockerfile
            JIRA Issue HDFS-15684
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13015360/HDFS-15684.002.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux e9b7ab1ca3dc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / dd85a90da6f3c475915def97ace73bb5162e3033
            Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
            Test Results https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/testReport/
            Max. process+thread count 2971 (vs. ulimit of 5500)
            modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
            Console output https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/console
            versions git=2.17.1 maven=3.6.0 findbugs=4.0.6
            Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 37m 6s   Docker mode activated.       Prechecks   +1 dupname 0m 0s   No case conflicting files found. +1 @author 0m 0s   The patch does not contain any @author tags. +1 test4tests 0m 0s   The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 2m 7s   Maven dependency ordering for branch +1 mvninstall 28m 3s   trunk passed +1 compile 5m 20s   trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 +1 compile 5m 2s   trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 +1 checkstyle 1m 11s   trunk passed +1 mvnsite 2m 35s   trunk passed +1 shadedclient 20m 33s   branch has no errors when building and testing our client artifacts. +1 javadoc 1m 27s   trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 +1 javadoc 1m 55s   trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 0 spotbugs 3m 19s   Used deprecated FindBugs config; considering switching to SpotBugs. +1 findbugs 5m 49s   trunk passed       Patch Compile Tests   0 mvndep 0m 23s   Maven dependency ordering for patch +1 mvninstall 2m 15s   the patch passed +1 compile 5m 3s   the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 +1 javac 5m 3s   the patch passed +1 compile 5m 4s   the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 +1 javac 5m 4s   the patch passed +1 blanks 0m 0s   The patch has no blanks issues. -0 checkstyle 0m 52s /results-checkstyle-hadoop-hdfs-project.txt hadoop-hdfs-project: The patch generated 4 new + 33 unchanged - 0 fixed = 37 total (was 33) +1 mvnsite 2m 0s   the patch passed +1 shadedclient 20m 56s   patch has no errors when building and testing our client artifacts. +1 javadoc 1m 25s   the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 +1 javadoc 1m 49s   the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 +1 findbugs 5m 53s   the patch passed       Other Tests   +1 unit 2m 18s   hadoop-hdfs-client in the patch passed. -1 unit 131m 40s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed. +1 asflicense 0m 41s   The patch does not generate ASF License warnings. 286m 50s   Reason Tests Failed junit tests hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/artifact/out/Dockerfile JIRA Issue HDFS-15684 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13015360/HDFS-15684.002.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux e9b7ab1ca3dc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / dd85a90da6f3c475915def97ace73bb5162e3033 Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 Test Results https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/testReport/ Max. process+thread count 2971 (vs. ulimit of 5500) modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/console versions git=2.17.1 maven=3.6.0 findbugs=4.0.6 Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            ferhui Hui Fei added a comment -

            wanghongbing Thanks for report and fix. It looks good. Could you please fix the checkstyle?
            hexiaoqiao leosun08 It is similar to HDFS-14694, Could you please take another look?

            ferhui Hui Fei added a comment - wanghongbing Thanks for report and fix. It looks good. Could you please fix the checkstyle? hexiaoqiao leosun08 It is similar to HDFS-14694 , Could you please take another look?
            hexiaoqiao Xiaoqiao He added a comment -

            Thanks wanghongbing and ferhui for your works. +1 from my side once fix the checkstyle.

            hexiaoqiao Xiaoqiao He added a comment - Thanks wanghongbing and ferhui for your works. +1 from my side once fix the checkstyle.
            wanghongbing Hongbing Wang added a comment -

            Thanks ferhui, hexiaoqiao. Fix the checkstyle in 003.patch. 

            wanghongbing Hongbing Wang added a comment - Thanks ferhui , hexiaoqiao . Fix the checkstyle in 003.patch. 
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 0m 40s   Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s   No case conflicting files found.
            0 codespell 0m 1s   codespell was not available.
            +1 @author 0m 0s   The patch does not contain any @author tags.
            +1 test4tests 0m 0s   The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 1m 56s   Maven dependency ordering for branch
            +1 mvninstall 20m 42s   trunk passed
            +1 compile 4m 19s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 compile 3m 47s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 checkstyle 0m 56s   trunk passed
            +1 mvnsite 2m 17s   trunk passed
            +1 shadedclient 18m 32s   branch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 34s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javadoc 2m 3s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            0 spotbugs 3m 9s   Used deprecated FindBugs config; considering switching to SpotBugs.
            +1 findbugs 5m 33s   trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 27s   Maven dependency ordering for patch
            +1 mvninstall 2m 1s   the patch passed
            +1 compile 4m 2s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javac 4m 2s   the patch passed
            +1 compile 3m 41s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 javac 3m 41s   the patch passed
            +1 blanks 0m 0s   The patch has no blanks issues.
            +1 checkstyle 0m 50s   the patch passed
            +1 mvnsite 2m 2s   the patch passed
            +1 shadedclient 18m 43s   patch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 25s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javadoc 1m 52s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 findbugs 5m 41s   the patch passed
                  Other Tests  
            +1 unit 2m 19s   hadoop-hdfs-client in the patch passed.
            -1 unit 114m 38s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
            -1 asflicense 0m 46s /results-asflicense.txt The patch generated 2 ASF License warnings.
            217m 7s  



            Reason Tests
            Failed junit tests hadoop.hdfs.TestDisableConnCache
              hadoop.hdfs.server.namenode.TestNamenodeRetryCache
              hadoop.hdfs.TestDatanodeReport
              hadoop.hdfs.TestPread
              hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
              hadoop.hdfs.TestDecommission
              hadoop.hdfs.TestGetFileChecksum
              hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
              hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
              hadoop.hdfs.TestParallelUnixDomainRead
              hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile
              hadoop.hdfs.TestErasureCodingPolicies
              hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy
              hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy
              hadoop.hdfs.TestMiniDFSCluster
              hadoop.hdfs.TestReconstructStripedFile
              hadoop.hdfs.TestMaintenanceState
              hadoop.hdfs.TestFileAppend2
              hadoop.hdfs.TestDFSStripedOutputStream
              hadoop.hdfs.TestReadStripedFileWithDNFailure
              hadoop.hdfs.server.balancer.TestBalancer
              hadoop.hdfs.TestDFSPermission
              hadoop.hdfs.server.namenode.TestFSImage
              hadoop.hdfs.TestDFSOutputStream
              hadoop.hdfs.TestDFSClientSocketSize
              hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
              hadoop.hdfs.client.impl.TestBlockReaderLocal
              hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
              hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
              hadoop.hdfs.TestEncryptionZones
              hadoop.hdfs.TestReadStripedFileWithMissingBlocks
              hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy



            Subsystem Report/Notes
            Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/artifact/out/Dockerfile
            JIRA Issue HDFS-15684
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13015695/HDFS-15684.003.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle codespell
            uname Linux e1787e5d1507 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / 8fa699b53fea8728e008c46af949f92543c08170
            Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            Test Results https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/testReport/
            Max. process+thread count 3266 (vs. ulimit of 5500)
            modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
            Console output https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/console
            versions git=2.17.1 maven=3.6.0 findbugs=4.0.6
            Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 0m 40s   Docker mode activated.       Prechecks   +1 dupname 0m 0s   No case conflicting files found. 0 codespell 0m 1s   codespell was not available. +1 @author 0m 0s   The patch does not contain any @author tags. +1 test4tests 0m 0s   The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 1m 56s   Maven dependency ordering for branch +1 mvninstall 20m 42s   trunk passed +1 compile 4m 19s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 compile 3m 47s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 checkstyle 0m 56s   trunk passed +1 mvnsite 2m 17s   trunk passed +1 shadedclient 18m 32s   branch has no errors when building and testing our client artifacts. +1 javadoc 1m 34s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javadoc 2m 3s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 0 spotbugs 3m 9s   Used deprecated FindBugs config; considering switching to SpotBugs. +1 findbugs 5m 33s   trunk passed       Patch Compile Tests   0 mvndep 0m 27s   Maven dependency ordering for patch +1 mvninstall 2m 1s   the patch passed +1 compile 4m 2s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javac 4m 2s   the patch passed +1 compile 3m 41s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 javac 3m 41s   the patch passed +1 blanks 0m 0s   The patch has no blanks issues. +1 checkstyle 0m 50s   the patch passed +1 mvnsite 2m 2s   the patch passed +1 shadedclient 18m 43s   patch has no errors when building and testing our client artifacts. +1 javadoc 1m 25s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javadoc 1m 52s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 findbugs 5m 41s   the patch passed       Other Tests   +1 unit 2m 19s   hadoop-hdfs-client in the patch passed. -1 unit 114m 38s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed. -1 asflicense 0m 46s /results-asflicense.txt The patch generated 2 ASF License warnings. 217m 7s   Reason Tests Failed junit tests hadoop.hdfs.TestDisableConnCache   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.TestDatanodeReport   hadoop.hdfs.TestPread   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDecommission   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor   hadoop.hdfs.TestParallelUnixDomainRead   hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile   hadoop.hdfs.TestErasureCodingPolicies   hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy   hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy   hadoop.hdfs.TestMiniDFSCluster   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestMaintenanceState   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.TestReadStripedFileWithDNFailure   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestDFSPermission   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.TestDFSOutputStream   hadoop.hdfs.TestDFSClientSocketSize   hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy   hadoop.hdfs.client.impl.TestBlockReaderLocal   hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy Subsystem Report/Notes Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/artifact/out/Dockerfile JIRA Issue HDFS-15684 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13015695/HDFS-15684.003.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle codespell uname Linux e1787e5d1507 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / 8fa699b53fea8728e008c46af949f92543c08170 Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 Test Results https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/testReport/ Max. process+thread count 3266 (vs. ulimit of 5500) modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/console versions git=2.17.1 maven=3.6.0 findbugs=4.0.6 Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            wanghongbing Hongbing Wang added a comment -

            `TestDFSOutputStream` pass in local. Other OOM failed tests also pass in local when Randomly testing some.

            wanghongbing Hongbing Wang added a comment - `TestDFSOutputStream` pass in local. Other OOM failed tests also pass in local when Randomly testing some.
            hexiaoqiao Xiaoqiao He added a comment -

            trigger Jenkins manually and let see what it says.

            hexiaoqiao Xiaoqiao He added a comment - trigger Jenkins manually and let see what it says.
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 0m 48s   Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s   No case conflicting files found.
            0 codespell 0m 0s   codespell was not available.
            +1 @author 0m 0s   The patch does not contain any @author tags.
            +1 test4tests 0m 0s   The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 1m 55s   Maven dependency ordering for branch
            +1 mvninstall 23m 3s   trunk passed
            +1 compile 4m 22s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 compile 3m 54s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 checkstyle 0m 56s   trunk passed
            +1 mvnsite 2m 8s   trunk passed
            +1 shadedclient 19m 49s   branch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 27s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javadoc 1m 55s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            0 spotbugs 3m 10s   Used deprecated FindBugs config; considering switching to SpotBugs.
            +1 findbugs 5m 40s   trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 22s   Maven dependency ordering for patch
            +1 mvninstall 1m 58s   the patch passed
            +1 compile 4m 13s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javac 4m 13s   the patch passed
            +1 compile 3m 47s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 javac 3m 47s   the patch passed
            +1 blanks 0m 1s   The patch has no blanks issues.
            +1 checkstyle 0m 52s   the patch passed
            +1 mvnsite 2m 0s   the patch passed
            +1 shadedclient 20m 40s   patch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 28s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
            +1 javadoc 1m 51s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            +1 findbugs 6m 12s   the patch passed
                  Other Tests  
            +1 unit 2m 15s   hadoop-hdfs-client in the patch passed.
            -1 unit 129m 49s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
            +1 asflicense 0m 53s   The patch does not generate ASF License warnings.
            238m 23s  



            Reason Tests
            Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMXBean
              hadoop.hdfs.TestMultipleNNPortQOP
              hadoop.hdfs.server.namenode.TestFSImage



            Subsystem Report/Notes
            Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/artifact/out/Dockerfile
            JIRA Issue HDFS-15684
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13015695/HDFS-15684.003.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle codespell
            uname Linux ed66460ced89 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / 740399a602d7ad0cd63eaccd2de6d5a506f3e913
            Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
            Test Results https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/testReport/
            Max. process+thread count 2704 (vs. ulimit of 5500)
            modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
            Console output https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/console
            versions git=2.17.1 maven=3.6.0 findbugs=4.0.6
            Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 0m 48s   Docker mode activated.       Prechecks   +1 dupname 0m 0s   No case conflicting files found. 0 codespell 0m 0s   codespell was not available. +1 @author 0m 0s   The patch does not contain any @author tags. +1 test4tests 0m 0s   The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 1m 55s   Maven dependency ordering for branch +1 mvninstall 23m 3s   trunk passed +1 compile 4m 22s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 compile 3m 54s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 checkstyle 0m 56s   trunk passed +1 mvnsite 2m 8s   trunk passed +1 shadedclient 19m 49s   branch has no errors when building and testing our client artifacts. +1 javadoc 1m 27s   trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javadoc 1m 55s   trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 0 spotbugs 3m 10s   Used deprecated FindBugs config; considering switching to SpotBugs. +1 findbugs 5m 40s   trunk passed       Patch Compile Tests   0 mvndep 0m 22s   Maven dependency ordering for patch +1 mvninstall 1m 58s   the patch passed +1 compile 4m 13s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javac 4m 13s   the patch passed +1 compile 3m 47s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 javac 3m 47s   the patch passed +1 blanks 0m 1s   The patch has no blanks issues. +1 checkstyle 0m 52s   the patch passed +1 mvnsite 2m 0s   the patch passed +1 shadedclient 20m 40s   patch has no errors when building and testing our client artifacts. +1 javadoc 1m 28s   the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 +1 javadoc 1m 51s   the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 +1 findbugs 6m 12s   the patch passed       Other Tests   +1 unit 2m 15s   hadoop-hdfs-client in the patch passed. -1 unit 129m 49s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed. +1 asflicense 0m 53s   The patch does not generate ASF License warnings. 238m 23s   Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.TestMultipleNNPortQOP   hadoop.hdfs.server.namenode.TestFSImage Subsystem Report/Notes Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/artifact/out/Dockerfile JIRA Issue HDFS-15684 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13015695/HDFS-15684.003.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle codespell uname Linux ed66460ced89 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / 740399a602d7ad0cd63eaccd2de6d5a506f3e913 Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 Test Results https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/testReport/ Max. process+thread count 2704 (vs. ulimit of 5500) modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/console versions git=2.17.1 maven=3.6.0 findbugs=4.0.6 Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            hexiaoqiao Xiaoqiao He added a comment -

            +1 for HDFS-15684.003.patch. Will commit shortly once ferhui give another confirms.

            hexiaoqiao Xiaoqiao He added a comment - +1 for HDFS-15684.003.patch . Will commit shortly once ferhui give another confirms.
            ferhui Hui Fei added a comment -

            +1 HDFS-15684.003.patch . Thanks!

            ferhui Hui Fei added a comment - +1 HDFS-15684.003.patch . Thanks!
            hexiaoqiao Xiaoqiao He added a comment -

            Committed to trunk.
            Thanks wanghongbing for contribution! Thanks ferhui for your reviews!

            hexiaoqiao Xiaoqiao He added a comment - Committed to trunk. Thanks wanghongbing for contribution! Thanks ferhui for your reviews!
            ferhui Hui Fei added a comment -

            wanghongbing cherry-pick failed, could you please give patches on branch 3.3, 3.2, 3.1? Thanks

            ferhui Hui Fei added a comment - wanghongbing  cherry-pick failed, could you please give patches on branch 3.3, 3.2, 3.1? Thanks
            wanghongbing Hongbing Wang added a comment -

            ferhui ok. Because this PR depends on  HDFS-14694, I will backport them in another Jira later.

            wanghongbing Hongbing Wang added a comment - ferhui  ok. Because this PR depends on  HDFS-14694 , I will backport them in another Jira later.

            People

              wanghongbing Hongbing Wang
              wanghongbing Hongbing Wang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: