Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9476

TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This test occasionally fail. For example, the most recent one is:
      https://builds.apache.org/job/Hadoop-Hdfs-trunk/2587/

      Error Message

      Cannot obtain block length for LocatedBlock{BP-1371507683-67.195.81.153-1448798439809:blk_7162739548153522810_1020; getBlockSize()=1024; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[127.0.0.1:33080,DS-c5eaf2b4-2ee6-419d-a8a0-44a5df5ef9a1,DISK]]}
      

      Stacktrace

      java.io.IOException: Cannot obtain block length for LocatedBlock{BP-1371507683-67.195.81.153-1448798439809:blk_7162739548153522810_1020; getBlockSize()=1024; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[127.0.0.1:33080,DS-c5eaf2b4-2ee6-419d-a8a0-44a5df5ef9a1,DISK]]}
      	at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:399)
      	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:343)
      	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:275)
      	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:265)
      	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1046)
      	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1011)
      	at org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.dfsOpenFileWithRetries(TestDFSUpgradeFromImage.java:177)
      	at org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.verifyDir(TestDFSUpgradeFromImage.java:213)
      	at org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.verifyFileSystem(TestDFSUpgradeFromImage.java:228)
      	at org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.upgradeAndVerify(TestDFSUpgradeFromImage.java:600)
      	at org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.testUpgradeFromRel1BBWImage(TestDFSUpgradeFromImage.java:622)
      
      1. HDFS-9476.002.patch
        2 kB
        Masatake Iwasaki
      2. HDFS-9476.01.patch
        1 kB
        Akira Ajisaka

        Activity

        Show
        liuml07 Mingliang Liu added a comment - It happens in recent build as well, see UT log at: https://builds.apache.org/job/PreCommit-HDFS-Build/14230/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        Hide
        ajisakaa Akira Ajisaka added a comment -

        It seems that lease recovery did not completed even after 10 retries. 01 patch raise the number of retries from 10 to 30.

        Show
        ajisakaa Akira Ajisaka added a comment - It seems that lease recovery did not completed even after 10 retries. 01 patch raise the number of retries from 10 to 30.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 15m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 39s trunk passed
        +1 compile 0m 54s trunk passed with JDK v1.8.0_72
        +1 compile 0m 46s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 1m 19s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 59s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 53s the patch passed with JDK v1.8.0_72
        +1 javac 0m 53s the patch passed
        +1 compile 0m 44s the patch passed with JDK v1.7.0_95
        +1 javac 0m 44s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 16s the patch passed
        +1 javadoc 1m 19s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 55s the patch passed with JDK v1.7.0_95
        -1 unit 81m 50s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        -1 unit 76m 27s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        202m 16s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
          hadoop.hdfs.server.namenode.TestFileTruncate



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787890/HDFS-9476.01.patch
        JIRA Issue HDFS-9476
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 143b982a51d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ac5da11
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14495/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14495/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 15m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 39s trunk passed +1 compile 0m 54s trunk passed with JDK v1.8.0_72 +1 compile 0m 46s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 19s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 59s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 52s the patch passed +1 compile 0m 53s the patch passed with JDK v1.8.0_72 +1 javac 0m 53s the patch passed +1 compile 0m 44s the patch passed with JDK v1.7.0_95 +1 javac 0m 44s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 55s the patch passed with JDK v1.7.0_95 -1 unit 81m 50s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 76m 27s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 202m 16s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787890/HDFS-9476.01.patch JIRA Issue HDFS-9476 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 143b982a51d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac5da11 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14495/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14495/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        The message "Cannot obtain block length for LocatedBlock..." implies that DFSInputStream#readBlockLength failed. TestDFSUpgradeFromImage#dfsOpenFileWithRetries will not retry in that case. Just increasing retries will not work.

        Show
        iwasakims Masatake Iwasaki added a comment - The message "Cannot obtain block length for LocatedBlock..." implies that DFSInputStream#readBlockLength failed. TestDFSUpgradeFromImage#dfsOpenFileWithRetries will not retry in that case. Just increasing retries will not work.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Sorry, this was wrong.. "Cannot obtain block length for LocatedBlock..." is the retrying case.

        Show
        iwasakims Masatake Iwasaki added a comment - Sorry, this was wrong.. "Cannot obtain block length for LocatedBlock..." is the retrying case.
        Hide
        walter.k.su Walter Su added a comment -

        +1.

        Show
        walter.k.su Walter Su added a comment - +1.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        The issue was reproduced even after the 001 was applied.

        2016-04-30 22:09:02,681 [IPC Server handler 1 on 38857] INFO  FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true       ugi=iwasakims (auth:SIMPLE)     ip=/127.0.0.1   cmd=open        src=/1kb-multiple-checksum-blocks-64-16 dst=null        perm=null       proto=rpc
        2016-04-30 22:09:02,753 [main] INFO  hdfs.TestDFSUpgradeFromImage (TestDFSUpgradeFromImage.java:dfsOpenFileWithRetries(187)) - Open failed. 0 times. Retrying.
        ...(snip)
        2016-04-30 22:09:32,048 [IPC Server handler 4 on 38857] INFO  FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true       ugi=iwasakims (auth:SIMPLE)     ip=/127.0.0.1   cmd=open        src=/1kb-multiple-checksum-blocks-64-16 dst=null        perm=null       proto=rpc
        2016-04-30 22:09:32,056 [main] INFO  hdfs.TestDFSUpgradeFromImage (TestDFSUpgradeFromImage.java:dfsOpenFileWithRetries(187)) - Open failed. 29 times. Retrying.
        

        The test seems to fail if lease recovery happens (due to client's request) before the datanode's storage report is processed. This looks like not a test problem but real race.

        2016-04-30 22:09:02,537 [IPC Server handler 2 on 38857] INFO  FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true       ugi=iwasakims (auth:SIMPLE)     ip=/127.0.0.1   cmd=getfileinfo src=/1kb-multiple-checksum-blocks-64-16 dst=null        perm=null       proto=rpc
        2016-04-30 22:09:02,544 [IPC Server handler 4 on 38857] INFO  namenode.FSNamesystem (FSNamesystem.java:recoverLeaseInternal(2365)) - recoverLease: [Lease.  Holder: DFSClient_8256078, pending creates: 13], src=/1kb-multiple-checksum-blocks-64-16 from client DFSClient_8256078
        2016-04-30 22:09:02,544 [IPC Server handler 4 on 38857] INFO  namenode.FSNamesystem (FSNamesystem.java:internalReleaseLease(3096)) - Recovering [Lease.  Holder: DFSClient_8256078, pending creates: 13], src=/1kb-multiple-checksum-blocks-64-16
        2016-04-30 22:09:02,545 [IPC Server handler 4 on 38857] WARN  BlockStateChange (BlockUnderConstructionFeature.java:initializeBlockRecovery(205)) - BLOCK* BlockUnderConstructionFeature.initializeBlockRecovery: No blocks found, lease removed.
        2016-04-30 22:09:02,545 [IPC Server handler 4 on 38857] WARN  hdfs.StateChange (FSNamesystem.java:internalReleaseLease(3220)) - DIR* NameSystem.internalReleaseLease: File /1kb-multiple-checksum-blocks-64-16 has not been closed. Lease recovery is in progress. RecoveryId = 1031 for block blk_7162739548153522810_1020
        2016-04-30 22:09:02,545 [Block report processor] INFO  blockmanagement.BlockManager (BlockManager.java:processReport(2174)) - Processing first storage report for DS-a78b43f5-967a-47e3-afe1-68bb2f73a7eb from datanode e6f67da5-812b-402b-b5ac-dad5deab2666
        
        Show
        iwasakims Masatake Iwasaki added a comment - The issue was reproduced even after the 001 was applied. 2016-04-30 22:09:02,681 [IPC Server handler 1 on 38857] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null proto=rpc 2016-04-30 22:09:02,753 [main] INFO hdfs.TestDFSUpgradeFromImage (TestDFSUpgradeFromImage.java:dfsOpenFileWithRetries(187)) - Open failed. 0 times. Retrying. ...(snip) 2016-04-30 22:09:32,048 [IPC Server handler 4 on 38857] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null proto=rpc 2016-04-30 22:09:32,056 [main] INFO hdfs.TestDFSUpgradeFromImage (TestDFSUpgradeFromImage.java:dfsOpenFileWithRetries(187)) - Open failed. 29 times. Retrying. The test seems to fail if lease recovery happens (due to client's request) before the datanode's storage report is processed. This looks like not a test problem but real race. 2016-04-30 22:09:02,537 [IPC Server handler 2 on 38857] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null proto=rpc 2016-04-30 22:09:02,544 [IPC Server handler 4 on 38857] INFO namenode.FSNamesystem (FSNamesystem.java:recoverLeaseInternal(2365)) - recoverLease: [Lease. Holder: DFSClient_8256078, pending creates: 13], src=/1kb-multiple-checksum-blocks-64-16 from client DFSClient_8256078 2016-04-30 22:09:02,544 [IPC Server handler 4 on 38857] INFO namenode.FSNamesystem (FSNamesystem.java:internalReleaseLease(3096)) - Recovering [Lease. Holder: DFSClient_8256078, pending creates: 13], src=/1kb-multiple-checksum-blocks-64-16 2016-04-30 22:09:02,545 [IPC Server handler 4 on 38857] WARN BlockStateChange (BlockUnderConstructionFeature.java:initializeBlockRecovery(205)) - BLOCK* BlockUnderConstructionFeature.initializeBlockRecovery: No blocks found, lease removed. 2016-04-30 22:09:02,545 [IPC Server handler 4 on 38857] WARN hdfs.StateChange (FSNamesystem.java:internalReleaseLease(3220)) - DIR* NameSystem.internalReleaseLease: File /1kb-multiple-checksum-blocks-64-16 has not been closed. Lease recovery is in progress. RecoveryId = 1031 for block blk_7162739548153522810_1020 2016-04-30 22:09:02,545 [Block report processor] INFO blockmanagement.BlockManager (BlockManager.java:processReport(2174)) - Processing first storage report for DS-a78b43f5-967a-47e3-afe1-68bb2f73a7eb from datanode e6f67da5-812b-402b-b5ac-dad5deab2666
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 51s trunk passed
        +1 compile 0m 55s trunk passed with JDK v1.8.0_92
        +1 compile 0m 47s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 1m 14s trunk passed
        +1 mvneclipse 0m 21s trunk passed
        +1 findbugs 2m 35s trunk passed
        +1 javadoc 1m 45s trunk passed with JDK v1.8.0_92
        +1 javadoc 2m 26s trunk passed with JDK v1.7.0_95
        +1 mvninstall 1m 1s the patch passed
        +1 compile 1m 5s the patch passed with JDK v1.8.0_92
        +1 javac 1m 5s the patch passed
        +1 compile 0m 51s the patch passed with JDK v1.7.0_95
        +1 javac 0m 51s the patch passed
        +1 checkstyle 0m 28s the patch passed
        +1 mvnsite 1m 16s the patch passed
        +1 mvneclipse 0m 18s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 54s the patch passed
        +1 javadoc 1m 47s the patch passed with JDK v1.8.0_92
        +1 javadoc 2m 8s the patch passed with JDK v1.7.0_95
        -1 unit 84m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_92.
        -1 unit 93m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        211m 0s



        Reason Tests
        JDK v1.8.0_92 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.TestFileAppend
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787890/HDFS-9476.01.patch
        JIRA Issue HDFS-9476
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 03d8fffb12ab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2753185
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15329/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15329/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15329/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15329/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15329/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15329/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 51s trunk passed +1 compile 0m 55s trunk passed with JDK v1.8.0_92 +1 compile 0m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 14s trunk passed +1 mvneclipse 0m 21s trunk passed +1 findbugs 2m 35s trunk passed +1 javadoc 1m 45s trunk passed with JDK v1.8.0_92 +1 javadoc 2m 26s trunk passed with JDK v1.7.0_95 +1 mvninstall 1m 1s the patch passed +1 compile 1m 5s the patch passed with JDK v1.8.0_92 +1 javac 1m 5s the patch passed +1 compile 0m 51s the patch passed with JDK v1.7.0_95 +1 javac 0m 51s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 54s the patch passed +1 javadoc 1m 47s the patch passed with JDK v1.8.0_92 +1 javadoc 2m 8s the patch passed with JDK v1.7.0_95 -1 unit 84m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_92. -1 unit 93m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 211m 0s Reason Tests JDK v1.8.0_92 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestFileAppend JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787890/HDFS-9476.01.patch JIRA Issue HDFS-9476 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 03d8fffb12ab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2753185 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15329/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15329/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15329/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15329/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15329/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15329/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        This looks like not a test problem but real race.

        This was not true. Checking return value of recoverLease and retrying in TestDFSUpgradeFromImage#recoverAllLeases fixed the issue. I attached update patch.

        Show
        iwasakims Masatake Iwasaki added a comment - This looks like not a test problem but real race. This was not true. Checking return value of recoverLease and retrying in TestDFSUpgradeFromImage#recoverAllLeases fixed the issue. I attached update patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 17s trunk passed
        +1 compile 0m 52s trunk passed with JDK v1.8.0_92
        +1 compile 0m 44s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 0s trunk passed
        +1 javadoc 1m 16s trunk passed with JDK v1.8.0_92
        +1 javadoc 1m 57s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 50s the patch passed
        +1 compile 0m 51s the patch passed with JDK v1.8.0_92
        +1 javac 0m 51s the patch passed
        +1 compile 0m 42s the patch passed with JDK v1.7.0_95
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 11s the patch passed
        +1 javadoc 1m 12s the patch passed with JDK v1.8.0_92
        +1 javadoc 1m 53s the patch passed with JDK v1.7.0_95
        -1 unit 107m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_92.
        -1 unit 86m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        220m 47s



        Reason Tests
        JDK v1.8.0_92 Failed junit tests hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
          hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.fs.TestSymlinkHdfsFileContext
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801622/HDFS-9476.002.patch
        JIRA Issue HDFS-9476
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c4409c43730f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 971af60
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15331/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15331/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 17s trunk passed +1 compile 0m 52s trunk passed with JDK v1.8.0_92 +1 compile 0m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_92 +1 javadoc 1m 57s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 50s the patch passed +1 compile 0m 51s the patch passed with JDK v1.8.0_92 +1 javac 0m 51s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_95 +1 javac 0m 42s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 11s the patch passed +1 javadoc 1m 12s the patch passed with JDK v1.8.0_92 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_95 -1 unit 107m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_92. -1 unit 86m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 220m 47s Reason Tests JDK v1.8.0_92 Failed junit tests hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestReconstructStripedFile   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801622/HDFS-9476.002.patch JIRA Issue HDFS-9476 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c4409c43730f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 971af60 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15331/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15331/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        The new patch looks good, +1.

        Show
        xiaobingo Xiaobing Zhou added a comment - The new patch looks good, +1.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1, committing this.

        Show
        ajisakaa Akira Ajisaka added a comment - +1, committing this.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed to branch-2.7 and above. Thanks Masatake Iwasaki for finding the root cause and updating the patch, and thanks Xiaobing Zhou and Walter Su for reviewing.

        Show
        ajisakaa Akira Ajisaka added a comment - Committed to branch-2.7 and above. Thanks Masatake Iwasaki for finding the root cause and updating the patch, and thanks Xiaobing Zhou and Walter Su for reviewing.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9897 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9897/)
        HDFS-9476. TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage (aajisaka: rev 69555fca066815053dd9168ebe15868a5c02cdcd)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9897 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9897/ ) HDFS-9476 . TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage (aajisaka: rev 69555fca066815053dd9168ebe15868a5c02cdcd) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Thanks, Akira Ajisaka.

        Show
        iwasakims Masatake Iwasaki added a comment - Thanks, Akira Ajisaka .
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            iwasakims Masatake Iwasaki
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development