Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9565

TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes is flaky due to race condition

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs, test
    • Labels:
      None
    • Environment:

      Jenkins

    • Hadoop Flags:
      Reviewed

      Description

      TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes occasionally fails with the following error:
      https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/699/testReport/org.apache.hadoop.hdfs/TestDistributedFileSystem/testLocatedFileStatusStorageIdsTypes/

      FAILED:  org.apache.hadoop.hdfs.TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes
      
      Error Message:
      Unexpected num storage ids expected:<2> but was:<1>
      
      Stack Trace:
      java.lang.AssertionError: Unexpected num storage ids expected:<2> but was:<1>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.apache.hadoop.hdfs.TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes(TestDistributedFileSystem.java:855)
      
      

      It appears that this test failed due to race condition: it does not wait for the file replication to finish, before checking the file's status.

      This flaky test can be fixed by using DFSTestUtil.waitForReplication()

      1. HDFS-9565.001.patch
        0.9 kB
        Wei-Chiu Chuang

        Activity

        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        rev01: use DFSTestUtil.waitForReplication() to avoid race condition between file creation and listLocatedStatus.

        Show
        jojochuang Wei-Chiu Chuang added a comment - rev01: use DFSTestUtil.waitForReplication() to avoid race condition between file creation and listLocatedStatus.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 pending Jenkins.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 pending Jenkins.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for +1! Arpit Agarwal

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for +1! Arpit Agarwal
        Hide
        liuml07 Mingliang Liu added a comment -

        +1 (non-binding)

        Show
        liuml07 Mingliang Liu added a comment - +1 (non-binding)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 1s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 0s trunk passed
        +1 compile 0m 45s trunk passed with JDK v1.8.0_66
        +1 compile 0m 45s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 2m 9s trunk passed
        +1 javadoc 1m 19s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 2s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 55s the patch passed
        +1 compile 0m 50s the patch passed with JDK v1.8.0_66
        +1 javac 0m 50s the patch passed
        +1 compile 0m 52s the patch passed with JDK v1.7.0_91
        +1 javac 0m 52s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 0m 59s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 16s the patch passed
        +1 javadoc 1m 18s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 56s the patch passed with JDK v1.7.0_91
        -1 unit 66m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 58m 33s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 asflicense 0m 26s Patch generated 58 ASF License warnings.
        155m 28s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.namenode.TestAuditLogs
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.TestDFSShell
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778097/HDFS-9565.001.patch
        JIRA Issue HDFS-9565
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0f760c742808 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0f708d4
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13901/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13901/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 0s trunk passed +1 compile 0m 45s trunk passed with JDK v1.8.0_66 +1 compile 0m 45s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 9s trunk passed +1 javadoc 1m 19s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 2s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 55s the patch passed +1 compile 0m 50s the patch passed with JDK v1.8.0_66 +1 javac 0m 50s the patch passed +1 compile 0m 52s the patch passed with JDK v1.7.0_91 +1 javac 0m 52s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 18s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 56s the patch passed with JDK v1.7.0_91 -1 unit 66m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 58m 33s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 26s Patch generated 58 ASF License warnings. 155m 28s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.TestAuditLogs   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestDFSShell   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.TestReadStripedFileWithMissingBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778097/HDFS-9565.001.patch JIRA Issue HDFS-9565 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0f760c742808 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0f708d4 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13901/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13901/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13901/console This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Test failures are all unrelated, because this patch only touches TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes

        Show
        jojochuang Wei-Chiu Chuang added a comment - Test failures are all unrelated, because this patch only touches TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        I committed this for 2.8.0. Thanks for helping improve test reliability Wei-Chiu Chuang.

        Show
        arpitagarwal Arpit Agarwal added a comment - I committed this for 2.8.0. Thanks for helping improve test reliability Wei-Chiu Chuang .
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thank you very much! Arpit Agarwal

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thank you very much! Arpit Agarwal
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8985/)
        HDFS-9565. (arp: rev 72a3f649e4db2b1cc18a8e203c321cc1832435e4)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8985/ ) HDFS-9565 . (arp: rev 72a3f649e4db2b1cc18a8e203c321cc1832435e4) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            jojochuang Wei-Chiu Chuang
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development