Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2498

TestParallelRead times out consistently on Jenkins

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Not a Problem
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: test
    • Labels:
      None

      Description

      During last several Jenkins builds TestParallelRead consistently fails. See Hadoop-Hdfs-22-branch for logs.

      1. TestParallelRead.txt
        32 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          I cannot reproduce it locally. So I ran Jenkins with -Dtest.output=yes. Here is the log.

          Show
          Konstantin Shvachko added a comment - I cannot reproduce it locally. So I ran Jenkins with -Dtest.output=yes . Here is the log.
          Hide
          Konstantin Shvachko added a comment -

          Upgrading to a blocker as test failure.

          Show
          Konstantin Shvachko added a comment - Upgrading to a blocker as test failure.
          Hide
          Konstantin Shvachko added a comment -

          See the following assert in the logs when the test actually succeeded:

          Exception in thread "ReadWorker-7-/TestParallelRead.dat.0" java.lang.AssertionError: Bad args: 262098 + 46 should be < 262144
          	at org.junit.Assert.fail(Assert.java:91)
          	at org.junit.Assert.assertTrue(Assert.java:43)
          	at org.apache.hadoop.hdfs.TestParallelRead$ReadWorker.read(TestParallelRead.java:137)
          	at org.apache.hadoop.hdfs.TestParallelRead$ReadWorker.run(TestParallelRead.java:104)
          

          This is a wrong condition, should be <=

          Show
          Konstantin Shvachko added a comment - See the following assert in the logs when the test actually succeeded: Exception in thread "ReadWorker-7-/TestParallelRead.dat.0" java.lang.AssertionError: Bad args: 262098 + 46 should be < 262144 at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.hadoop.hdfs.TestParallelRead$ReadWorker.read(TestParallelRead.java:137) at org.apache.hadoop.hdfs.TestParallelRead$ReadWorker.run(TestParallelRead.java:104) This is a wrong condition, should be <=
          Hide
          Todd Lipcon added a comment -

          Dup of HDFS-2287

          Show
          Todd Lipcon added a comment - Dup of HDFS-2287
          Hide
          Konstantin Shvachko added a comment -

          You are right, Todd. I'll incorporate.
          I see that even when TestParallelRead succeeds (rather rarely) on Jenkins its running time is around 14 minutes. While on my linux box it completes in less than 20 seconds.
          And in the log attached I see

          [junit] 2011-10-24 19:58:35,474 INFO  hdfs.TestParallelRead (TestParallelRead.java:runParallelRead(269)) - === Report: 16 threads read 941298 KB (across 1 file(s)) in 550.051s; average 1711.292225629987 KB/s
          

          So the second runParallelRead() is actually running more than 9 minutes (550.051s). Again, on my box it completes in only 2-3 seconds.

          Is it possible that because ReadWorker,run() does not catch Error (only Exception) the thread cannot complete?

          Show
          Konstantin Shvachko added a comment - You are right, Todd. I'll incorporate. I see that even when TestParallelRead succeeds (rather rarely) on Jenkins its running time is around 14 minutes. While on my linux box it completes in less than 20 seconds. And in the log attached I see [junit] 2011-10-24 19:58:35,474 INFO hdfs.TestParallelRead (TestParallelRead.java:runParallelRead(269)) - === Report: 16 threads read 941298 KB (across 1 file(s)) in 550.051s; average 1711.292225629987 KB/s So the second runParallelRead() is actually running more than 9 minutes (550.051s). Again, on my box it completes in only 2-3 seconds. Is it possible that because ReadWorker,run() does not catch Error (only Exception) the thread cannot complete?
          Hide
          Todd Lipcon added a comment -

          That's strange that it's so slow - hundreds of times slower. Is the Jenkins build directory hosted on NFS or something? Is this the Apache jenkins boxes or one of your internal hosts?

          Show
          Todd Lipcon added a comment - That's strange that it's so slow - hundreds of times slower. Is the Jenkins build directory hosted on NFS or something? Is this the Apache jenkins boxes or one of your internal hosts?
          Hide
          Konstantin Shvachko added a comment -

          This is Apache Jenkins. Here is how I just reproduced it:
          https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-22-branch/113/console

          Login to Jenkins.
          Go to: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-22-branch/
          Click on "Build now"
          In the BUILD_ARGS add -Dtestcase=TestParallelRead -Dtest.output=yes
          In the BUILD_TARGETS replace with clean test

          Show
          Konstantin Shvachko added a comment - This is Apache Jenkins. Here is how I just reproduced it: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-22-branch/113/console Login to Jenkins. Go to: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-22-branch/ Click on "Build now" In the BUILD_ARGS add -Dtestcase=TestParallelRead -Dtest.output=yes In the BUILD_TARGETS replace with clean test
          Hide
          Suresh Srinivas added a comment -

          This is a very old jira. I am going to resolve this as not a problem.

          Show
          Suresh Srinivas added a comment - This is a very old jira. I am going to resolve this as not a problem.

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development