Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3243

TestParallelRead timing out on jenkins

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: hdfs-client, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Trunk builds have been failing recently due to a TestParallelRead timeout. It doesn't report in the Jenkins failure list because surefire handles timeouts really poorly.

      1. HDFS-3243.0.patch
        0.7 kB
        Henry Robinson

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Seems like this test was lengthened by HDFS-2834. I'm not sure yet whether it represents a performance regression, or if the test itself was just changed in such a way that it runs much longer.

          On branch-2:

            <testcase time="9.15" classname="org.apache.hadoop.hdfs.TestParallelRead" name="testParallelRead"/>
          

          On trunk:

            <testcase time="23.397" classname="org.apache.hadoop.hdfs.TestParallelRead" name="testParallelReadCopying"/>
            <testcase time="133.218" classname="org.apache.hadoop.hdfs.TestParallelRead" name="testParallelReadByteBuffer"/>
            <testcase time="61.364" classname="org.apache.hadoop.hdfs.TestParallelRead" name="testParallelReadMixed"/>
          

          I also see a lot of blocked threads in the jstack on trunk. I asked Henry to take a look at this.

          Show
          Todd Lipcon added a comment - Seems like this test was lengthened by HDFS-2834 . I'm not sure yet whether it represents a performance regression, or if the test itself was just changed in such a way that it runs much longer. On branch-2: <testcase time= "9.15" classname= "org.apache.hadoop.hdfs.TestParallelRead" name= "testParallelRead" /> On trunk: <testcase time= "23.397" classname= "org.apache.hadoop.hdfs.TestParallelRead" name= "testParallelReadCopying" /> <testcase time= "133.218" classname= "org.apache.hadoop.hdfs.TestParallelRead" name= "testParallelReadByteBuffer" /> <testcase time= "61.364" classname= "org.apache.hadoop.hdfs.TestParallelRead" name= "testParallelReadMixed" /> I also see a lot of blocked threads in the jstack on trunk. I asked Henry to take a look at this.
          Hide
          Todd Lipcon added a comment -

          fwiw I tried copying the old TestParallelRead from branch-2 into trunk, and it runs just as fast there as it does in branch-2. So this seems like an issue with the new test code, rather than a regression in the read performance of the existing path.

          Show
          Todd Lipcon added a comment - fwiw I tried copying the old TestParallelRead from branch-2 into trunk, and it runs just as fast there as it does in branch-2. So this seems like an issue with the new test code, rather than a regression in the read performance of the existing path.
          Hide
          Henry Robinson added a comment -

          Todd mentions that the test does eventually finish, so I don't think this is a correctness bug (thankfully). I'm looking to see how the test changes could have made this much slower.

          Show
          Henry Robinson added a comment - Todd mentions that the test does eventually finish, so I don't think this is a correctness bug (thankfully). I'm looking to see how the test changes could have made this much slower.
          Hide
          Henry Robinson added a comment -

          Turns out I left N_ITERATIONS as 4096 in HDFS-2834, not 1024 as it used to be. The tests run much more quickly for me now, as you might expect, and directly comparable with the old test code run against trunk.

          This does shine a light on a separate issue, which is that ByteBufferReadable lacks a lockless pread, which is faked out in this test by seeking-then-reading under a lock, with a bit of unnecessary serialisation as a result. I've opened HDSF-3246 to track this.

          Show
          Henry Robinson added a comment - Turns out I left N_ITERATIONS as 4096 in HDFS-2834 , not 1024 as it used to be. The tests run much more quickly for me now, as you might expect, and directly comparable with the old test code run against trunk. This does shine a light on a separate issue, which is that ByteBufferReadable lacks a lockless pread, which is faked out in this test by seeking-then-reading under a lock, with a bit of unnecessary serialisation as a result. I've opened HDSF-3246 to track this.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522151/HDFS-3243.0.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2243//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2243//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522151/HDFS-3243.0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2243//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2243//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1, verified in the test results above that TestParallelRead passed relatively quickly.

          Show
          Todd Lipcon added a comment - +1, verified in the test results above that TestParallelRead passed relatively quickly.
          Hide
          Todd Lipcon added a comment -

          Committed to trunk, thanks

          Show
          Todd Lipcon added a comment - Committed to trunk, thanks
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2120/)
          HDFS-3243. TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2120/ ) HDFS-3243 . TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2046 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2046/)
          HDFS-3243. TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2046 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2046/ ) HDFS-3243 . TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2059 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2059/)
          HDFS-3243. TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029)

          Result = ABORTED
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2059 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2059/ ) HDFS-3243 . TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1011 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1011/)
          HDFS-3243. TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1011 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1011/ ) HDFS-3243 . TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1046 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1046/)
          HDFS-3243. TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1046 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1046/ ) HDFS-3243 . TestParallelRead timing out on jenkins. Contributed by Henry Robinson. (Revision 1312029) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1312029 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java
          Hide
          Todd Lipcon added a comment -

          Committed to branch-2 as well.

          Show
          Todd Lipcon added a comment - Committed to branch-2 as well.

            People

            • Assignee:
              Henry Robinson
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development