Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In recent Hudson builds, TestBlockFixer has been timing out. Not clear how long it has been broken since MAPREDUCE-2394 was hiding the RAID tests from Hudson's test result parsing.

      1. MAPREDUCE-2395.patch
        30 kB
        Ramkumar Vadali

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)
        MAPREDUCE-2395. TestBlockFixer timing out on trunk. Contributed by Ramkumar Vadali.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ ) MAPREDUCE-2395 . TestBlockFixer timing out on trunk. Contributed by Ramkumar Vadali.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #638 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/638/)
        MAPREDUCE-2395. TestBlockFixer timing out on trunk. Contributed by Ramkumar Vadali.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #638 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/638/ ) MAPREDUCE-2395 . TestBlockFixer timing out on trunk. Contributed by Ramkumar Vadali.
        Hide
        Todd Lipcon added a comment -

        Committed to trunk, thanks Ramkumar!

        Show
        Todd Lipcon added a comment - Committed to trunk, thanks Ramkumar!
        Hide
        Todd Lipcon added a comment -

        Yes, I ran this test a few times and it looks like that failure only happens intermittently (though I was able to reproduce it once). I will commit this to trunk momentarily.

        Show
        Todd Lipcon added a comment - Yes, I ran this test a few times and it looks like that failure only happens intermittently (though I was able to reproduce it once). I will commit this to trunk momentarily.
        Hide
        Tom White added a comment -

        Is it worth committing this as it stands and fixing the intermittent failure in a follow up JIRA?

        Show
        Tom White added a comment - Is it worth committing this as it stands and fixing the intermittent failure in a follow up JIRA?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473938/MAPREDUCE-2395.patch
        against trunk revision 1087098.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/153//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/153//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/153//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473938/MAPREDUCE-2395.patch against trunk revision 1087098. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/153//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/153//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/153//console This message is automatically generated.
        Hide
        Ramkumar Vadali added a comment -

        Yes, I saw that but could not reproduce it. Also, it is weird since this patch has only test code changes.

        Show
        Ramkumar Vadali added a comment - Yes, I saw that but could not reproduce it. Also, it is weird since this patch has only test code changes.
        Hide
        Todd Lipcon added a comment -

        Hi Ramkumar. It looks like one of the tests is failing:

        org.apache.hadoop.fs.ChecksumException: Checksum error: /blk_-5688804776476378478:of:/user/dhruba/raidtest/file2 at 3584 exp: 924592111 got: -781589875
        at org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:320)
        at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
        at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:225)
        at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:193)
        at org.apache.hadoop.hdfs.BlockReader.read(BlockReader.java:117)
        at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:477)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:528)
        at java.io.DataInputStream.read(DataInputStream.java:83)
        at org.apache.hadoop.hdfs.TestRaidDfs.validateFile(TestRaidDfs.java:402)
        at org.apache.hadoop.raid.TestBlockFixerDistConcurrency.testConcurrentJobs(TestBlockFixerDistConcurrency.java:161)

        Show
        Todd Lipcon added a comment - Hi Ramkumar. It looks like one of the tests is failing: org.apache.hadoop.fs.ChecksumException: Checksum error: /blk_-5688804776476378478:of:/user/dhruba/raidtest/file2 at 3584 exp: 924592111 got: -781589875 at org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:320) at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276) at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:225) at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:193) at org.apache.hadoop.hdfs.BlockReader.read(BlockReader.java:117) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:477) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:528) at java.io.DataInputStream.read(DataInputStream.java:83) at org.apache.hadoop.hdfs.TestRaidDfs.validateFile(TestRaidDfs.java:402) at org.apache.hadoop.raid.TestBlockFixerDistConcurrency.testConcurrentJobs(TestBlockFixerDistConcurrency.java:161)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473938/MAPREDUCE-2395.patch
        against trunk revision 1082617.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/141//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/141//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/141//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473938/MAPREDUCE-2395.patch against trunk revision 1082617. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/141//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/141//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/141//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Wow, that was fast! Let's see how the test-patch goes. +1 but will wait for Hudson

        Show
        Todd Lipcon added a comment - Wow, that was fast! Let's see how the test-patch goes. +1 but will wait for Hudson
        Hide
        Ramkumar Vadali added a comment -

        Breaks TestBlockFixer into several tests. The file TestBlockFixer.java now has tests that do not use a MiniMRCluster. The other TestBlockFixer*.java files have a few tests each that use MiniMRCluster.

        Show
        Ramkumar Vadali added a comment - Breaks TestBlockFixer into several tests. The file TestBlockFixer.java now has tests that do not use a MiniMRCluster. The other TestBlockFixer*.java files have a few tests each that use MiniMRCluster.
        Hide
        Todd Lipcon added a comment -

        With Tom's patch, the test runtime was 10 minutes on my box (with SSD).

        We should consider speeding this up or splitting it into multiple test classes, just in case build machines run a bit slower.

        Show
        Todd Lipcon added a comment - With Tom's patch, the test runtime was 10 minutes on my box (with SSD). We should consider speeding this up or splitting it into multiple test classes, just in case build machines run a bit slower.
        Hide
        Todd Lipcon added a comment -

        I ran this on my box and it looks like it's super slow due to MAPREDUCE-2392. It timed out after 15 minutes but spent a lot of its time waiting on TTs to shut down. I'm re-running now with Tom's patch from that JIRA applied for comparison.

        Show
        Todd Lipcon added a comment - I ran this on my box and it looks like it's super slow due to MAPREDUCE-2392 . It timed out after 15 minutes but spent a lot of its time waiting on TTs to shut down. I'm re-running now with Tom's patch from that JIRA applied for comparison.

          People

          • Assignee:
            Ramkumar Vadali
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development