Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2467

HDFS-1052 changes break the raid contrib module in MapReduce

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Raid contrib module requires changes to work with the federation changes made in HDFS-1052.

      1. MR-2467.3.patch
        43 kB
        Suresh Srinivas
      2. MR-2467.2.patch
        43 kB
        Suresh Srinivas
      3. MR-2467.1.patch
        39 kB
        Suresh Srinivas
      4. MR-2467.patch
        9 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #679 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/679/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #679 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/679/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #663 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/663/)
          MAPREDUCE-2467. HDFS-1052 changes break the raid contrib module in MapReduce. (suresh srinivas via mahadev)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #663 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/663/ ) MAPREDUCE-2467 . HDFS-1052 changes break the raid contrib module in MapReduce. (suresh srinivas via mahadev)
          Hide
          Mahadev konar added a comment -

          I just committed this to trunk. thanks suresh.

          Show
          Mahadev konar added a comment - I just committed this to trunk. thanks suresh.
          Hide
          Suresh Srinivas added a comment -

          Thanks Ramkumar for the feedback. Then I will go ahead and commit this patch. Separate issues can be filed to track the test failures.

          Show
          Suresh Srinivas added a comment - Thanks Ramkumar for the feedback. Then I will go ahead and commit this patch. Separate issues can be filed to track the test failures.
          Hide
          Ramkumar Vadali added a comment -

          Hi Suresh

          Sorry for the delay in responding. I think the test failures are unrelated
          1. testConcurrentJobs is failing because one file is not detected as corrupt and so block fixer does not fix it. This seems to be intermittent, since I tried running the test and it succeeded twice. I dont think this is related to federation. Perhaps we can track it separately?
          2. testFileBlockAndParityBlockMissingHar2 failed because of insufficient heap space when running a HAR job through LocalJobRunner. Again, unrelated to federation
          3. testJobQueues - Failed because of a timeout. Also, RAID changes cannot affect core mapred tests, so this must be unrelated.

          Show
          Ramkumar Vadali added a comment - Hi Suresh Sorry for the delay in responding. I think the test failures are unrelated 1. testConcurrentJobs is failing because one file is not detected as corrupt and so block fixer does not fix it. This seems to be intermittent, since I tried running the test and it succeeded twice. I dont think this is related to federation. Perhaps we can track it separately? 2. testFileBlockAndParityBlockMissingHar2 failed because of insufficient heap space when running a HAR job through LocalJobRunner. Again, unrelated to federation 3. testJobQueues - Failed because of a timeout. Also, RAID changes cannot affect core mapred tests, so this must be unrelated.
          Hide
          Suresh Srinivas added a comment -

          Following tests fail:

          1. org.apache.hadoop.raid.TestBlockFixerDistConcurrency.testConcurrentJobs
          2. org.apache.hadoop.raid.TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2
          3. org.apache.hadoop.mapred.TestJobQueueInformation.testJobQueues

          Ramkumar, can you please help with addressing this. I can work with you on this, if you need information about federation changes.

          Show
          Suresh Srinivas added a comment - Following tests fail: org.apache.hadoop.raid.TestBlockFixerDistConcurrency.testConcurrentJobs org.apache.hadoop.raid.TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2 org.apache.hadoop.mapred.TestJobQueueInformation.testJobQueues Ramkumar, can you please help with addressing this. I can work with you on this, if you need information about federation changes.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478325/MR-2467.3.patch
          against trunk revision 1099847.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.mapred.TestJobQueueInformation

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/221//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/221//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/221//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478325/MR-2467.3.patch against trunk revision 1099847. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestJobQueueInformation -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/221//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/221//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/221//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478325/MR-2467.3.patch
          against trunk revision 1099847.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/222//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/222//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/222//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478325/MR-2467.3.patch against trunk revision 1099847. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/222//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/222//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/222//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Failed test has lot of OutOfMemory errors. Increasing the junit test heap size from 512m to 1024m.

          Show
          Suresh Srinivas added a comment - Failed test has lot of OutOfMemory errors. Increasing the junit test heap size from 512m to 1024m.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478110/MR-2467.2.patch
          against trunk revision 1099590.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/218//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/218//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/218//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478110/MR-2467.2.patch against trunk revision 1099590. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/218//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/218//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/218//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Failed test TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2 seems to be related to some timing issue. When I run this test on my machine, it passes. Ramkumar, can you please comment on this?

          Here the failure due to exception:

          Failing for the past 1 build (Since Failed#215 )
          Took 41 sec.
          Error Message
          
          parity file not HARed after 40s
          
          Stacktrace
          
          java.io.IOException: parity file not HARed after 40s
          	at org.apache.hadoop.raid.TestRaidShellFsck.raidTestFiles(TestRaidShellFsck.java:281)
          	at org.apache.hadoop.raid.TestRaidShellFsck.setUp(TestRaidShellFsck.java:181)
          	at org.apache.hadoop.raid.TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2(TestRaidShellFsck.java:666)
          
          Show
          Suresh Srinivas added a comment - Failed test TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2 seems to be related to some timing issue. When I run this test on my machine, it passes. Ramkumar, can you please comment on this? Here the failure due to exception: Failing for the past 1 build (Since Failed#215 ) Took 41 sec. Error Message parity file not HARed after 40s Stacktrace java.io.IOException: parity file not HARed after 40s at org.apache.hadoop.raid.TestRaidShellFsck.raidTestFiles(TestRaidShellFsck.java:281) at org.apache.hadoop.raid.TestRaidShellFsck.setUp(TestRaidShellFsck.java:181) at org.apache.hadoop.raid.TestRaidShellFsck.testFileBlockAndParityBlockMissingHar2(TestRaidShellFsck.java:666)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478110/MR-2467.2.patch
          against trunk revision 1099590.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/215//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/215//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/215//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478110/MR-2467.2.patch against trunk revision 1099590. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/215//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/215//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/215//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478110/MR-2467.2.patch
          against trunk revision 1098853.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/206//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/206//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/206//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478110/MR-2467.2.patch against trunk revision 1098853. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/206//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/206//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/206//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Updated patch with change from MAPREDUCE-2465 reverted.

          Show
          Suresh Srinivas added a comment - Updated patch with change from MAPREDUCE-2465 reverted.
          Hide
          Ramkumar Vadali added a comment -

          Thanks for making the changes!

          Show
          Ramkumar Vadali added a comment - Thanks for making the changes!
          Hide
          Ramkumar Vadali added a comment -

          +1 looks good

          Show
          Ramkumar Vadali added a comment - +1 looks good
          Hide
          Eli Collins added a comment -

          +1 lgtm

          Show
          Eli Collins added a comment - +1 lgtm
          Hide
          Suresh Srinivas added a comment -

          Can Dhruba or Todd, can you please review this patch.

          Show
          Suresh Srinivas added a comment - Can Dhruba or Todd, can you please review this patch.
          Hide
          Suresh Srinivas added a comment -

          Changes:

          1. Made changes corresponding to federation changes.
          2. Cleaned up many warnings related to unused imports, dead code, unused methods.
          3. Added maxmemory parameter to junit test. Without this, tests launched under contrib directory fail with OutOfMemoryError.
          4. With federation, MiniDFSCluster to corrupt files and getting at a block file for a block is much cleaner. Changed the raid contrib to adapt this cleaner code.
          Show
          Suresh Srinivas added a comment - Changes: Made changes corresponding to federation changes. Cleaned up many warnings related to unused imports, dead code, unused methods. Added maxmemory parameter to junit test. Without this, tests launched under contrib directory fail with OutOfMemoryError. With federation, MiniDFSCluster to corrupt files and getting at a block file for a block is much cleaner. Changed the raid contrib to adapt this cleaner code.
          Hide
          Suresh Srinivas added a comment -

          Please hold off on the review, I still need some more changes.

          Show
          Suresh Srinivas added a comment - Please hold off on the review, I still need some more changes.
          Hide
          Suresh Srinivas added a comment -

          This patch fixes the build issues. I also cleaned up imports and dead code warnings.

          Show
          Suresh Srinivas added a comment - This patch fixes the build issues. I also cleaned up imports and dead code warnings.

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development