Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is MapReduce part of HADOOP-6332

      1. MAPREDUCE-1774.patch
        202 kB
        Konstantin Boudnik
      2. MAPREDUCE-1774.patch
        234 kB
        Konstantin Boudnik
      3. MAPREDUCE-1774.patch
        235 kB
        Konstantin Boudnik
      4. MAPREDUCE-1774.patch
        239 kB
        Konstantin Boudnik
      5. MAPREDUCE-1774.patch
        242 kB
        Konstantin Boudnik
      6. MAPREDUCE-1774.patch
        249 kB
        Konstantin Boudnik
      7. MAPREDUCE-1774.patch
        255 kB
        Konstantin Boudnik
      8. MAPREDUCE-1774.patch
        255 kB
        Konstantin Boudnik
      9. MAPREDUCE-1774.patch
        249 kB
        Konstantin Boudnik
      10. MAPREDUCE-1774.patch
        249 kB
        Konstantin Boudnik
      11. MAPREDUCE-1774.patch
        250 kB
        Konstantin Boudnik
      12. MAPREDUCE-1774.patch
        250 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Same as for HDFS-1134: initial version of source code forward patch. No build changes are included yet.

          Show
          Konstantin Boudnik added a comment - Same as for HDFS-1134 : initial version of source code forward patch. No build changes are included yet.
          Hide
          Konstantin Boudnik added a comment -

          This patch version has all correct build modifications in please. However, because of the code changes between MR in 0.20 and in the trunk aspects aren't binding anymore and this needs to be fixed.

          Show
          Konstantin Boudnik added a comment - This patch version has all correct build modifications in please. However, because of the code changes between MR in 0.20 and in the trunk aspects aren't binding anymore and this needs to be fixed.
          Hide
          Konstantin Boudnik added a comment -

          Workaround for test-patch version setting. Fixing too wide a mask to include aspect files.

          Show
          Konstantin Boudnik added a comment - Workaround for test-patch version setting. Fixing too wide a mask to include aspect files.
          Hide
          Konstantin Boudnik added a comment -

          Changes similar to latest HDFS ones: moving tests to src/test/system/test for better build handling; changing Common artifacts from core to common, etc.
          Can't fully verify the patch at the moment because aspects are still broken and need more work there.

          Show
          Konstantin Boudnik added a comment - Changes similar to latest HDFS ones: moving tests to src/test/system/test for better build handling; changing Common artifacts from core to common , etc. Can't fully verify the patch at the moment because aspects are still broken and need more work there.
          Hide
          Konstantin Boudnik added a comment -

          Merging in Sreekanth's modifications of aspects addressing changes in 0.22 MR API.
          Also, rename of core to common was lost somehow, so I had to re-merge this particular modification once more.
          At the moment all code is being properly woven, tests are mostly compiled except for TestFileOwner which fails with follow message:

              [javac] .../H0.22/git/mapreduce/src/test/system/test/org/apache/hadoop/mapred/TestFileOwner.java:183: cannot find symbol
              [javac] symbol  : variable TASKJARDIR
              [javac] location: class org.apache.hadoop.mapred.TaskTracker
              [javac]           if (filename.equals(TaskTracker.TASKJARDIR)) {
              [javac]                                          ^
          

          However, the brief check shows that the symbol is present in the woven class file.

          Show
          Konstantin Boudnik added a comment - Merging in Sreekanth's modifications of aspects addressing changes in 0.22 MR API. Also, rename of core to common was lost somehow, so I had to re-merge this particular modification once more. At the moment all code is being properly woven, tests are mostly compiled except for TestFileOwner which fails with follow message: [javac] .../H0.22/git/mapreduce/src/test/system/test/org/apache/hadoop/mapred/TestFileOwner.java:183: cannot find symbol [javac] symbol : variable TASKJARDIR [javac] location: class org.apache.hadoop.mapred.TaskTracker [javac] if (filename.equals(TaskTracker.TASKJARDIR)) { [javac] ^ However, the brief check shows that the symbol is present in the woven class file.
          Hide
          Konstantin Boudnik added a comment -

          Two missed classes for testjar; system-test.conf is renamed to {{system-test-mapred.conf

          Show
          Konstantin Boudnik added a comment - Two missed classes for testjar ; system-test.conf is renamed to {{ system-test-mapred.conf
          Hide
          Konstantin Boudnik added a comment -

          This version of the patch has all issues with the build resolved. Template for POM files are also created and instrumented build is assembled properly.

          The only issue I see at the moment that tests can not connect to 0.20 cluster. It seems like the changes in the aspects couldn't cause that, so there should be another reason for this. Will keep looking and/or try to get together and start 0.22 cluster to make sure the this is a real problem.

          Patch is pretty much ready for verification.

          Show
          Konstantin Boudnik added a comment - This version of the patch has all issues with the build resolved. Template for POM files are also created and instrumented build is assembled properly. The only issue I see at the moment that tests can not connect to 0.20 cluster. It seems like the changes in the aspects couldn't cause that, so there should be another reason for this. Will keep looking and/or try to get together and start 0.22 cluster to make sure the this is a real problem. Patch is pretty much ready for verification.
          Hide
          Konstantin Boudnik added a comment -

          Let's check.

          Show
          Konstantin Boudnik added a comment - Let's check.
          Hide
          Konstantin Boudnik added a comment -

          Coupla more javadoc and import corrections.

          Show
          Konstantin Boudnik added a comment - Coupla more javadoc and import corrections.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446937/MAPREDUCE-1774.patch
          against trunk revision 953879.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 105 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 2226 javac compiler warnings (more than the trunk's current 2219 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 0 warnings).

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446937/MAPREDUCE-1774.patch against trunk revision 953879. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 105 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2226 javac compiler warnings (more than the trunk's current 2219 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/238/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -
          • This patch addresses audit warnings caused by missing Apache license boiler plate in a couple of places.
          • Javac warnings are caused by using deprecated JobConf and JobContext in two new classes from testjar package. While this is a valid issue I am not sure if it has to fought considering 2K+ of similar warnings all over the MR code.
          • Core tests failures are old: they are around for at least 6 days and this patch hasn't cause any ones
            Contrib test failure seems irrelevant (a Mumak testcase TestSimulatorDeterministicReplay timing out for over 10 days).
          Show
          Konstantin Boudnik added a comment - This patch addresses audit warnings caused by missing Apache license boiler plate in a couple of places. Javac warnings are caused by using deprecated JobConf and JobContext in two new classes from testjar package. While this is a valid issue I am not sure if it has to fought considering 2K+ of similar warnings all over the MR code. Core tests failures are old: they are around for at least 6 days and this patch hasn't cause any ones Contrib test failure seems irrelevant (a Mumak testcase TestSimulatorDeterministicReplay timing out for over 10 days).
          Hide
          Konstantin Boudnik added a comment -

          Includes Sreekanth fix for the connectivity problem. It has been caused by property names change between MR 0.20 and 0.22

          • mapred.job.tracker -> mapreduce.jobtracker.address
          • mapred.task.tracker.report.address ->mapreduce.tasktracker.report.address
            But because MR lacks a container similar to DFSConfigKeys there's no way to track such change by Java means. Except by knowing by heart ;(

          Tests are running against a real 0.22 cluster now! Some issues are addressed by HADOOP-6828 (another neat change). A couple more of issues will addressed in the next iteration. The patch is almost ready at the moment!

          Show
          Konstantin Boudnik added a comment - Includes Sreekanth fix for the connectivity problem. It has been caused by property names change between MR 0.20 and 0.22 mapred.job.tracker -> mapreduce.jobtracker.address mapred.task.tracker.report.address ->mapreduce.tasktracker.report.address But because MR lacks a container similar to DFSConfigKeys there's no way to track such change by Java means. Except by knowing by heart ;( Tests are running against a real 0.22 cluster now! Some issues are addressed by HADOOP-6828 (another neat change). A couple more of issues will addressed in the next iteration. The patch is almost ready at the moment!
          Hide
          Konstantin Boudnik added a comment -

          With another Sreekanth fix jobhistory is retrieved properly now.
          Tests are running. The only issue I am seeing at the moment is incorrect use of daemon scripts which needs to be addressed separately (HADOOP-6829)

          I think the patch is ready for the commit as the MR part of the framework works. Some of the test issues should be addressed separate. Longer this patch stays out higher the changes it will get stale.

          Show
          Konstantin Boudnik added a comment - With another Sreekanth fix jobhistory is retrieved properly now. Tests are running. The only issue I am seeing at the moment is incorrect use of daemon scripts which needs to be addressed separately ( HADOOP-6829 ) I think the patch is ready for the commit as the MR part of the framework works. Some of the test issues should be addressed separate. Longer this patch stays out higher the changes it will get stale.
          Hide
          Konstantin Boudnik added a comment -

          Let's see once more.

          Show
          Konstantin Boudnik added a comment - Let's see once more.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12447371/MAPREDUCE-1774.patch
          against trunk revision 955543.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 103 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 2226 javac compiler warnings (more than the trunk's current 2219 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/251/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/251/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/251/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/251/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447371/MAPREDUCE-1774.patch against trunk revision 955543. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 103 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2226 javac compiler warnings (more than the trunk's current 2219 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/251/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/251/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/251/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/251/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Test failure is unrelated. The number of warnings is increasing because of the some deprecated APIs usage. They are all used over the place and perhaps their elimination should be a matter of separate JIRA.

          Show
          Konstantin Boudnik added a comment - Test failure is unrelated. The number of warnings is increasing because of the some deprecated APIs usage. They are all used over the place and perhaps their elimination should be a matter of separate JIRA.
          Hide
          Konstantin Boudnik added a comment -

          Same as before but all string literals are replaced with proper named constants from container classes.

          Show
          Konstantin Boudnik added a comment - Same as before but all string literals are replaced with proper named constants from container classes.
          Hide
          Sharad Agarwal added a comment -

          Retrying Hudson

          Show
          Sharad Agarwal added a comment - Retrying Hudson
          Hide
          Sharad Agarwal added a comment -

          Tested binary-system, jar-system and jar-test-system. Worked fine.
          Tried running 'test-system' against a real deployed cluster. The test client could contact the cluster but some of the test cases are failing for me. The framework is working fine. I am ok fixing testcases in subsequent issues. +1 on committing this.

          Show
          Sharad Agarwal added a comment - Tested binary-system, jar-system and jar-test-system. Worked fine. Tried running 'test-system' against a real deployed cluster. The test client could contact the cluster but some of the test cases are failing for me. The framework is working fine. I am ok fixing testcases in subsequent issues. +1 on committing this.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12447409/MAPREDUCE-1774.patch
          against trunk revision 956335.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 103 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 2226 javac compiler warnings (more than the trunk's current 2219 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/580/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/580/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/580/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/580/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447409/MAPREDUCE-1774.patch against trunk revision 956335. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 103 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2226 javac compiler warnings (more than the trunk's current 2219 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/580/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/580/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/580/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/580/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          I have committed this both to trunk and 0.21 branch. Thanks Sharad, Sreekanth, and everybody else!

          Show
          Konstantin Boudnik added a comment - I have committed this both to trunk and 0.21 branch. Thanks Sharad, Sreekanth, and everybody else!
          Hide
          Konstantin Boudnik added a comment -

          This is going to be closed as soon as the subtasks are fixed.

          Show
          Konstantin Boudnik added a comment - This is going to be closed as soon as the subtasks are fixed.
          Hide
          Konstantin Boudnik added a comment -

          This has been committed and all subtasks are resolved. Closing it as fixed.

          Show
          Konstantin Boudnik added a comment - This has been committed and all subtasks are resolved. Closing it as fixed.

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development