Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4695

Fix LocalRunner on trunk after MAPREDUCE-3223 broke it

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      MAPREDUCE-3223 removed mapreduce.cluster.local.dir property from mapred-default.xml (since NM local dirs are now used) but failed to counter that LocalJobRunner, etc. still use it.

      mr-3223.txt:-  <name>mapreduce.cluster.local.dir</name>
      mr-3223.txt--  <value>${hadoop.tmp.dir}/mapred/local</value>
      

      All local job tests have been failing since then.

      This JIRA is to reintroduce it or provide an equivalent new config for fixing it.

      1. MAPREDUCE-4695.patch
        1 kB
        Harsh J
      2. MAPREDUCE-4695.patch
        2 kB
        Harsh J

        Issue Links

          Activity

          Hide
          qwertymaniac Harsh J added a comment -

          Here is a fix patch (not for commit) reintroducing that parameter as before, with a dummy change in the jobclient package to enforce tests.

          When committing, I'll post another patch minus the dummy changes.

          Show
          qwertymaniac Harsh J added a comment - Here is a fix patch (not for commit) reintroducing that parameter as before, with a dummy change in the jobclient package to enforce tests. When committing, I'll post another patch minus the dummy changes.
          Hide
          qwertymaniac Harsh J added a comment -

          So far things seem to be passing at: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2895/console

          Will commit this fix (minus dummy forced changes) and open a new JIRA to see what we ought to do about that parameter name as it has "cluster" in it and only LocalRunner uses it anymore per my greps. But the property can stay and continue to be harmless for now.

          Running org.apache.hadoop.mapreduce.TestLocalRunner
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.904 sec
          
          Show
          qwertymaniac Harsh J added a comment - So far things seem to be passing at: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2895/console Will commit this fix (minus dummy forced changes) and open a new JIRA to see what we ought to do about that parameter name as it has "cluster" in it and only LocalRunner uses it anymore per my greps. But the property can stay and continue to be harmless for now. Running org.apache.hadoop.mapreduce.TestLocalRunner Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.904 sec
          Hide
          revans2 Robert Joseph Evans added a comment -

          The patch looks good. I don't think that ClientServiceDelegate.java needs to be changed at all.

          It would also be nice to add in a test so that changes like this don't break it again in the future, but I am not sure how easy that would be so I am OK without it.

          Show
          revans2 Robert Joseph Evans added a comment - The patch looks good. I don't think that ClientServiceDelegate.java needs to be changed at all. It would also be nice to add in a test so that changes like this don't break it again in the future, but I am not sure how easy that would be so I am OK without it.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12547227/MAPREDUCE-4695.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2895//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2895//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547227/MAPREDUCE-4695.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2895//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2895//console This message is automatically generated.
          Hide
          qwertymaniac Harsh J added a comment -

          The patch looks good. I don't think that ClientServiceDelegate.java needs to be changed at all.

          Thats my dummy change to trigger tests in jobclient (tests reside there, while default xml is in core, a problem we've yet to fix, will follow up on this afterwards). It won't go into the final patch.

          It would also be nice to add in a test so that changes like this don't break it again in the future, but I am not sure how easy that would be so I am OK without it.

          If we moved all tests to their right module we wouldn't have this issue (i.e. we won't miss it in pre commit builds).

          Thanks Robert, I'll commit this in (new patch minus jobclient package change).

          Show
          qwertymaniac Harsh J added a comment - The patch looks good. I don't think that ClientServiceDelegate.java needs to be changed at all. Thats my dummy change to trigger tests in jobclient (tests reside there, while default xml is in core, a problem we've yet to fix, will follow up on this afterwards). It won't go into the final patch. It would also be nice to add in a test so that changes like this don't break it again in the future, but I am not sure how easy that would be so I am OK without it. If we moved all tests to their right module we wouldn't have this issue (i.e. we won't miss it in pre commit builds). Thanks Robert, I'll commit this in (new patch minus jobclient package change).
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12547236/MAPREDUCE-4695.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2896//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2896//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547236/MAPREDUCE-4695.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2896//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2896//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2797 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2797/)
          MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2797 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2797/ ) MAPREDUCE-4695 . Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2860 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2860/)
          MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2860 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2860/ ) MAPREDUCE-4695 . Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2819 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2819/)
          MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2819 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2819/ ) MAPREDUCE-4695 . Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1183/)
          MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1183/ ) MAPREDUCE-4695 . Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1214 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1214/)
          MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1214 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1214/ ) MAPREDUCE-4695 . Fix LocalRunner on trunk after MAPREDUCE-3223 broke it. Contributed by Harsh J. (harsh) (Revision 1392446) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392446 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml

            People

            • Assignee:
              qwertymaniac Harsh J
              Reporter:
              qwertymaniac Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development