Hadoop Common
  1. Hadoop Common
  2. HADOOP-3743

-libjars, -files and -archives options do not work with 0.18

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I am not sure how to get -libjars and -files working with 0.18. I tried all the options but cannot get it runnning. I am filing this as a blocker until we find out that its not broken. – in that case we need to update the docs with an example to say how it works.

      1. patch-3743.txt
        13 kB
        Amareshwari Sriramadasu
      2. patch-3743.txt
        13 kB
        Amareshwari Sriramadasu
      3. patch-3743.txt
        13 kB
        Amareshwari Sriramadasu
      4. patch-3743.txt
        13 kB
        Amareshwari Sriramadasu
      5. patch-3743.txt
        1 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Mahadev konar added a comment -

          The problem seems to be that

          1) Runjar does nto implement tools.
          2) also if runjar implements tools then the parsing done by the generic opptions parser will be lost when a user does new JobConf() --(unless the user implements his/hers job as a Tools which breaks backwards compatibility and would break all the user applications). The problem seems to be that the libjars and other parsed options needs to be available to the jobclient when jobclient.runJob() is inovoked.

          I do not have any suggestions right now. I will update the jira if I have any suggestions regarding this.

          Show
          Mahadev konar added a comment - The problem seems to be that 1) Runjar does nto implement tools. 2) also if runjar implements tools then the parsing done by the generic opptions parser will be lost when a user does new JobConf() --(unless the user implements his/hers job as a Tools which breaks backwards compatibility and would break all the user applications). The problem seems to be that the libjars and other parsed options needs to be available to the jobclient when jobclient.runJob() is inovoked. I do not have any suggestions right now. I will update the jira if I have any suggestions regarding this.
          Hide
          Amareshwari Sriramadasu added a comment -

          Now (after HADOOP-3417), the -libjars, -files and -archives options are Generic command line options. So, they should be passed to the ToolRunner.
          For example, wordcount should be run as :

          hadoop jar build/hadoop-0.18.0-dev-examples.jar wordcount -libjars mylib.jar -files cache-file.txt input output
          

          And I see no documentation available for the options in mapred-tutorial. The tutorial has to be updated with the usage.

          Show
          Amareshwari Sriramadasu added a comment - Now (after HADOOP-3417 ), the -libjars, -files and -archives options are Generic command line options. So, they should be passed to the ToolRunner. For example, wordcount should be run as : hadoop jar build/hadoop-0.18.0-dev-examples.jar wordcount -libjars mylib.jar -files cache-file.txt input output And I see no documentation available for the options in mapred-tutorial. The tutorial has to be updated with the usage.
          Hide
          Amareshwari Sriramadasu added a comment -

          Here is a patch adding the usage documentation in mapred_tutorial for -libjars, -files and -archives.
          And there is a link from tutorial to commands_manual.

          Show
          Amareshwari Sriramadasu added a comment - Here is a patch adding the usage documentation in mapred_tutorial for -libjars, -files and -archives. And there is a link from tutorial to commands_manual.
          Hide
          Mahadev konar added a comment -

          please read my previous comment. The above example that you attcahed is working because it implemenets tools. this will nto work with user applications that have been running without implementing tools and i think most of them dont implement tools. 0.17 works with users mapreducde job not implementing tools so we should try and make this work in 0.18 as well.

          Show
          Mahadev konar added a comment - please read my previous comment. The above example that you attcahed is working because it implemenets tools. this will nto work with user applications that have been running without implementing tools and i think most of them dont implement tools. 0.17 works with users mapreducde job not implementing tools so we should try and make this work in 0.18 as well.
          Hide
          Milind Bhandarkar added a comment -

          I agree with Mahadev. User's main class should not have to implement Tool in order to run. The lesser hard requirements the better.

          Show
          Milind Bhandarkar added a comment - I agree with Mahadev. User's main class should not have to implement Tool in order to run. The lesser hard requirements the better.
          Hide
          Amareshwari Sriramadasu added a comment -

          One solution I could see is to make org.apache.hadoop.streaming.StreamJob and org.apache.hadoop.mapred.pipes.Submitter implement ToolRunner

          Show
          Amareshwari Sriramadasu added a comment - One solution I could see is to make org.apache.hadoop.streaming.StreamJob and org.apache.hadoop.mapred.pipes.Submitter implement ToolRunner
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12385837/patch-3743.txt
          against trunk revision 676069.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2851/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2851/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2851/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2851/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12385837/patch-3743.txt against trunk revision 676069. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2851/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2851/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2851/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2851/console This message is automatically generated.
          Hide
          Devaraj Das added a comment - - edited

          I think we should have users move to the style of the hadoop examples programs (e.g. o.a.h.e.RandomWriter) which implements Tool and all that. I can't see any other way other than hacking a solution to work around this constraint and HADOOP-3417 had actually removed that hack. I think long term it makes sense to have the users migrate to implement apps using Tool, etc. That includes the framework client classes o.a.h.s.StreamJob and o.a.h.m.p.Submitter as Amareshwari pointed out.

          I understand that we have broken 0.17 java apps that use -files, -achives, and -libjars arguments and don't implement Tool, but unless we revert HADOOP-3417, I can't see how we can provide this functionality.

          Till the time user apps are moved to implementing Tool, with 0.18 and beyond, users could set the values for the options tmpfiles, tmparchives and tmpjars in the configuration they use to submit jobs (that's what the three arguments, -files, -archives and -jars, do).

          Thoughts?

          Show
          Devaraj Das added a comment - - edited I think we should have users move to the style of the hadoop examples programs (e.g. o.a.h.e.RandomWriter) which implements Tool and all that. I can't see any other way other than hacking a solution to work around this constraint and HADOOP-3417 had actually removed that hack. I think long term it makes sense to have the users migrate to implement apps using Tool, etc. That includes the framework client classes o.a.h.s.StreamJob and o.a.h.m.p.Submitter as Amareshwari pointed out. I understand that we have broken 0.17 java apps that use -files, -achives, and -libjars arguments and don't implement Tool, but unless we revert HADOOP-3417 , I can't see how we can provide this functionality. Till the time user apps are moved to implementing Tool, with 0.18 and beyond, users could set the values for the options tmpfiles, tmparchives and tmpjars in the configuration they use to submit jobs (that's what the three arguments, -files, -archives and -jars, do). Thoughts?
          Hide
          Runping Qi added a comment -

          I think Devaraj's suggestion should work:
          Change StreamJob to implement Tools,
          and ask normal user jobs to explicitly add -Dn=v for libjars and archives.
          However, we should choose more informative names such as mapred.use.additional.jars and mapred.use.archive

          Show
          Runping Qi added a comment - I think Devaraj's suggestion should work: Change StreamJob to implement Tools, and ask normal user jobs to explicitly add -Dn=v for libjars and archives. However, we should choose more informative names such as mapred.use.additional.jars and mapred.use.archive
          Hide
          Devaraj Das added a comment -

          Had an offline discussion with Sameer on the aspect of backward compatibility and here is a proposal to address that:
          1) Put back JobShell and let JobShell forward the arguments to GenericOptionsParser
          2) Protect the access to the static jobconf object
          So this is mostly HADOOP-3417 reverted + the above two changes implemented.

          Show
          Devaraj Das added a comment - Had an offline discussion with Sameer on the aspect of backward compatibility and here is a proposal to address that: 1) Put back JobShell and let JobShell forward the arguments to GenericOptionsParser 2) Protect the access to the static jobconf object So this is mostly HADOOP-3417 reverted + the above two changes implemented.
          Hide
          Owen O'Malley added a comment -

          I think Devaraj's and Sameer's plan is a big step backwards. We need to have the users using GenericOptionsParser (http://hadoop.apache.org/core/docs/r0.17.1/api/org/apache/hadoop/util/GenericOptionsParser.html), regardless of whether they use the Tool/ToolRunner interface to do it. The proper place to add options that are generic for Hadoop applications is in GenericOptionsParser, not in JobShell. I had started a patch for HADOOP-3676 that moves Streaming and Pipes over to use the GenericOptionsParser, which is a better way to go.

          Show
          Owen O'Malley added a comment - I think Devaraj's and Sameer's plan is a big step backwards. We need to have the users using GenericOptionsParser ( http://hadoop.apache.org/core/docs/r0.17.1/api/org/apache/hadoop/util/GenericOptionsParser.html ), regardless of whether they use the Tool/ToolRunner interface to do it. The proper place to add options that are generic for Hadoop applications is in GenericOptionsParser, not in JobShell. I had started a patch for HADOOP-3676 that moves Streaming and Pipes over to use the GenericOptionsParser, which is a better way to go.
          Hide
          Sameer Paranjpye added a comment -

          I don't understand. Why is providing back compatibility a step backwards? I'm not arguing against the use of GenericOptionsParser, I'm arguing that we should invoke it for users that don't implement Tools via JobShell.

          Show
          Sameer Paranjpye added a comment - I don't understand. Why is providing back compatibility a step backwards? I'm not arguing against the use of GenericOptionsParser, I'm arguing that we should invoke it for users that don't implement Tools via JobShell.
          Hide
          Owen O'Malley added a comment -

          Ok, I propose the following:

          1. We put back JobShell for 0.18, but we make it pass the original arguments strings to runJar.
          2. We leave the -files, -libjars, and -archives int the GenericOptionsParser.
          3. In JobClient.submitJob we warn to log4j that the application should use GenericOptionParser if the values of the attributes for tmpfiles, tmplibjars, and tmparchives in the submitted JobConf and static JobConf are different.

          Thoughts?

          Show
          Owen O'Malley added a comment - Ok, I propose the following: 1. We put back JobShell for 0.18, but we make it pass the original arguments strings to runJar. 2. We leave the -files, -libjars, and -archives int the GenericOptionsParser. 3. In JobClient.submitJob we warn to log4j that the application should use GenericOptionParser if the values of the attributes for tmpfiles, tmplibjars, and tmparchives in the submitted JobConf and static JobConf are different. Thoughts?
          Hide
          Mahadev konar added a comment -

          seems fine to me.. also can we print a warning message if the main class does nto implment Tools? that way we can get rid of the static jobconf in 0.19.

          Show
          Mahadev konar added a comment - seems fine to me.. also can we print a warning message if the main class does nto implment Tools? that way we can get rid of the static jobconf in 0.19.
          Hide
          Owen O'Malley added a comment -

          Mahadev, that was my goal. Actually, I guess we could go one step farther and add a new attribute "mapred.used.genericoptionparser" that is set to true by the generic option parser. Then submitJob can print the warning if they use a JobConf that isn't from the generic option parser. (Tool/ToolRuner use the generic option parser internally.)

          Show
          Owen O'Malley added a comment - Mahadev, that was my goal. Actually, I guess we could go one step farther and add a new attribute "mapred.used.genericoptionparser" that is set to true by the generic option parser. Then submitJob can print the warning if they use a JobConf that isn't from the generic option parser. (Tool/ToolRuner use the generic option parser internally.)
          Hide
          Amareshwari Sriramadasu added a comment -

          Here is a patch doing the following:

          1. Adds org.apache.hadoop.mapred.JobShell and org.apache.hadoop.mapred.TestJobShell
          2. Adds the static Configuration in JobClient
          3. Parsing in JobShell is removed as is not necessary. Because JobShell implements Tool and ToolRunner does the parsing for it. So, JobShell is used to set the static config for JobClient for the applications that doesnt implement ToolRunner.

          but we make it pass the original arguments strings to runJar.

          We cannot send all the remaining arguments to RunJar, because RunJar expects the first argument to be the MainClass if there is no MainClass in the manifest.

          A warning is printed in JobClient if mapred.used.genericoptionparser is not set to yes
          And also applications that implement ToolRunner, but use JobShell for -libjars or -files or -archives are warned.

          Added earlier usage documentation patch again.

          Show
          Amareshwari Sriramadasu added a comment - Here is a patch doing the following: 1. Adds org.apache.hadoop.mapred.JobShell and org.apache.hadoop.mapred.TestJobShell 2. Adds the static Configuration in JobClient 3. Parsing in JobShell is removed as is not necessary. Because JobShell implements Tool and ToolRunner does the parsing for it. So, JobShell is used to set the static config for JobClient for the applications that doesnt implement ToolRunner. but we make it pass the original arguments strings to runJar. We cannot send all the remaining arguments to RunJar, because RunJar expects the first argument to be the MainClass if there is no MainClass in the manifest. A warning is printed in JobClient if mapred.used.genericoptionparser is not set to yes And also applications that implement ToolRunner, but use JobShell for -libjars or -files or -archives are warned. Added earlier usage documentation patch again.
          Hide
          Amareshwari Sriramadasu added a comment -

          The patch applies to both trunk and branch 0.18

          Show
          Amareshwari Sriramadasu added a comment - The patch applies to both trunk and branch 0.18
          Hide
          Mahadev konar added a comment -

          review comments:

          1) the file Jobshell has all the parser and options – is it needed? since we are parsing with genericoptions parser? we can get rid of them.
          2) also since the generic options parser always does the parsing — in case of JobShell as well — the property conf.set("mapred.used.genericoptionsparser", "yes"); will always be set? right? In that case it will always print the warning always.

          3)
          Here is a problem –

          case 1)
          bin/hadoop jar -libjars <comma seperated jars> userjar

          case 2)

          bin/hadoop jar userjar -libjars <comma seperated jars>

          I am not sure if in both the cases JobShell tools will parse the -libjars option. If it does then the warning is no use since it will be printed even if the userjar implements tools. If JobShell does not parse -libjars in case 2) then we can print the warning in JobShell if the properties of libjars and others are set in JobShell.

          Show
          Mahadev konar added a comment - review comments: 1) the file Jobshell has all the parser and options – is it needed? since we are parsing with genericoptions parser? we can get rid of them. 2) also since the generic options parser always does the parsing — in case of JobShell as well — the property conf.set("mapred.used.genericoptionsparser", "yes"); will always be set? right? In that case it will always print the warning always. 3) Here is a problem – case 1) bin/hadoop jar -libjars <comma seperated jars> userjar case 2) bin/hadoop jar userjar -libjars <comma seperated jars> I am not sure if in both the cases JobShell tools will parse the -libjars option. If it does then the warning is no use since it will be printed even if the userjar implements tools. If JobShell does not parse -libjars in case 2) then we can print the warning in JobShell if the properties of libjars and others are set in JobShell.
          Hide
          Amareshwari Sriramadasu added a comment -

          Here is patch with:

          • Removed code for Parser and Options in JobShell.
          • TestJobShell is modified not to create temporary file (files_tmp) in the cwd

          also since the generic options parser always does the parsing — in case of JobShell as well — the property conf.set("mapred.used.genericoptionsparser", "yes"); will always be set? right?

          Yes. The static configuration has mapred.used.genericoptionsparser value set to yes.

          In that case it will always print the warning always.

          Warning will not be printed, because JobClient checks for the configuration property in JobConf submitted to the submitJob. And mapred.used.genericoptionsparser will be set in JobConf, submitted to the submitJob, only if the application is implementing Tool. Thus applications not implementing Tool will be warned to do the same.

          3)Here is a problem -

          In case 1) -libjars is passed by JobShell. In case 2) -libjars is parsed by application.

          If JobShell does not parse -libjars in case 2) then we can print the warning in JobShell if the properties of libjars and others are set in JobShell.

          No. Because the actual parsing is done by GenericOptionsParser.

          Show
          Amareshwari Sriramadasu added a comment - Here is patch with: Removed code for Parser and Options in JobShell. TestJobShell is modified not to create temporary file (files_tmp) in the cwd also since the generic options parser always does the parsing — in case of JobShell as well — the property conf.set("mapred.used.genericoptionsparser", "yes"); will always be set? right? Yes. The static configuration has mapred.used.genericoptionsparser value set to yes. In that case it will always print the warning always. Warning will not be printed, because JobClient checks for the configuration property in JobConf submitted to the submitJob . And mapred.used.genericoptionsparser will be set in JobConf, submitted to the submitJob , only if the application is implementing Tool. Thus applications not implementing Tool will be warned to do the same. 3)Here is a problem - In case 1) -libjars is passed by JobShell. In case 2) -libjars is parsed by application. If JobShell does not parse -libjars in case 2) then we can print the warning in JobShell if the properties of libjars and others are set in JobShell. No. Because the actual parsing is done by GenericOptionsParser.
          Hide
          Amareshwari Sriramadasu added a comment -

          changed mapred.used.genericoptionsparser to set boolean instead of 'yes/no'

          Show
          Amareshwari Sriramadasu added a comment - changed mapred.used.genericoptionsparser to set boolean instead of 'yes/no'
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12386135/patch-3743.txt
          against trunk revision 677127.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2872/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2872/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2872/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2872/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12386135/patch-3743.txt against trunk revision 677127. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2872/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2872/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2872/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2872/console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          its failing its own test case.

          Show
          Mahadev konar added a comment - its failing its own test case.
          Hide
          Amareshwari Sriramadasu added a comment -

          Test was failing saying files_tmp doesnt exist, because it was looking in wrong path. It was passing on my machine because the file was already there in the path. It was my mistake.

          Show
          Amareshwari Sriramadasu added a comment - Test was failing saying files_tmp doesnt exist, because it was looking in wrong path. It was passing on my machine because the file was already there in the path. It was my mistake.
          Hide
          Amareshwari Sriramadasu added a comment -

          Here is patch with the path fixed in the testcase.

          Show
          Amareshwari Sriramadasu added a comment - Here is patch with the path fixed in the testcase.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12386261/patch-3743.txt
          against trunk revision 677470.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2890/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2890/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2890/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2890/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12386261/patch-3743.txt against trunk revision 677470. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2890/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2890/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2890/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2890/console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          +1 .. looks good.

          Show
          Mahadev konar added a comment - +1 .. looks good.
          Hide
          Mahadev konar added a comment -

          i just committed this to trunk and 0.18. Thanks Amareshwari.

          Show
          Mahadev konar added a comment - i just committed this to trunk and 0.18. Thanks Amareshwari.
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/ )
          Hide
          Robert added a comment -

          I use Hadoop inside a larger system, and create JobConf instances programmatically, not via command line tools. With 0.18 I now have to set a magic flag, simply to silence the unrelated warning:

          job.setBoolean("mapred.used.genericoptionsparser", true);

          Wouldn't it be better to add this as a part of the JobConf API, so that it can be set or cleared as needed? This way client code wouldn't have to resort to sprinkling magic strings around.

          Show
          Robert added a comment - I use Hadoop inside a larger system, and create JobConf instances programmatically, not via command line tools. With 0.18 I now have to set a magic flag, simply to silence the unrelated warning: job.setBoolean("mapred.used.genericoptionsparser", true); Wouldn't it be better to add this as a part of the JobConf API, so that it can be set or cleared as needed? This way client code wouldn't have to resort to sprinkling magic strings around.
          Hide
          Amareshwari Sriramadasu added a comment -

          job.setBoolean("mapred.used.genericoptionsparser", true);

          This configuration is set by the framework to know whether the application has used GenericOptionsParser or not. I dont think we can expose this to set by users through JobConf API.

          Show
          Amareshwari Sriramadasu added a comment - job.setBoolean("mapred.used.genericoptionsparser", true); This configuration is set by the framework to know whether the application has used GenericOptionsParser or not. I dont think we can expose this to set by users through JobConf API.
          Hide
          Robert added a comment -

          My concern is: I'm not using command-line options at all, the JobConf is generated by my code. Should I be getting a warning that there is something wrong with command-line parsing?

          And if not, what's the best way to get rid of the warning?

          Show
          Robert added a comment - My concern is: I'm not using command-line options at all, the JobConf is generated by my code. Should I be getting a warning that there is something wrong with command-line parsing? And if not, what's the best way to get rid of the warning?
          Hide
          Benjamin Darfler added a comment -
          Show
          Benjamin Darfler added a comment - This is broken in 0.20, see https://issues.apache.org/jira/browse/MAPREDUCE-2056

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Mahadev konar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development