Pig
  1. Pig
  2. PIG-1154

local mode fails when hadoop config directory is specified in classpath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None

      Description

      In local mode, the hadoop configuration should not be taken from the classpath .

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        49d 4h 36m 1 Ankit Modi 02/Feb/10 23:28
        Patch Available Patch Available Resolved Resolved
        1d 1h 18m 1 Alan Gates 04/Feb/10 00:46
        Resolved Resolved Closed Closed
        99d 5h 59m 1 Daniel Dai 14/May/10 07:46
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Alan Gates made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Alan Gates added a comment -

        Patch checked in. Thanks Ankit.

        Show
        Alan Gates added a comment - Patch checked in. Thanks Ankit.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12434613/pig_1154.patch
        against trunk revision 905377.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/188/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/188/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/188/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12434613/pig_1154.patch against trunk revision 905377. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/188/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/188/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/188/console This message is automatically generated.
        Ankit Modi made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ankit Modi added a comment -

        This patch only affects only Local Mode in pig.

        Show
        Ankit Modi added a comment - This patch only affects only Local Mode in pig.
        Ankit Modi made changes -
        Attachment pig_1154.patch [ 12434613 ]
        Hide
        Ankit Modi added a comment -

        Patch according to comments mentioned above.

        Show
        Ankit Modi added a comment - Patch according to comments mentioned above.
        Hide
        Ankit Modi added a comment -

        It will provide warning whenever the files are encountered in Local Mode.

        On top of that it will exit with error if mapred.system.dir is different from the default one and it does not exist.

        Show
        Ankit Modi added a comment - It will provide warning whenever the files are encountered in Local Mode. On top of that it will exit with error if mapred.system.dir is different from the default one and it does not exist.
        Hide
        Alan Gates added a comment -

        I'm assuming you mean it will provide a warning and error out when the user is in local mode and mapred-site.xml or hdfs-site.xml are found in the classpath?

        Show
        Alan Gates added a comment - I'm assuming you mean it will provide a warning and error out when the user is in local mode and mapred-site.xml or hdfs-site.xml are found in the classpath?
        Hide
        Ankit Modi added a comment -

        It looks like the problem is caused by overwritten value of mapred.system.dir from mapred-default.xml and the path mentioned above "/mapredsystem/hadoop/mapredsystem/" may not exist.

        This cannot be solved in local mode as it is not possible to change classpath at runtime.

        I'll provide a patch which would

        • Provide a warning whenever classpath contains mapred-site.xml or hdfs-site.xml.
        • It'll exit pig with an error message if above case is encountered.
        Show
        Ankit Modi added a comment - It looks like the problem is caused by overwritten value of mapred.system.dir from mapred-default.xml and the path mentioned above "/mapredsystem/hadoop/mapredsystem/" may not exist. This cannot be solved in local mode as it is not possible to change classpath at runtime. I'll provide a patch which would Provide a warning whenever classpath contains mapred-site.xml or hdfs-site.xml. It'll exit pig with an error message if above case is encountered.
        Olga Natkovich made changes -
        Field Original Value New Value
        Assignee Ankit Modi [ ankit.modi ]
        Hide
        Thejas M Nair added a comment -

        It is difficult to identify the problem from the exception thrown -

        Backend error message during job submission
        -------------------------------------------
        org.apache.hadoop.util.Shell$ExitCodeException: chmod: cannot access `/mapredsystem/hadoop/mapredsystem/job_local_0001': No such file or directory

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:245)
        at org.apache.hadoop.util.Shell.run(Shell.java:172)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:365)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:451)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:434)
        at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:481)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:473)
        at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:280)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:261)
        at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:573)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
        at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
        at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
        at java.lang.Thread.run(Thread.java:619)

        Error before Pig is launched
        ----------------------------
        ERROR 1951: Caught exception

        org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1951: Caught exception
        at org.apache.pig.PigServer.genStorePlanAndExecute(PigServer.java:594)
        at org.apache.pig.PigServer.openInputStream(PigServer.java:511)
        at org.apache.pig.impl.logicalLayer.SQLParser.QueryProcessor.executeAndWriteToSTDOUT(QueryProcessor.java:251)
        at org.apache.pig.impl.logicalLayer.SQLParser.QueryProcessor.execute(QueryProcessor.java:203)
        at org.apache.pig.SQLDriver.execute(SQLDriver.java:56)
        at org.apache.pig.Main.processSQL(Main.java:465)
        at org.apache.pig.Main.main(Main.java:400)
        Caused by: java.io.IOException: Job terminated with anomalous status FAILED
        at org.apache.pig.PigServer.genStorePlanAndExecute(PigServer.java:584)
        ... 6 more
        ================================================================================

        Show
        Thejas M Nair added a comment - It is difficult to identify the problem from the exception thrown - Backend error message during job submission ------------------------------------------- org.apache.hadoop.util.Shell$ExitCodeException: chmod: cannot access `/mapredsystem/hadoop/mapredsystem/job_local_0001': No such file or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:245) at org.apache.hadoop.util.Shell.run(Shell.java:172) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:365) at org.apache.hadoop.util.Shell.execCommand(Shell.java:451) at org.apache.hadoop.util.Shell.execCommand(Shell.java:434) at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:481) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:473) at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:280) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:261) at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:573) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) Error before Pig is launched ---------------------------- ERROR 1951: Caught exception org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1951: Caught exception at org.apache.pig.PigServer.genStorePlanAndExecute(PigServer.java:594) at org.apache.pig.PigServer.openInputStream(PigServer.java:511) at org.apache.pig.impl.logicalLayer.SQLParser.QueryProcessor.executeAndWriteToSTDOUT(QueryProcessor.java:251) at org.apache.pig.impl.logicalLayer.SQLParser.QueryProcessor.execute(QueryProcessor.java:203) at org.apache.pig.SQLDriver.execute(SQLDriver.java:56) at org.apache.pig.Main.processSQL(Main.java:465) at org.apache.pig.Main.main(Main.java:400) Caused by: java.io.IOException: Job terminated with anomalous status FAILED at org.apache.pig.PigServer.genStorePlanAndExecute(PigServer.java:584) ... 6 more ================================================================================
        Thejas M Nair created issue -

          People

          • Assignee:
            Ankit Modi
            Reporter:
            Thejas M Nair
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development