Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8089

cannot submit job from Eclipse plugin running on Windows

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: contrib/eclipse-plugin
    • Labels:
      None
    • Environment:

      Windows XP

      Description

      Built eclipse plugin for 1.0.0 on Windows and using it to submit a job, I got the following error:

      12/02/17 17:14:24 ERROR security.UserGroupInformation: PriviledgedActionException as:weilis cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-weilis\mapred\staging\weilis753422487\.staging to 0700
      Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-weilis\mapred\staging\weilis753422487\.staging to 0700
      at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682)
      at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
      at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
      at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
      at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
      at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
      at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
      at org.apache.hadoop.mapreduce.Job.submit(Job.java:465)
      at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:495)

      The job staging area local on Windows system and I don't think there is a way to set a local file's permission to 0700 anyway. Please have a look.

        Issue Links

          Activity

          Hide
          sureshms Suresh Srinivas added a comment -

          This issue is related to submitting a job to a hadoop cluster from a client submitting a job on windows machine.

          I have verified that the above work arounds solves the issue. Since there are other issue related to RawLocalFileSystem setpermission issues on windows, I am going to resolve this jira.

          Show
          sureshms Suresh Srinivas added a comment - This issue is related to submitting a job to a hadoop cluster from a client submitting a job on windows machine. I have verified that the above work arounds solves the issue. Since there are other issue related to RawLocalFileSystem setpermission issues on windows, I am going to resolve this jira.
          Hide
          wshao518 Weili Shao added a comment -

          I think I found the reason at high level, after I put a return to avoid that permission setting, the job started with a localjobrunner and the result was written to the remote hdfs, which confused me for a while, since I had no knowledge of localjobrunner before. Now I realize that it's really that the JobClient wasn't initiated properly from the plugin xml conf, so without any mapred.job.tracker settings, it just started the localjobrunner. After I added the code:
          Configuration conf = new Configuration();
          conf.set("mapred.job.tracker", "[server]:9001");

          It's working now. However. This configuration should be loaded from plugin's server configuration and it used to be working.

          Show
          wshao518 Weili Shao added a comment - I think I found the reason at high level, after I put a return to avoid that permission setting, the job started with a localjobrunner and the result was written to the remote hdfs, which confused me for a while, since I had no knowledge of localjobrunner before. Now I realize that it's really that the JobClient wasn't initiated properly from the plugin xml conf, so without any mapred.job.tracker settings, it just started the localjobrunner. After I added the code: Configuration conf = new Configuration(); conf.set("mapred.job.tracker", " [server] :9001"); It's working now. However. This configuration should be loaded from plugin's server configuration and it used to be working.

            People

            • Assignee:
              Unassigned
              Reporter:
              wshao518 Weili Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development