Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7175

JobSubmitter: validateFilePath() throws an exception because it requests a local FS unnecessarily

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 3.1.1, 2.9.2
    • Fix Version/s: None
    • Component/s: client
    • Labels:
      None

      Description

      After a security fix, we receive the following exception in Oozie if we want to use mapreduce.job.log4j-properties-file

      org.apache.oozie.action.ActionExecutorException: UnsupportedOperationException: Accessing local file system is not allowed
      at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
      at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1246)
      at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1424)
      at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
      at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
      at org.apache.oozie.command.XCommand.call(XCommand.java:286)
      at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
      at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.UnsupportedOperationException: Accessing local file system is not allowed
      at org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
      at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2816)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
      at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:358)
      at org.apache.hadoop.mapreduce.JobResourceUploader.validateFilePath(JobResourceUploader.java:303)
      at org.apache.hadoop.mapreduce.JobResourceUploader.copyLog4jPropertyFile(JobResourceUploader.java:248)
      at org.apache.hadoop.mapreduce.JobResourceUploader.addLog4jToDistributedCache(JobResourceUploader.java:223)
      at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:175)
      at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
      at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
      at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
      at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
      at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
      at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:578)
      at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
      at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:573)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:564)
      at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1231)
      ... 11 more

       

      Note that this happens even if the scheme is hdfs://. The solution is what mentioned in MAPREDUCE-6052: move

      FileSystem localFs = FileSystem.getLocal(conf);

      inside the if block.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pbacsko Peter Bacsko
                Reporter:
                pbacsko Peter Bacsko
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: