Description
Many customer who uses s3 file system as secondary one experiences the following error when Oozie tries to submit the Yarn application:
2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 - SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[] APP[demo-wf] JOB[0000001-190423141707256-oozie-oozi-W] ACTION[0000001-190423141707256-oozie-oozi-W@streaming-node] Error starting action [streaming-node]. ErrorType [ERROR], ErrorCode [UnsupportedOperationException], Message [UnsupportedOperationException: Accessing local file system is not allowed] org.apache.oozie.action.ActionExecutorException: UnsupportedOperationException: Accessing local file system is not allowed at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446) at org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092) at org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234) at org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41) at org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30) at org.apache.oozie.command.XCommand.call(XCommand.java:287) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.UnsupportedOperationException: Accessing local file system is not allowed at org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48) at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461) at org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200) at org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572) at org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811) at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190) at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.<init>(S3ABlockOutputStream.java:168) at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026) at org.apache.oozie.action.hadoop.LauncherMapperHelper.setupLauncherInfo(LauncherMapperHelper.java:156) at org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1033) ... 12 more
Does anybody has any idea how could we modify the RawLocalFileSystem class to make it a bit less strict?
Thank you for the repro wf to Soumitra Sulav.
Attachments
Attachments
Issue Links
- causes
-
OOZIE-3546 Oozie prints out warning messages when non-s3 file system custom properties are set
- Open
-
OOZIE-3547 Oozie does not allow comma in custom file system properties
- Open
- links to