[OOZIE-2246] CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.3.0
• Assignee: Satish Saley
• Reporter: Ryota Egashira
• Created: 21/May/15 23:36
• Updated: 02/Dec/16 21:04
• Resolved: 03/Mar/16 17:56

Description

when har file doesn't exist, FileSystem.createFileSystem in HadoopAccessorService throw exception, failing CoordActionInputCheck.
thus, even thought there are other data dependencies which already exist, it is not reflected on DB.
coordinator job cannot start until the har file becomes available anyway, and once available, this error doesn't happen, so basic functionality is fine, but it's misleading.

2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542 [pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] exception callable [coord_action_input], E1021: Coord Action Input Check Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har]
org.apache.oozie.command.CommandException: E1021: Coord Action Input Check Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har]
at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har]
at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
... 6 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har]
at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
... 10 more
Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har
at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)