Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2246

CoordinatorInputCheckCommand does not behave properly when har file is one of data dependency and doesn't exist

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.0
    • Component/s: None
    • Labels:
      None

      Description

      when har file doesn't exist, FileSystem.createFileSystem in HadoopAccessorService throw exception, failing CoordActionInputCheck.
      thus, even thought there are other data dependencies which already exist, it is not reflected on DB.
      coordinator job cannot start until the har file becomes available anyway, and once available, this error doesn't happen, so basic functionality is fine, but it's misleading.

      2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542 [pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] exception callable [coord_action_input], E1021: Coord Action Input Check Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har]
      org.apache.oozie.command.CommandException: E1021: Coord Action Input Check Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har]
      at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
      at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
      at org.apache.oozie.command.XCommand.call(XCommand.java:280)
      at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.io.IOException: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har]
      at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
      at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
      at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
      at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
      at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
      ... 6 more
      Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har]
      at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
      at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
      at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
      at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
      ... 10 more
      Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index file in har://aaaa:8020/data/2014031322/archive.har
      at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
      at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
      at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
      at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
      

        Attachments

        1. OOZIE-2246-v4.patch
          3 kB
          Satish Subhashrao Saley
        2. OOZIE-2246-v3.patch
          3 kB
          Ryota Egashira
        3. OOZIE-2246-v2.patch
          3 kB
          Ryota Egashira
        4. OOZIE-2246.patch
          4 kB
          Ryota Egashira

          Activity

            People

            • Assignee:
              satishsaley Satish Subhashrao Saley
              Reporter:
              egashira Ryota Egashira
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: