Uploaded image for project: 'Falcon'
  1. Falcon
  2. FALCON-1312

Falcon post processing action should use Oozie prepared configuration

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.6, 0.6.1
    • 0.7
    • common
    • None

    Description

      Without this, in an environment with NN-HA or security setup, the postprocessing action will fail accdssing the filesystem

      Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice
      org.apache.oozie.action.hadoop.JavaMainException: java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice
      at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
      at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
      at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice
      at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)
      at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
      at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
      at org.apache.falcon.hadoop.HadoopClientFactory$1.run(HadoopClientFactory.java:200)
      at org.apache.falcon.hadoop.HadoopClientFactory$1.run(HadoopClientFactory.java:198)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      at org.apache.falcon.hadoop.HadoopClientFactory.createFileSystem(HadoopClientFactory.java:198)
      at org.apache.falcon.hadoop.HadoopClientFactory.createProxiedFileSystem(HadoopClientFactory.java:153)
      at org.apache.falcon.hadoop.HadoopClientFactory.createProxiedFileSystem(HadoopClientFactory.java:145)
      at org.apache.falcon.workflow.WorkflowExecutionContext.serialize(WorkflowExecutionContext.java:313)
      at org.apache.falcon.workflow.WorkflowExecutionContext.serialize(WorkflowExecutionContext.java:299)
      at org.apache.falcon.workflow.FalconPostProcessing.run(FalconPostProcessing.java:49)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.falcon.workflow.FalconPostProcessing.main(FalconPostProcessing.java:39)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
      ... 15 more
      Caused by: java.net.UnknownHostException: nameservice
      ... 44 more

      In the above stack trace it is clear that we are using non-HA proxy because the configuration used does not have the HA information

      Attachments

        1. FALCON-1312.patch
          1 kB
          Venkat Ranganathan
        2. FALCON-1312.patch.2
          6 kB
          Venkat Ranganathan
        3. FALCON-1312-v3.patch
          6 kB
          Ajay Yadav

        Issue Links

          Activity

            People

              venkatnrangan Venkat Ranganathan
              venkatnrangan Venkat Ranganathan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Slack

                  Issue deployment