Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3499

Pig job fails when run in local mode with namenode HA(QJM)

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.10.0
    • Fix Version/s: None
    • Component/s: grunt, parser, tools
    • Labels:
      None

      Description

      when we run a pig script with namenode HA(QJM) we always get unknown host exception. The nameserviceID is being considered as host and pig job giving unknown host exception.

      I am working on this fix but want community to validate whether any bug reported similar to this. If not I will provide the fix as soon as possible.

      The pig script is as below:

      testTable = LOAD 'hdfs://kdvenkata/user/kd/test.csv'
      USING PigStorage(',')
      AS (col1:chararray, col2:chararray, col3:int);
      STORE testTable into '/tmp/test_pig_output';

      Exception:

      Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: kdvenkata

      at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:417)

      at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)

      at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)

      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:412)

      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:379)

      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:123)

      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2278)

      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)

      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2312)

      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2294)

      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:317)

      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:70)

      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:53)

      at org.apache.pig.builtin.JsonMetadata.findMetaFile(JsonMetadata.java:106)

      at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:188)

      at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:465)

      at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)

      at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:110)

      at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)

      at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)

      at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)

      at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)

      at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)

      at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)

      at org.apache.pig.PigServer$Graph.compile(PigServer.java:1626)

      at org.apache.pig.PigServer$Graph.compile(PigServer.java:1620)

      at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1343)

      at org.apache.pig.PigServer.storeEx(PigServer.java:960)

      at org.apache.pig.PigServer.store(PigServer.java:928)

      at org.apache.pig.PigServer.openIterator(PigServer.java:841)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kdvenkata venkata kamalnath
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: