Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4814

AvroStorage does not take namenode HA as part of schema file url

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following script fail:

      in = LOAD 'recordsOfStringArrays.avro' USING AvroStorage('', '-d -f hdfs://ml-ha/user/aeckstein/notower-rw/streamRunwaysInputSchema.avsc');
      

      Where ml-ha is the configured HA name for namenode.

      The reason is AvroStorage creates a new Configuration object in the backend, which missing the namenode HA configuration in hdfs-site.xml:

      <property>
        <name>dfs.client.failover.proxy.provider.ml-ha</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
      </property>
      

      It should use the configuration passed from frontend rather than construct a new one.

        Attachments

        1. PIG-4814-1.patch
          0.9 kB
          Jianyong Dai

          Activity

            People

            • Assignee:
              daijy Jianyong Dai
              Reporter:
              daijy Jianyong Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: