[PIG-4814] AvroStorage does not take namenode HA as part of schema file url

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 19/Feb/16 22:27
• Updated: 08/Jun/16 20:48
• Resolved: 20/Feb/16 00:40

Description

The following script fail:

in = LOAD 'recordsOfStringArrays.avro' USING AvroStorage('', '-d -f hdfs://ml-ha/user/aeckstein/notower-rw/streamRunwaysInputSchema.avsc');

Where ml-ha is the configured HA name for namenode.

The reason is AvroStorage creates a new Configuration object in the backend, which missing the namenode HA configuration in hdfs-site.xml:

<property>
  <name>dfs.client.failover.proxy.provider.ml-ha</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

It should use the configuration passed from frontend rather than construct a new one.