[HIVE-3645] RCFileWriter does not implement the right function to support Federation

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0, 0.10.0
• Fix Version/s: 0.11.0
• Component/s: Serializers/Deserializers
• Labels: namenode_federation

Environment

Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10

Release Note

HIVE-3645 shimmed getDefaultBlockSize/getDefaultReplication to make RCFiles work with federated namenode

• Assignee: Arup Malakar Assign to me
• Reporter: Viraj Bhat
• Created: 31/Oct/12 18:07
• Updated: 16/May/13 21:10
• Resolved: 27/Nov/12 17:51

Description

Create a table using Hive DDL

CREATE TABLE tmp_hcat_federated_numbers_part_1 (
  id       int,  
  intnum   int,
  floatnum float
)partitioned by (
  part1    string,
  part2    string
)
STORED AS rcfile
LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1';

Populate it using Pig:

A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader();
B = filter A by id <=  500;
C = foreach B generate (int)id, (int)intnum, (float)floatnum;
store C into
        'default.tmp_hcat_federated_numbers_part_1'
        using org.apache.hcatalog.pig.HCatStorer
       ('part1=pig, part2=hcat_pig_insert',
        'id: int,intnum: int,floatnum: float');

Generates the following error when running on a Federated Cluster:

2012-10-29 20:40:25,011 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate
exception from backed error: AttemptID:attempt_1348522594824_0846_m_000000_3
Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException:
getDefaultReplication on empty path is invalid
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479)
at org.apache.hadoop.hive.ql.io.RCFile$Writer.<init>(RCFile.java:723)
at org.apache.hadoop.hive.ql.io.RCFile$Writer.<init>(RCFile.java:705)
at
org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86)
at
org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100)
at
org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:587)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)

Issue Links

• relates to: PIG-2791 Pig does not work with ViewFileSystem
• relates to: HIVE-3754 Trunk hadoop 23 build fails