[HIVE-4175] Injection of emptyFile into input splits for empty partitions causes Deserializer to fail

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.10.0

Environment

CDH4.2, using MR1

• Assignee: Unassigned
• Reporter: James Kebinger
• Created: 14/Mar/13 14:55
• Updated: 26/Mar/18 07:22

Description

• My deserializer is expecting to receive one of 2 different subclasses of Writable, but in certain circumstances it receives an empty instance of org.apache.hadoop.io.Text. This only happens for task attempts where I observe the file called "emptyFile" in the list of input splits.

I'm doing queries over an external year/month/day partitioned table that have eagerly created partitions for, so as of today for example, I may do a query where year = 2013 and month = 3 which includes empty partitions.

In the course of investigation I downloaded the sequence files to confirm they were ok. Once I realized that processing of empty partitions was to blame, I am able to work around the issue by bounding my queries to populated partitions.

Can the need for the emptyFile be eliminated in the case where there's already a bunch of splits being processed? Failing that, can the mapper detect the current input is from emptyFile and not call the deserializer.