[HIVE-3387] meta data file size exceeds limit

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.1
• Fix Version/s: 0.10.0
• Hadoop Flags: Reviewed
• Assignee: Navis Ryu
• Reporter: Alexander Alten-Lorenz
• Created: 15/Aug/12 16:10
• Updated: 02/May/13 02:30
• Resolved: 09/Sep/12 22:24

Description

The cause is certainly that we use an array list instead of a set structure in the split locations API. Looks like a bug in Hive's CombineFileInputFormat.

Reproduce:
Set mapreduce.jobtracker.split.metainfo.maxsize=100000000 when submitting the Hive query. Run a big hive query that write data into a partitioned table. Due to the large number of splits, you encounter an exception on the job submitted to Hadoop and the exception said:

meta data size exceeds 100000000.

Issue Links

• is related to: MAPREDUCE-2021 CombineFileInputFormat returns duplicate hostnames in split locations