[HIVE-439] merge small files after a map-only job

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.3.0
• Fix Version/s: 0.4.0
• Component/s: Query Processor
• Hadoop Flags: Reviewed
• Release Note: HIVE-439. Merge small files after a map-only job. (Namit Jain via zshao)
• Assignee: Namit Jain
• Reporter: Namit Jain
• Created: 21/Apr/09 16:43
• Updated: 17/Dec/11 00:07
• Resolved: 23/Jun/09 05:15

Description

There are cases when the input to a Hive job are thousands of small files. In this case, there is a mapper for each file. Most of the overhead for spawning all these mappers can be avoided if these small files are combined into fewer larger files.

The problem can also be addressed by having a mapper span multiple blocks as in:

https://issues.apache.org/jira/browse/HIVE-74

Bit, it also makes sense in HIVE to merge files whenever possible.

<property>
  <name>hive.merge.mapfiles</name>
  <value>true</value>
  <description>Merge small files at the end of the job</description>
</property>

<property>
  <name>hive.merge.size.per.task</name>
  <value>256000000</value>
  <description>Size of merged files at the end of the job</description>
</property>