[IMPALA-577] Batch hdfsWrite calls for hdfs-text-writer

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.1.1
• Fix Version/s: Impala 1.2
• Assignee: Nong Li
• Reporter: Nong Li
• Created: 04/Sep/13 20:54
• Updated: 19/Sep/13 20:53
• Resolved: 19/Sep/13 20:53

Description

We should look into the right batch size to send to hdfswrite. We previously called it once per row, which lead to very poor performance. Now we batch based on the input batch size.

This is not effective for partitioned tables where the input batch is split. We should also see if this is the best size to pass to hdfs in general.