[HIVE-2680] In FileSinkOperator if the RecordWriter.write throw IOException, we should call RecordWriter's close method.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Assignee: Unassigned
• Reporter: Lijin Bin
• Created: 26/Dec/11 14:28
• Updated: 21/Apr/12 16:27

Description

Dynamic-partition Insert, if the partition that will be created is large, many files will be created and if the input is large the DataNode's xceiverCount will easily exceeds the limit of concurrent xcievers (default 1024), The RecordWriter.write(recordValue) will throw the Exception "Could not read from stream". After an hour the lease timeout, there will many commitBlockSynchronization requests and the namenode's load will be very high, so abortWriters should be called.