[SQOOP-1125] Out of memory errors when number of records to import < 0.5 * splitSize

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.3
• Fix Version/s: 1.4.6
• Labels: newbie
• Assignee: Sai Karthik Ganguru
• Reporter: Dave Kincaid
• Created: 08/Jul/13 19:18
• Updated: 20/Nov/14 00:14
• Resolved: 19/Nov/14 23:16

Description

We are getting out of memory errors during import if the number of records to import is less than 0.5*splitSize (and is nonterminating decimal).

For example, if the numSplits = 3, minVal = 100, maxVal = 101 then in BigDecimalSplitter.split() an extraordinary number of tiny values will be added to the splits List and run out of memory eventually.

I also noticed that there are no tests for BigDecimalSplitter.