[HADOOP-5967] Sqoop should only use a single map task

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.21.0
• Hadoop Flags: Reviewed
• Assignee: Aaron Kimball
• Reporter: Aaron Kimball
• Created: 04/Jun/09 01:02
• Updated: 24/Aug/10 20:38
• Resolved: 23/Jun/09 19:52

Description

The current DBInputFormat implementation uses SELECT ... LIMIT ... OFFSET statements to read from a database table. This actually results in several queries all accessing the same table at the same time. Most database implementations will actually use a full table scan for each such query, starting at row 1 and scanning down until the OFFSET is reached before emitting data to the client. The upshot of this is that we see O(n^2) performance in the size of the table when using a large number of mappers, when a single mapper would read through the table in O time in the number of rows.

This patch sets the number of map tasks to 1 in the MapReduce job sqoop launches.