Hadoop Common
  1. Hadoop Common
  2. HADOOP-5967

Sqoop should only use a single map task


    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:


      The current DBInputFormat implementation uses SELECT ... LIMIT ... OFFSET statements to read from a database table. This actually results in several queries all accessing the same table at the same time. Most database implementations will actually use a full table scan for each such query, starting at row 1 and scanning down until the OFFSET is reached before emitting data to the client. The upshot of this is that we see O(n^2) performance in the size of the table when using a large number of mappers, when a single mapper would read through the table in O time in the number of rows.

      This patch sets the number of map tasks to 1 in the MapReduce job sqoop launches.

      1. single-mapper.patch
        0.6 kB
        Aaron Kimball


        Aaron Kimball created issue -
        Aaron Kimball made changes -
        Field Original Value New Value
        Attachment single-mapper.patch [ 12409836 ]
        Aaron Kimball made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.21.0 [ 12313563 ]
        Resolution Fixed [ 1 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]


          • Assignee:
            Aaron Kimball
            Aaron Kimball
          • Votes:
            0 Vote for this issue
            2 Start watching this issue


            • Created: