Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5967

Sqoop should only use a single map task

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 0.21.0
    • None
    • None
    • Reviewed

    Description

      The current DBInputFormat implementation uses SELECT ... LIMIT ... OFFSET statements to read from a database table. This actually results in several queries all accessing the same table at the same time. Most database implementations will actually use a full table scan for each such query, starting at row 1 and scanning down until the OFFSET is reached before emitting data to the client. The upshot of this is that we see O(n^2) performance in the size of the table when using a large number of mappers, when a single mapper would read through the table in O time in the number of rows.

      This patch sets the number of map tasks to 1 in the MapReduce job sqoop launches.

      Attachments

        1. single-mapper.patch
          0.6 kB
          Aaron Kimball

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kimballa Aaron Kimball
            kimballa Aaron Kimball
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment