Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-3336

Splitting on integer column can create more splits than necessary

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 1.4.7
    • 1.5.0, 3.0.0
    • None
    • None

    Description

      Running an import with -m 2 will result in three splits if there are only three consecutive integers in the table ({1, 2, 3}).

      Work is (probably) spread more evenly between mappers this way, but ending up with more files than expected could be an issue.

      Split-limit can also result in more values than asked for in the last chunk (due to the closed interval in the end).

      Attachments

        Issue Links

          Activity

            People

              dvoros Daniel Voros
              dvoros Daniel Voros
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: