Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-3330

Sqoop --append does not work with -Dmapreduce.output.basename

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.7
    • Fix Version/s: 3.0.0
    • Component/s: tools
    • Labels:
      None

      Description

      When adding --append to Sqoop directory import with -Dmapreduce.output.basename, all files will be ignored, which end up nothing being imported. See below DEBUG output:

      sqoop import -Dmapreduce.output.basename="eric-test" --connect jdbc:mysql://mysql-host.com/test --username root --password 'root' --table test --target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' --verbose --append
      
      18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
      18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
      18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00000 ignored
      18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00001 ignored
      18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00002 ignored
      18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 14935e396acc4ea7b9a6236c66064c9b_test
      

      This is due to AppendUtils only recognizes file name starts with "part.*-([0-9]"

      https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46

        Attachments

        1. SQOOP-3330.3.patch
          6 kB
          Eric Lin
        2. SQOOP-3330-2.patch
          6 kB
          Eric Lin
        3. SQOOP-3330-1.patch
          7 kB
          Eric Lin

          Issue Links

            Activity

              People

              • Assignee:
                ericlin Eric Lin
                Reporter:
                ericlin Eric Lin
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: