[SQOOP-3330] Sqoop --append does not work with -Dmapreduce.output.basename

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.7
• Fix Version/s: 3.0.0
• Component/s: tools
• Assignee: Eric Lin
• Reporter: Eric Lin
• Created: 29/May/18 06:37
• Updated: 13/Jul/18 07:36
• Resolved: 13/Jul/18 07:36

Description

When adding --append to Sqoop directory import with -Dmapreduce.output.basename, all files will be ignored, which end up nothing being imported. See below DEBUG output:

sqoop import -Dmapreduce.output.basename="eric-test" --connect jdbc:mysql://mysql-host.com/test --username root --password 'root' --table test --target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' --verbose --append

18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00000 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00001 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00002 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder 14935e396acc4ea7b9a6236c66064c9b_test

This is due to AppendUtils only recognizes file name starts with "part.*-([0-9]"

https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46

Issue Links

• links to: Review