[SQOOP-1078] incremental import from database in direct mode

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.4.2, 1.4.3
• Fix Version/s: 1.4.4
• Component/s: connectors, connectors/mysql, connectors/postgresql
• Labels: patch
• Assignee: Tim Howe
• Reporter: Tim Howe
• Created: 13/Jun/13 18:15
• Updated: 20/Jun/13 05:11
• Resolved: 19/Jun/13 22:41

Description

A problem exists in Sqoop's incremental import, namely that any imports
after the first report success but the data never appears.
A temporary file created on HDFS with the data but is deleted upon
completion rather than being moved into place.

It turns out to be a conflict between the "direct mode" database
managers and "incremental mode" import. Ordinarily Sqoop ends up
creating files named part-m-nnnnn where nnnnn is an incrementing file
partition number. However the direct mode importer creates files of
the form data-nnnnn. This poses a problem because AppendUtils, which
is used to move files into place at the end of a direct import, only
copies files which match that part-m-nnnnn format and discards the
rest.