Sqoop
  1. Sqoop
  2. SQOOP-1078

incremental import from database in direct mode

    Details

      Description

      A problem exists in Sqoop's incremental import, namely that any imports
      after the first report success but the data never appears.
      A temporary file created on HDFS with the data but is deleted upon
      completion rather than being moved into place.

      It turns out to be a conflict between the "direct mode" database
      managers and "incremental mode" import. Ordinarily Sqoop ends up
      creating files named part-m-nnnnn where nnnnn is an incrementing file
      partition number. However the direct mode importer creates files of
      the form data-nnnnn. This poses a problem because AppendUtils, which
      is used to move files into place at the end of a direct import, only
      copies files which match that part-m-nnnnn format and discards the
      rest.

        Activity

          People

          • Assignee:
            Tim Howe
            Reporter:
            Tim Howe
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development