Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1138

incremental lastmodified should re-use output directory

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.4
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None

      Description

      When using --incremental lastmodified option in sqoop command line "the second job will take both the old and new data and will merge them together into the final output, preserving only the last updated value for each row."

      However, when I try to run incremental lastmodified twice, the second time I get "FileAlreadyExistsException: Output directory already exists".

      The incremental lastmodified should read from this directory to get the data from the last run and should not throw an exception.

      1. SQOOP-1138.0.patch
        4 kB
        Abraham Elmahrek
      2. SQOOP-1138.1.patch
        20 kB
        Abraham Elmahrek
      3. SQOOP-1138.2.patch
        19 kB
        Abraham Elmahrek
      4. SQOOP-1138.4.patch
        19 kB
        Abraham Elmahrek
      5. SQOOP-1138.5.patch
        20 kB
        Abraham Elmahrek
      6. SQOOP-1138.6.patch
        20 kB
        Abraham Elmahrek

        Issue Links

          Activity

          Hide
          amos.wood@lifeway.com Amos Wood added a comment -

          I am also experiencing this issue which is causing my active archiving scheme to fail. I am using the distribution version from HDP 2.1 (1.4.4.2.1.2.1-471).

          Show
          amos.wood@lifeway.com Amos Wood added a comment - I am also experiencing this issue which is causing my active archiving scheme to fail. I am using the distribution version from HDP 2.1 (1.4.4.2.1.2.1-471).
          Hide
          abec Abraham Elmahrek added a comment -

          Rebase + changes from review.

          Show
          abec Abraham Elmahrek added a comment - Rebase + changes from review.
          Hide
          abec Abraham Elmahrek added a comment -

          Rebased against trunk.

          Show
          abec Abraham Elmahrek added a comment - Rebased against trunk.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9 in sqoop's branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=34e4efd ]

          SQOOP-1138: incremental lastmodified should re-use output directory

          (Abraham Elmahrek via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9 in sqoop's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=34e4efd ] SQOOP-1138 : incremental lastmodified should re-use output directory (Abraham Elmahrek via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          The patch is in, thank you for your contribution Abraham Elmahrek]!

          Show
          jarcec Jarek Jarcec Cecho added a comment - The patch is in, thank you for your contribution Abraham Elmahrek ]!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #862 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/862/)
          SQOOP-1138: incremental lastmodified should re-use output directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9)

          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/test/com/cloudera/sqoop/TestIncrementalImport.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #862 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/862/ ) SQOOP-1138 : incremental lastmodified should re-use output directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9 ) src/java/org/apache/sqoop/tool/ImportTool.java src/test/com/cloudera/sqoop/TestIncrementalImport.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #903 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/903/)
          SQOOP-1138: incremental lastmodified should re-use output directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9)

          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/test/com/cloudera/sqoop/TestIncrementalImport.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #903 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/903/ ) SQOOP-1138 : incremental lastmodified should re-use output directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9 ) src/java/org/apache/sqoop/tool/ImportTool.java src/test/com/cloudera/sqoop/TestIncrementalImport.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #897 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/897/)
          SQOOP-1138: incremental lastmodified should re-use output directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9)

          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/test/com/cloudera/sqoop/TestIncrementalImport.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #897 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/897/ ) SQOOP-1138 : incremental lastmodified should re-use output directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9 ) src/java/org/apache/sqoop/tool/ImportTool.java src/test/com/cloudera/sqoop/TestIncrementalImport.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1100 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1100/)
          SQOOP-1138: incremental lastmodified should re-use output directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9)

          • src/test/com/cloudera/sqoop/TestIncrementalImport.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1100 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1100/ ) SQOOP-1138 : incremental lastmodified should re-use output directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=34e4efd0d7a6d34b3b89a7d43271ebb5aa8193a9 ) src/test/com/cloudera/sqoop/TestIncrementalImport.java src/java/org/apache/sqoop/tool/ImportTool.java

            People

            • Assignee:
              abec Abraham Elmahrek
              Reporter:
              jmspaggi Jean-Marc Spaggiari
            • Votes:
              5 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development