Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1303

Can only write to default file system on incremental import

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None

      Description

      Running 'sqoop import' with the --incremental option fails if the --target-dir option points to a directory that is not in fs.default.name. In my case I was writing to s3 on a machine not configured so default fs was file:///.

      When running I received this error:
      ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.IllegalArgumentException: Wrong FS: s3://mybucket/mydirectory, expected: file:///

        Activity

        Hide
        markroddy Mark Roddy added a comment -

        Attached patch fixes bug by sourcing file system from the user supplied destination path rather than using the default file system.

        Show
        markroddy Mark Roddy added a comment - Attached patch fixes bug by sourcing file system from the user supplied destination path rather than using the default file system.
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        +1

        Show
        jarcec Jarek Jarcec Cecho added a comment - +1
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 7fb097829f27b1c78248b632852c948898165366 in sqoop's branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=7fb0978 ]

        SQOOP-1303: Can only write to default file system on incremental import

        (Mark Roddy via Jarek Jarcec Cecho)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 7fb097829f27b1c78248b632852c948898165366 in sqoop's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=7fb0978 ] SQOOP-1303 : Can only write to default file system on incremental import (Mark Roddy via Jarek Jarcec Cecho)
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Thank you for your contribution Mark Roddy!

        Show
        jarcec Jarek Jarcec Cecho added a comment - Thank you for your contribution Mark Roddy !
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #888 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/888/)
        SQOOP-1303: Can only write to default file system on incremental import (jarcec: https://git-wip-us.apache.org/repos/asf/?p=sqoop.git&a=commit&h=7fb097829f27b1c78248b632852c948898165366)

        • src/java/org/apache/sqoop/util/AppendUtils.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #888 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/888/ ) SQOOP-1303 : Can only write to default file system on incremental import (jarcec: https://git-wip-us.apache.org/repos/asf/?p=sqoop.git&a=commit&h=7fb097829f27b1c78248b632852c948898165366 ) src/java/org/apache/sqoop/util/AppendUtils.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #882 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/882/)
        SQOOP-1303: Can only write to default file system on incremental import (jarcec: https://git-wip-us.apache.org/repos/asf/?p=sqoop.git&a=commit&h=7fb097829f27b1c78248b632852c948898165366)

        • src/java/org/apache/sqoop/util/AppendUtils.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #882 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/882/ ) SQOOP-1303 : Can only write to default file system on incremental import (jarcec: https://git-wip-us.apache.org/repos/asf/?p=sqoop.git&a=commit&h=7fb097829f27b1c78248b632852c948898165366 ) src/java/org/apache/sqoop/util/AppendUtils.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1084 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1084/)
        SQOOP-1303: Can only write to default file system on incremental import (jarcec: https://git-wip-us.apache.org/repos/asf/?p=sqoop.git&a=commit&h=7fb097829f27b1c78248b632852c948898165366)

        • src/java/org/apache/sqoop/util/AppendUtils.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1084 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1084/ ) SQOOP-1303 : Can only write to default file system on incremental import (jarcec: https://git-wip-us.apache.org/repos/asf/?p=sqoop.git&a=commit&h=7fb097829f27b1c78248b632852c948898165366 ) src/java/org/apache/sqoop/util/AppendUtils.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #847 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/847/)
        SQOOP-1303: Can only write to default file system on incremental import (jarcec: https://git-wip-us.apache.org/repos/asf/?p=sqoop.git&a=commit&h=7fb097829f27b1c78248b632852c948898165366)

        • src/java/org/apache/sqoop/util/AppendUtils.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #847 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/847/ ) SQOOP-1303 : Can only write to default file system on incremental import (jarcec: https://git-wip-us.apache.org/repos/asf/?p=sqoop.git&a=commit&h=7fb097829f27b1c78248b632852c948898165366 ) src/java/org/apache/sqoop/util/AppendUtils.java
        Hide
        khariharan Karthic Hariharan added a comment - - edited

        I'm still getting this error when I used 1.4.5 version. The fix seems to be in the source code but it doesn't work on my EMR cluster.
        EDIT: Never mind I found the issue.

        Show
        khariharan Karthic Hariharan added a comment - - edited I'm still getting this error when I used 1.4.5 version. The fix seems to be in the source code but it doesn't work on my EMR cluster. EDIT: Never mind I found the issue.
        Hide
        anjireddy3 anji added a comment -

        Hi,
        i am using Sqoop 1.4.6 version and facing the same issue while importing data from Oracle DB to S3 directly. Please some one help on how to fix this issue. Is there any property that i can set at Sqoop configuration level. If only way is to build the sqoop with source code, please mention the steps. Waiting for ur help.

        Show
        anjireddy3 anji added a comment - Hi, i am using Sqoop 1.4.6 version and facing the same issue while importing data from Oracle DB to S3 directly. Please some one help on how to fix this issue. Is there any property that i can set at Sqoop configuration level. If only way is to build the sqoop with source code, please mention the steps. Waiting for ur help.

          People

          • Assignee:
            markroddy Mark Roddy
            Reporter:
            markroddy Mark Roddy
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development