Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2880

Provide argument for overriding temporary directory

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.6
    • Fix Version/s: 1.4.7
    • Component/s: None
    • Labels:
      None

      Description

      Several types of imports (incremental, hive, ...) might require import into temporary directory first. We're currently putting the temporary directory into _sqoop and we're allowing user to override it with sqoop.test.import.rootDir. However this property is not documented anywhere and I found people confused about it as it contains the string test which is really confusing. The most concerning thing is that as the properties are not persisted in metastore, this option won't work for stored jobs at all. Hence I would like to propose a first class argument to handle the temporary directory (e.g. something like --temporary-directory).

      1. SQOOP-2880_review_1.patch
        8 kB
        Attila Szabo
      2. SQOOP-2880.patch
        8 kB
        Attila Szabo

        Issue Links

          Activity

          Hide
          maugli Attila Szabo added a comment -

          Hi Jarcec,

          May I take over this JIRA, and implement it?

          Thanks

          Show
          maugli Attila Szabo added a comment - Hi Jarcec, May I take over this JIRA, and implement it? Thanks
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Sure, be my guest Attila Szabo!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Sure, be my guest Attila Szabo !
          Hide
          maugli Attila Szabo added a comment -

          The implemented changes. Please review it.

          Show
          maugli Attila Szabo added a comment - The implemented changes. Please review it.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a0b730c77e297a62909063289ef37a2b993ff5e1 in sqoop's branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=a0b730c ]

          SQOOP-2880: Provide argument for overriding temporary directory

          (Attila Szabo via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit a0b730c77e297a62909063289ef37a2b993ff5e1 in sqoop's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=a0b730c ] SQOOP-2880 : Provide argument for overriding temporary directory (Attila Szabo via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Thank you for your contribution Attila Szabo!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Thank you for your contribution Attila Szabo !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-hadoop20 #1042 (See https://builds.apache.org/job/Sqoop-hadoop20/1042/)
          SQOOP-2880: Provide argument for overriding temporary directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a0b730c77e297a62909063289ef37a2b993ff5e1)

          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/test/com/cloudera/sqoop/orm/TestParseMethods.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/util/AppendUtils.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-hadoop20 #1042 (See https://builds.apache.org/job/Sqoop-hadoop20/1042/ ) SQOOP-2880 : Provide argument for overriding temporary directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a0b730c77e297a62909063289ef37a2b993ff5e1 ) src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/ImportTool.java src/test/com/cloudera/sqoop/orm/TestParseMethods.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/util/AppendUtils.java src/test/com/cloudera/sqoop/TestSqoopOptions.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-hadoop23 #1245 (See https://builds.apache.org/job/Sqoop-hadoop23/1245/)
          SQOOP-2880: Provide argument for overriding temporary directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a0b730c77e297a62909063289ef37a2b993ff5e1)

          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/util/AppendUtils.java
          • src/test/com/cloudera/sqoop/orm/TestParseMethods.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-hadoop23 #1245 (See https://builds.apache.org/job/Sqoop-hadoop23/1245/ ) SQOOP-2880 : Provide argument for overriding temporary directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a0b730c77e297a62909063289ef37a2b993ff5e1 ) src/java/org/apache/sqoop/SqoopOptions.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/util/AppendUtils.java src/test/com/cloudera/sqoop/orm/TestParseMethods.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/tool/ImportTool.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-hadoop200 #1047 (See https://builds.apache.org/job/Sqoop-hadoop200/1047/)
          SQOOP-2880: Provide argument for overriding temporary directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a0b730c77e297a62909063289ef37a2b993ff5e1)

          • src/test/com/cloudera/sqoop/orm/TestParseMethods.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/java/org/apache/sqoop/util/AppendUtils.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-hadoop200 #1047 (See https://builds.apache.org/job/Sqoop-hadoop200/1047/ ) SQOOP-2880 : Provide argument for overriding temporary directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a0b730c77e297a62909063289ef37a2b993ff5e1 ) src/test/com/cloudera/sqoop/orm/TestParseMethods.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/util/AppendUtils.java src/test/com/cloudera/sqoop/TestSqoopOptions.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Sqoop-hadoop100 #1007 (See https://builds.apache.org/job/Sqoop-hadoop100/1007/)
          SQOOP-2880: Provide argument for overriding temporary directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a0b730c77e297a62909063289ef37a2b993ff5e1)

          • src/test/com/cloudera/sqoop/orm/TestParseMethods.java
          • src/java/org/apache/sqoop/tool/ImportTool.java
          • src/java/org/apache/sqoop/util/AppendUtils.java
          • src/test/com/cloudera/sqoop/TestSqoopOptions.java
          • src/java/org/apache/sqoop/SqoopOptions.java
          • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Sqoop-hadoop100 #1007 (See https://builds.apache.org/job/Sqoop-hadoop100/1007/ ) SQOOP-2880 : Provide argument for overriding temporary directory (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a0b730c77e297a62909063289ef37a2b993ff5e1 ) src/test/com/cloudera/sqoop/orm/TestParseMethods.java src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/util/AppendUtils.java src/test/com/cloudera/sqoop/TestSqoopOptions.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java

            People

            • Assignee:
              maugli Attila Szabo
              Reporter:
              jarcec Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development