Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1321

Add ability to serialize SqoopOption into JobConf

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.4
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None

      Description

      While working on various user issues, I've several times came into need to see what exact parameters were used when starting Sqoop. This seems relatively easy, but can get complicated when users are using some scripts on top of Sqoop binary or are directly integrating with the unsupported Java API.

      Hence I would like to propose to add option to simply dump entire SqoopOptions class into created Job object and make the used parameters available and persisted in the job.xml.

      This will also enable audit tools that are inspecting job.xml for various information.

      1. SQOOP-1321.patch
        5 kB
        Jarek Jarcec Cecho
      2. SQOOP-1321.patch
        4 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1087 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1087/)
          SQOOP-1321: Add ability to serialize SqoopOption into JobConf (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=18f5b2a77e95e412e806cabf4080a991f79aca92)

          • conf/sqoop-site-template.xml
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • src/java/org/apache/sqoop/mapreduce/MergeJob.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1087 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1087/ ) SQOOP-1321 : Add ability to serialize SqoopOption into JobConf (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=18f5b2a77e95e412e806cabf4080a991f79aca92 ) conf/sqoop-site-template.xml src/java/org/apache/sqoop/mapreduce/JobBase.java src/java/org/apache/sqoop/mapreduce/MergeJob.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #884 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/884/)
          SQOOP-1321: Add ability to serialize SqoopOption into JobConf (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=18f5b2a77e95e412e806cabf4080a991f79aca92)

          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/MergeJob.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • conf/sqoop-site-template.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #884 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/884/ ) SQOOP-1321 : Add ability to serialize SqoopOption into JobConf (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=18f5b2a77e95e412e806cabf4080a991f79aca92 ) src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/mapreduce/MergeJob.java src/java/org/apache/sqoop/mapreduce/JobBase.java conf/sqoop-site-template.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #890 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/890/)
          SQOOP-1321: Add ability to serialize SqoopOption into JobConf (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=18f5b2a77e95e412e806cabf4080a991f79aca92)

          • conf/sqoop-site-template.xml
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/MergeJob.java
          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #890 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/890/ ) SQOOP-1321 : Add ability to serialize SqoopOption into JobConf (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=18f5b2a77e95e412e806cabf4080a991f79aca92 ) conf/sqoop-site-template.xml src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/MergeJob.java src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/mapreduce/JobBase.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #849 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/849/)
          SQOOP-1321: Add ability to serialize SqoopOption into JobConf (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=18f5b2a77e95e412e806cabf4080a991f79aca92)

          • src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
          • src/java/org/apache/sqoop/mapreduce/JobBase.java
          • conf/sqoop-site-template.xml
          • src/java/org/apache/sqoop/mapreduce/MergeJob.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #849 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/849/ ) SQOOP-1321 : Add ability to serialize SqoopOption into JobConf (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=18f5b2a77e95e412e806cabf4080a991f79aca92 ) src/java/org/apache/sqoop/mapreduce/ExportJobBase.java src/java/org/apache/sqoop/mapreduce/ImportJobBase.java src/java/org/apache/sqoop/mapreduce/JobBase.java conf/sqoop-site-template.xml src/java/org/apache/sqoop/mapreduce/MergeJob.java
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Thank you [~nrv]!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Thank you [~nrv] !
          Hide
          venkatnrangan Venkat Ranganathan added a comment -

          Thanks for your contribution Jarek Jarcec Cecho

          Show
          venkatnrangan Venkat Ranganathan added a comment - Thanks for your contribution Jarek Jarcec Cecho
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 18f5b2a77e95e412e806cabf4080a991f79aca92 in sqoop's branch refs/heads/trunk from [~nrv]
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=18f5b2a ]

          SQOOP-1321: Add ability to serialize SqoopOption into JobConf
          (Jarek Jarcec Cecho via Venkat Ranganathan)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 18f5b2a77e95e412e806cabf4080a991f79aca92 in sqoop's branch refs/heads/trunk from [~nrv] [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=18f5b2a ] SQOOP-1321 : Add ability to serialize SqoopOption into JobConf (Jarek Jarcec Cecho via Venkat Ranganathan)

            People

            • Assignee:
              jarcec Jarek Jarcec Cecho
              Reporter:
              jarcec Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development