Sqoop
  1. Sqoop
  2. SQOOP-1232

Sqoop2: Provide tooling infrastructure for Sqoop2

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.3
    • Fix Version/s: 1.99.4
    • Component/s: None
    • Labels:
      None

      Description

      To improve administrator experience it would be great to provide tooling infrastructure that can be used to do various administrative and maintenance task on the Sqoop2 server, such as:

      • Validate the configuration
      • Upgrade repository and metadata structures (regardless of what usual configs says)
      • Dump and load repository
      1. SQOOP-1232.patch
        21 kB
        Jarek Jarcec Cecho
      2. SQOOP-1232.patch
        21 kB
        Jarek Jarcec Cecho
      3. SQOOP-1232.patch
        21 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          Sqoop QA bot added a comment -

          Here are the results of testing the latest attachment
          https://issues.apache.org/jira/secure/attachment/12614117/SQOOP-1232.patch against branch sqoop2.

          Overall: -1 due to an error

          ERROR: Patch does not add/modifny any test case
          SUCCESS: Clean was successful
          SUCCESS: Patch applied correctly
          SUCCESS: Patch compiled
          SUCCESS: All tests passed

          Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/184/console

          This message is automatically generated.

          Show
          Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12614117/SQOOP-1232.patch against branch sqoop2. Overall: -1 due to an error ERROR: Patch does not add/modifny any test case SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/184/console This message is automatically generated.
          Hide
          Sqoop QA bot added a comment -

          Here are the results of testing the latest attachment
          https://issues.apache.org/jira/secure/attachment/12615736/SQOOP-1232.patch against branch sqoop2.

          Overall: -1 due to 2 errors

          ERROR: failed to build with patch (exit code 1)
          ERROR: Patch does not add/modifny any test case
          SUCCESS: Clean was successful
          SUCCESS: Patch applied correctly

          Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/200/console

          This message is automatically generated.

          Show
          Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12615736/SQOOP-1232.patch against branch sqoop2. Overall: -1 due to 2 errors ERROR: failed to build with patch (exit code 1) ERROR: Patch does not add/modifny any test case SUCCESS: Clean was successful SUCCESS: Patch applied correctly Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/200/console This message is automatically generated.
          Hide
          Jarek Jarcec Cecho added a comment -

          The pre-commit hook failure with "failed to build with patch" is a glitch on Jenkins side (Maven failed to find it's own dependencies).

          Show
          Jarek Jarcec Cecho added a comment - The pre-commit hook failure with "failed to build with patch" is a glitch on Jenkins side (Maven failed to find it's own dependencies).
          Hide
          Sqoop QA bot added a comment -

          Here are the results of testing the latest attachment
          https://issues.apache.org/jira/secure/attachment/12616576/SQOOP-1232.patch against branch sqoop2.

          Overall: -1 due to an error

          ERROR: Patch does not add/modifny any test case
          SUCCESS: Clean was successful
          SUCCESS: Patch applied correctly
          SUCCESS: Patch compiled
          SUCCESS: All tests passed

          Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/201/console

          This message is automatically generated.

          Show
          Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12616576/SQOOP-1232.patch against branch sqoop2. Overall: -1 due to an error ERROR: Patch does not add/modifny any test case SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/201/console This message is automatically generated.
          Hide
          Hari Shreedharan added a comment -

          This looks good. +1. Running tests and committing

          Show
          Hari Shreedharan added a comment - This looks good. +1. Running tests and committing
          Hide
          ASF subversion and git services added a comment -

          Commit 1705d417a24ff78b9f00041b0ba07244f527cb1b in branch refs/heads/sqoop2 from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=1705d41 ]

          SQOOP-1232. Sqoop2: Provide tooling infrastructure for Sqoop2

          (Jarek Jarcec Cecho via Hari Shreedharan)

          Show
          ASF subversion and git services added a comment - Commit 1705d417a24ff78b9f00041b0ba07244f527cb1b in branch refs/heads/sqoop2 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=1705d41 ] SQOOP-1232 . Sqoop2: Provide tooling infrastructure for Sqoop2 (Jarek Jarcec Cecho via Hari Shreedharan)
          Hide
          Hari Shreedharan added a comment -

          Committed, thanks Jarcec!

          Show
          Hari Shreedharan added a comment - Committed, thanks Jarcec!
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Sqoop2-hadoop200 #528 (See https://builds.apache.org/job/Sqoop2-hadoop200/528/)
          SQOOP-1232. Sqoop2: Provide tooling infrastructure for Sqoop2 (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=1705d417a24ff78b9f00041b0ba07244f527cb1b)

          • tomcat/src/main/java/org/apache/sqoop/tomcat/TomcatToolRunner.java
          • pom.xml
          • tools/src/main/java/org/apache/sqoop/tools/tool/BuiltinTools.java
          • tools/src/main/java/org/apache/sqoop/tools/Tool.java
          • tools/pom.xml
          • tomcat/pom.xml
          • dist/src/main/bin/sqoop.sh
          • tools/src/main/java/org/apache/sqoop/tools/tool/VerifyTool.java
          • tools/src/main/java/org/apache/sqoop/tools/ToolRunner.java
          • dist/pom.xml
          • server/pom.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Sqoop2-hadoop200 #528 (See https://builds.apache.org/job/Sqoop2-hadoop200/528/ ) SQOOP-1232 . Sqoop2: Provide tooling infrastructure for Sqoop2 (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=1705d417a24ff78b9f00041b0ba07244f527cb1b ) tomcat/src/main/java/org/apache/sqoop/tomcat/TomcatToolRunner.java pom.xml tools/src/main/java/org/apache/sqoop/tools/tool/BuiltinTools.java tools/src/main/java/org/apache/sqoop/tools/Tool.java tools/pom.xml tomcat/pom.xml dist/src/main/bin/sqoop.sh tools/src/main/java/org/apache/sqoop/tools/tool/VerifyTool.java tools/src/main/java/org/apache/sqoop/tools/ToolRunner.java dist/pom.xml server/pom.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Sqoop2-hadoop100 #594 (See https://builds.apache.org/job/Sqoop2-hadoop100/594/)
          SQOOP-1232. Sqoop2: Provide tooling infrastructure for Sqoop2 (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=1705d417a24ff78b9f00041b0ba07244f527cb1b)

          • pom.xml
          • tools/src/main/java/org/apache/sqoop/tools/tool/BuiltinTools.java
          • tools/src/main/java/org/apache/sqoop/tools/Tool.java
          • tools/pom.xml
          • tomcat/src/main/java/org/apache/sqoop/tomcat/TomcatToolRunner.java
          • tools/src/main/java/org/apache/sqoop/tools/tool/VerifyTool.java
          • tomcat/pom.xml
          • tools/src/main/java/org/apache/sqoop/tools/ToolRunner.java
          • server/pom.xml
          • dist/src/main/bin/sqoop.sh
          • dist/pom.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Sqoop2-hadoop100 #594 (See https://builds.apache.org/job/Sqoop2-hadoop100/594/ ) SQOOP-1232 . Sqoop2: Provide tooling infrastructure for Sqoop2 (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=1705d417a24ff78b9f00041b0ba07244f527cb1b ) pom.xml tools/src/main/java/org/apache/sqoop/tools/tool/BuiltinTools.java tools/src/main/java/org/apache/sqoop/tools/Tool.java tools/pom.xml tomcat/src/main/java/org/apache/sqoop/tomcat/TomcatToolRunner.java tools/src/main/java/org/apache/sqoop/tools/tool/VerifyTool.java tomcat/pom.xml tools/src/main/java/org/apache/sqoop/tools/ToolRunner.java server/pom.xml dist/src/main/bin/sqoop.sh dist/pom.xml
          Hide
          Jarek Jarcec Cecho added a comment -

          The Jenkins build failures seems to be relevant as they are failing with following exception:

          An Ant BuildException has occured: Warning: Could not find file /home/jenkins/jenkins-slave/workspace/Sqoop2-hadoop200/tomcat/target/sqoop-tomcat-2.0.0-SNAPSHOT.jar to copy. -> [Help 1]
          

          I'm able to reproduce the problem in my local environment. Will take a look.

          Show
          Jarek Jarcec Cecho added a comment - The Jenkins build failures seems to be relevant as they are failing with following exception: An Ant BuildException has occured: Warning: Could not find file /home/jenkins/jenkins-slave/workspace/Sqoop2-hadoop200/tomcat/target/sqoop-tomcat-2.0.0-SNAPSHOT.jar to copy. -> [Help 1] I'm able to reproduce the problem in my local environment. Will take a look.
          Hide
          Jarek Jarcec Cecho added a comment -

          The problem seems to be in the order the modules are built:

          [INFO] Sqoop Distribution ................................ FAILURE [27.344s]
          [INFO] Sqoop Integration Tests ........................... SKIPPED
          [INFO] Sqoop Tomcat additions ............................ SKIPPED
          

          The distribution module is being built before the Tomcat module, but that should not be the case as the distribution module do have a dependency on tomcat module. It seems that I've missed adding this dependency to the pom files when working on SQOOP-1232. From whatever reason maven used different order for building the modules at that time and as a result this has slipped me. I've filled SQOOP-1247 to fix that.

          Show
          Jarek Jarcec Cecho added a comment - The problem seems to be in the order the modules are built: [INFO] Sqoop Distribution ................................ FAILURE [27.344s] [INFO] Sqoop Integration Tests ........................... SKIPPED [INFO] Sqoop Tomcat additions ............................ SKIPPED The distribution module is being built before the Tomcat module, but that should not be the case as the distribution module do have a dependency on tomcat module. It seems that I've missed adding this dependency to the pom files when working on SQOOP-1232 . From whatever reason maven used different order for building the modules at that time and as a result this has slipped me. I've filled SQOOP-1247 to fix that.

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development