Sqoop
  1. Sqoop
  2. SQOOP-943

Sqoop2: Repository wide configuration of immutable disk structures

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.1
    • Fix Version/s: 1.99.3
    • Component/s: None
    • Labels:
      None

      Description

      During investigation of SQOOP-878 I've realized that it would be quite helpful to promote the idea of Repository on disk structures to first class citizens. As such we can explicitly ask repository implementation if the internals are suitable for use or not. We can also limit all major changes to one method that will be conditionally called only if user explicitly allows on disk structure changes (on disk structure upgrade of JDBC repository would be table schema changes).

      1. bugSQOOP-943.patch
        12 kB
        Jarek Jarcec Cecho
      2. bugSQOOP-943.patch
        12 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #413 (See https://builds.apache.org/job/Sqoop2-hadoop200/413/)
          SQOOP-943: Sqoop2: Repository wide configuration of immutable disk structures (Revision 681914c851cb275f1d803ffe057c2a7a53436330)

          Result = SUCCESS
          kathleen : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=681914c851cb275f1d803ffe057c2a7a53436330
          Files :

          • core/src/main/java/org/apache/sqoop/repository/Repository.java
          • core/src/main/java/org/apache/sqoop/repository/RepoConfigurationConstants.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryProvider.java
          • test/src/main/java/org/apache/sqoop/test/minicluster/SqoopMiniCluster.java
          • core/src/main/java/org/apache/sqoop/repository/RepositoryManager.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryContext.java
          • dist/src/main/server/conf/sqoop.properties
          • core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #413 (See https://builds.apache.org/job/Sqoop2-hadoop200/413/ ) SQOOP-943 : Sqoop2: Repository wide configuration of immutable disk structures (Revision 681914c851cb275f1d803ffe057c2a7a53436330) Result = SUCCESS kathleen : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=681914c851cb275f1d803ffe057c2a7a53436330 Files : core/src/main/java/org/apache/sqoop/repository/Repository.java core/src/main/java/org/apache/sqoop/repository/RepoConfigurationConstants.java core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryProvider.java test/src/main/java/org/apache/sqoop/test/minicluster/SqoopMiniCluster.java core/src/main/java/org/apache/sqoop/repository/RepositoryManager.java core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryContext.java dist/src/main/server/conf/sqoop.properties core/src/main/java/org/apache/sqoop/repository/RepositoryError.java core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #411 (See https://builds.apache.org/job/Sqoop2-hadoop100/411/)
          SQOOP-943: Sqoop2: Repository wide configuration of immutable disk structures (Revision 681914c851cb275f1d803ffe057c2a7a53436330)

          Result = SUCCESS
          kathleen : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=681914c851cb275f1d803ffe057c2a7a53436330
          Files :

          • core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java
          • test/src/main/java/org/apache/sqoop/test/minicluster/SqoopMiniCluster.java
          • core/src/main/java/org/apache/sqoop/repository/RepoConfigurationConstants.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryProvider.java
          • core/src/main/java/org/apache/sqoop/repository/RepositoryManager.java
          • core/src/main/java/org/apache/sqoop/repository/Repository.java
          • dist/src/main/server/conf/sqoop.properties
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryContext.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #411 (See https://builds.apache.org/job/Sqoop2-hadoop100/411/ ) SQOOP-943 : Sqoop2: Repository wide configuration of immutable disk structures (Revision 681914c851cb275f1d803ffe057c2a7a53436330) Result = SUCCESS kathleen : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=681914c851cb275f1d803ffe057c2a7a53436330 Files : core/src/main/java/org/apache/sqoop/repository/RepositoryError.java core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java test/src/main/java/org/apache/sqoop/test/minicluster/SqoopMiniCluster.java core/src/main/java/org/apache/sqoop/repository/RepoConfigurationConstants.java core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryProvider.java core/src/main/java/org/apache/sqoop/repository/RepositoryManager.java core/src/main/java/org/apache/sqoop/repository/Repository.java dist/src/main/server/conf/sqoop.properties core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryContext.java
          Hide
          Kathleen Ting added a comment -

          Thanks Jarcec for the contribution.

          Show
          Kathleen Ting added a comment - Thanks Jarcec for the contribution.
          Hide
          ASF subversion and git services added a comment -

          Commit 681914c851cb275f1d803ffe057c2a7a53436330 in branch refs/heads/sqoop2 from Kathleen Ting
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=681914c ]

          SQOOP-943: Sqoop2: Repository wide configuration of immutable disk structures

          (Jarek Jarcec Cecho via Kate Ting)

          Show
          ASF subversion and git services added a comment - Commit 681914c851cb275f1d803ffe057c2a7a53436330 in branch refs/heads/sqoop2 from Kathleen Ting [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=681914c ] SQOOP-943 : Sqoop2: Repository wide configuration of immutable disk structures (Jarek Jarcec Cecho via Kate Ting)
          Hide
          Sqoop QA bot added a comment -

          Here are the results of testing the latest attachment
          https://issues.apache.org/jira/secure/attachment/12588050/bugSQOOP-943.patch against branch SQOOP-1082.

          Overall: +1 all checks pass

          SUCCESS: Clean was successful
          SUCCESS: Patch applied correctly
          SUCCESS: Patch compiled
          SUCCESS: All tests passed

          Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/20/console

          This message is automatically generated.

          Show
          Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12588050/bugSQOOP-943.patch against branch SQOOP-1082 . Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/20/console This message is automatically generated.
          Hide
          Sqoop QA bot added a comment -

          Here are the results of testing the latest attachment
          https://issues.apache.org/jira/secure/attachment/12573021/bugSQOOP-943.patch against branch SQOOP-1082.

          Overall: -1 due to an error

          ERROR: failed to apply patch (exit code 1):

          error: patch failed: core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java:31
          error: core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java: patch does not apply
          error: patch failed: core/src/main/java/org/apache/sqoop/repository/Repository.java:37
          error: core/src/main/java/org/apache/sqoop/repository/Repository.java: patch does not apply
          

          SUCCESS: Clean was successful

          Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/17/console

          This message is automatically generated.

          Show
          Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12573021/bugSQOOP-943.patch against branch SQOOP-1082 . Overall: -1 due to an error ERROR: failed to apply patch (exit code 1): error: patch failed: core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java:31 error: core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java: patch does not apply error: patch failed: core/src/main/java/org/apache/sqoop/repository/Repository.java:37 error: core/src/main/java/org/apache/sqoop/repository/Repository.java: patch does not apply SUCCESS: Clean was successful Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/17/console This message is automatically generated.
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Venkat Ranganathan,
          thank you very much for checking my patch, greatly appreciated! To answer your question, I honestly believe that future changes to repository are inevitable, thus I would like to make them as simple as possible.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Venkat Ranganathan , thank you very much for checking my patch, greatly appreciated! To answer your question, I honestly believe that future changes to repository are inevitable, thus I would like to make them as simple as possible. Jarcec
          Hide
          Venkat Ranganathan added a comment -

          Hi Jarek Jarcec Cecho

          I looked throw the patch (and it is always good to have version of schema in a table) for the benefit of the application. A general question though. Are there any plans for upgrades to these structures because of function/feature/product update

          Thanks

          Show
          Venkat Ranganathan added a comment - Hi Jarek Jarcec Cecho I looked throw the patch (and it is always good to have version of schema in a table) for the benefit of the application. A general question though. Are there any plans for upgrades to these structures because of function/feature/product update Thanks

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development