Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 1.99.3
    • Component/s: None
    • Labels:
      None

      Description

      We need to have procedure for metadata upgrade in place before first release.

      1. SQOOP-659-5.patch
        56 kB
        Hari Shreedharan
      2. SqoopMetadataUpgrade.pdf
        33 kB
        Hari Shreedharan
      3. SqoopMetadataUpgrade.pdf
        31 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          Hari Shreedharan added a comment -

          I am thinking of taking a stab at this. I think we need to modify the connector interface to add this method:

          public List<PropertyDetails> upgrade(List<PropertyDetails>);
          
          class PropertyDetails<T> {
            Class<?> propertyType; //for Type safety, so we can safely cast.
            String propertyName;
            Object propertyValue;
          }
          
          

          Does this make sense?

          Show
          Hari Shreedharan added a comment - I am thinking of taking a stab at this. I think we need to modify the connector interface to add this method: public List<PropertyDetails> upgrade(List<PropertyDetails>); class PropertyDetails<T> { Class <?> propertyType; // for Type safety, so we can safely cast . String propertyName; Object propertyValue; } Does this make sense?
          Hide
          Hari Shreedharan added a comment -

          Here is a rough document on the design and implementation plan. It is not very detailed, but this is an approximate plan.

          Show
          Hari Shreedharan added a comment - Here is a rough document on the design and implementation plan. It is not very detailed, but this is an approximate plan.
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Hari Shreedharan,
          thank you very much for taking this JIRA. I appreciate your time and effort. I've read the document and I like suggested approach. Please go ahead with the implementation. I do have just one minor suggestion, I would recommend changing the Upgrader methods to:

          public upgrade(MConnection old, MConnection new);
          public upgrade(MJob old, MJon new);
          

          The main difference here is that in this case the Upgraded is responsible only for moving the values specified by user (parameters, metadata) from one metadata structure to another whereas in the previous case it was also responsible for creating the structure itself.

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Hari Shreedharan , thank you very much for taking this JIRA. I appreciate your time and effort. I've read the document and I like suggested approach. Please go ahead with the implementation. I do have just one minor suggestion, I would recommend changing the Upgrader methods to: public upgrade(MConnection old, MConnection new ); public upgrade(MJob old, MJon new ); The main difference here is that in this case the Upgraded is responsible only for moving the values specified by user (parameters, metadata) from one metadata structure to another whereas in the previous case it was also responsible for creating the structure itself. Jarcec
          Hide
          Hari Shreedharan added a comment -

          Agreed. I think there needs to be some changes in the jdbc repository and repository handler as well. This is to change the findConnector/registerConnector classes. We will need to also add upgradeConnector (or modify registerConnector to handle the upgrade as well).

          Show
          Hari Shreedharan added a comment - Agreed. I think there needs to be some changes in the jdbc repository and repository handler as well. This is to change the findConnector/registerConnector classes. We will need to also add upgradeConnector (or modify registerConnector to handle the upgrade as well).
          Hide
          Jarek Jarcec Cecho added a comment -

          Yup, I'm fully expecting that this will be quite intrusive change, but nevertheless very necessary.

          Show
          Jarek Jarcec Cecho added a comment - Yup, I'm fully expecting that this will be quite intrusive change, but nevertheless very necessary.
          Hide
          Hari Shreedharan added a comment -

          Committers - Ping for review.

          Show
          Hari Shreedharan added a comment - Committers - Ping for review.
          Hide
          Venkat Ranganathan added a comment -

          +1 from me (non committer). This is a good one to get into 1.99.2 if we can get it verified

          Show
          Venkat Ranganathan added a comment - +1 from me (non committer). This is a good one to get into 1.99.2 if we can get it verified
          Hide
          Jarek Jarcec Cecho added a comment -

          Changing status to "patch available" so that this will show up in the review queue.

          Show
          Jarek Jarcec Cecho added a comment - Changing status to "patch available" so that this will show up in the review queue.
          Hide
          Hari Shreedharan added a comment -

          Patch from review board

          Show
          Hari Shreedharan added a comment - Patch from review board
          Hide
          Hari Shreedharan added a comment -

          Now that we are done with this, we need to:

          • Add validation to the upgrade method.
          • Move the cloneForms method to MForm class
          • Add unit and integration tests.

          There are probably a bunch of other things I missed - but these seem to be minimally required.

          Show
          Hari Shreedharan added a comment - Now that we are done with this, we need to: Add validation to the upgrade method. Move the cloneForms method to MForm class Add unit and integration tests. There are probably a bunch of other things I missed - but these seem to be minimally required.
          Hide
          Jarek Jarcec Cecho added a comment -
          Show
          Jarek Jarcec Cecho added a comment - The patch is in: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=c4467c6770c99b4f7e231ee0af162fa825791656 Thank you Hari for this big chunk of work! Jarcec
          Hide
          Jarek Jarcec Cecho added a comment -

          I agree with Hari Shreedharan that we still need to take this a bit further. I've change type of this ticket to sub-task and created additional sub-tasks for other associated work. Please do not hesitate and create new sub tasks if needed.

          Show
          Jarek Jarcec Cecho added a comment - I agree with Hari Shreedharan that we still need to take this a bit further. I've change type of this ticket to sub-task and created additional sub-tasks for other associated work. Please do not hesitate and create new sub tasks if needed.
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #271 (See https://builds.apache.org/job/Sqoop2-hadoop100/271/)
          SQOOP-659: Design metadata upgrade procedure (Revision c4467c6770c99b4f7e231ee0af162fa825791656)

          Result = FAILURE
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4467c6770c99b4f7e231ee0af162fa825791656
          Files :

          • repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
          • spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java
          • repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepoError.java
          • common/src/main/java/org/apache/sqoop/model/MJob.java
          • core/src/main/java/org/apache/sqoop/repository/Repository.java
          • repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java
          • spi/src/main/java/org/apache/sqoop/connector/spi/MetadataUpgrader.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java
          • common/src/main/java/org/apache/sqoop/model/MConnection.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryHandler.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #271 (See https://builds.apache.org/job/Sqoop2-hadoop100/271/ ) SQOOP-659 : Design metadata upgrade procedure (Revision c4467c6770c99b4f7e231ee0af162fa825791656) Result = FAILURE jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4467c6770c99b4f7e231ee0af162fa825791656 Files : repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepoError.java common/src/main/java/org/apache/sqoop/model/MJob.java core/src/main/java/org/apache/sqoop/repository/Repository.java repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java spi/src/main/java/org/apache/sqoop/connector/spi/MetadataUpgrader.java core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java common/src/main/java/org/apache/sqoop/model/MConnection.java core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryHandler.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #270 (See https://builds.apache.org/job/Sqoop2-hadoop200/270/)
          SQOOP-659: Design metadata upgrade procedure (Revision c4467c6770c99b4f7e231ee0af162fa825791656)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4467c6770c99b4f7e231ee0af162fa825791656
          Files :

          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java
          • spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java
          • common/src/main/java/org/apache/sqoop/model/MJob.java
          • spi/src/main/java/org/apache/sqoop/connector/spi/MetadataUpgrader.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryHandler.java
          • repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
          • repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
          • connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java
          • repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepoError.java
          • core/src/main/java/org/apache/sqoop/repository/Repository.java
          • core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java
          • common/src/main/java/org/apache/sqoop/model/MConnection.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #270 (See https://builds.apache.org/job/Sqoop2-hadoop200/270/ ) SQOOP-659 : Design metadata upgrade procedure (Revision c4467c6770c99b4f7e231ee0af162fa825791656) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=c4467c6770c99b4f7e231ee0af162fa825791656 Files : connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java common/src/main/java/org/apache/sqoop/model/MJob.java spi/src/main/java/org/apache/sqoop/connector/spi/MetadataUpgrader.java core/src/main/java/org/apache/sqoop/repository/JdbcRepositoryHandler.java repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepoError.java core/src/main/java/org/apache/sqoop/repository/Repository.java core/src/main/java/org/apache/sqoop/repository/JdbcRepository.java common/src/main/java/org/apache/sqoop/model/MConnection.java
          Hide
          Jarek Jarcec Cecho added a comment -

          The failure in profile 100 is yet another jenkins issue.

          Show
          Jarek Jarcec Cecho added a comment - The failure in profile 100 is yet another jenkins issue.

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development