Sqoop
  1. Sqoop
  2. SQOOP-993 Sqoop2: Metadata upgrade
  3. SQOOP-994

Sqoop2: Upgrade: Add calling validation to the upgrade method

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.99.2
    • Fix Version/s: 1.99.3
    • Component/s: None
    • Labels:
      None

      Description

      We should call connector's validator to see if the object case is good enough.

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop100 #456 (See https://builds.apache.org/job/Sqoop2-hadoop100/456/)
          SQOOP-994: Sqoop2: Upgrade: Add calling validation to the upgrade method (Revision 96a02dfee99132c867424091ad114626b61c0894)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=96a02dfee99132c867424091ad114626b61c0894
          Files :

          • core/src/main/java/org/apache/sqoop/repository/Repository.java
          • core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop100 #456 (See https://builds.apache.org/job/Sqoop2-hadoop100/456/ ) SQOOP-994 : Sqoop2: Upgrade: Add calling validation to the upgrade method (Revision 96a02dfee99132c867424091ad114626b61c0894) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=96a02dfee99132c867424091ad114626b61c0894 Files : core/src/main/java/org/apache/sqoop/repository/Repository.java core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
          Hide
          Hudson added a comment -

          Integrated in Sqoop2-hadoop200 #436 (See https://builds.apache.org/job/Sqoop2-hadoop200/436/)
          SQOOP-994: Sqoop2: Upgrade: Add calling validation to the upgrade method (Revision 96a02dfee99132c867424091ad114626b61c0894)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=96a02dfee99132c867424091ad114626b61c0894
          Files :

          • core/src/main/java/org/apache/sqoop/repository/Repository.java
          • core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
          Show
          Hudson added a comment - Integrated in Sqoop2-hadoop200 #436 (See https://builds.apache.org/job/Sqoop2-hadoop200/436/ ) SQOOP-994 : Sqoop2: Upgrade: Add calling validation to the upgrade method (Revision 96a02dfee99132c867424091ad114626b61c0894) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=96a02dfee99132c867424091ad114626b61c0894 Files : core/src/main/java/org/apache/sqoop/repository/Repository.java core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
          Jarek Jarcec Cecho made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 1.99.3 [ 12324298 ]
          Fix Version/s 2.0.0 [ 12319272 ]
          Resolution Fixed [ 1 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you Mengwei for your contribution!

          Show
          Jarek Jarcec Cecho added a comment - Thank you Mengwei for your contribution!
          Hide
          ASF subversion and git services added a comment -

          Commit 96a02dfee99132c867424091ad114626b61c0894 in branch refs/heads/sqoop2 from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=96a02df ]

          SQOOP-994: Sqoop2: Upgrade: Add calling validation to the upgrade method

          (Mengwei Ding via Jarek Jarcec Cecho)

          Show
          ASF subversion and git services added a comment - Commit 96a02dfee99132c867424091ad114626b61c0894 in branch refs/heads/sqoop2 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=96a02df ] SQOOP-994 : Sqoop2: Upgrade: Add calling validation to the upgrade method (Mengwei Ding via Jarek Jarcec Cecho)
          Hide
          Sqoop QA bot added a comment -

          Here are the results of testing the latest attachment
          https://issues.apache.org/jira/secure/attachment/12590631/SQOOP-994.1.patch against branch sqoop2.

          Overall: +1 all checks pass

          SUCCESS: Clean was successful
          SUCCESS: Patch applied correctly
          SUCCESS: Patch compiled
          SUCCESS: All tests passed

          Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/39/console

          This message is automatically generated.

          Show
          Sqoop QA bot added a comment - Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12590631/SQOOP-994.1.patch against branch sqoop2. Overall: +1 all checks pass SUCCESS: Clean was successful SUCCESS: Patch applied correctly SUCCESS: Patch compiled SUCCESS: All tests passed Console output: https://builds.apache.org/job/PreCommit-SQOOP-Build/39/console This message is automatically generated.
          Mengwei Ding made changes -
          Remote Link This issue links to "ReviewBoard (Web Link)" [ 12388 ]
          Mengwei Ding made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Mengwei Ding added a comment -

          Submit patch for review.

          Show
          Mengwei Ding added a comment - Submit patch for review.
          Mengwei Ding made changes -
          Attachment SQOOP-994.1.patch [ 12590631 ]
          Mengwei Ding made changes -
          Assignee Mengwei Ding [ mengweid ]
          Jarek Jarcec Cecho made changes -
          Field Original Value New Value
          Fix Version/s 2.0.0 [ 12319272 ]
          Hide
          Hari Shreedharan added a comment -

          Returning to the old connector probably makes sense, but even to drop the object causing issues the user would need to either (1) disable automatically upgrading or (2) downgrade the connector, disable and drop the connections/jobs causing issues.

          Show
          Hari Shreedharan added a comment - Returning to the old connector probably makes sense, but even to drop the object causing issues the user would need to either (1) disable automatically upgrading or (2) downgrade the connector, disable and drop the connections/jobs causing issues.
          Hide
          Jarek Jarcec Cecho added a comment -

          That's an excellent point, indeed it's not clearly visible what to do in this case. As we're trying to guarantee that repository contain only properly validated entries, I would personally thrown an exception and shut down Sqoop server boot procedure. In this case user will be given option to either return to old connector version or simply drop the connection/job object that is causing upgrade issues. What do you think?

          Show
          Jarek Jarcec Cecho added a comment - That's an excellent point, indeed it's not clearly visible what to do in this case. As we're trying to guarantee that repository contain only properly validated entries, I would personally thrown an exception and shut down Sqoop server boot procedure. In this case user will be given option to either return to old connector version or simply drop the connection/job object that is causing upgrade issues. What do you think?
          Hide
          Hari Shreedharan added a comment -

          This is an interesting problem. It is easy enough to call the validate methods, but the question is how to handle a validation failure? Since the new connector may not work with older metadata for connections and jobs, rolling back the metadata upgrade is not likely to be good enough. We can somehow mark the connections and jobs as invalid and make sure the user (and later if the user is an admin) sees it when they login. But that is only error reporting, recovery may not even be possible.

          Show
          Hari Shreedharan added a comment - This is an interesting problem. It is easy enough to call the validate methods, but the question is how to handle a validation failure? Since the new connector may not work with older metadata for connections and jobs, rolling back the metadata upgrade is not likely to be good enough. We can somehow mark the connections and jobs as invalid and make sure the user (and later if the user is an admin) sees it when they login. But that is only error reporting, recovery may not even be possible.
          Jarek Jarcec Cecho created issue -

            People

            • Assignee:
              Mengwei Ding
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development