Hive
  1. Hive
  2. HIVE-3255

Add DBTokenStore to store Delegation Tokens in DB

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.12.0
    • Component/s: Metastore, Security
    • Labels:
      None

      Description

      Before HIVE-1696 metastore was completely stateless and all the data is in backend db. HIVE-1696 added delegation tokens for metastore which metastore needs to keep in memory. HIVE-2467 added support for storing delegation tokens in ZooKeeper via an interface DelegationTokenStore. This jira is about using DB backend for storing tokens which metastore uses for storing data.

      1. hive-3255.patch
        28 kB
        Ashutosh Chauhan
      2. HIVE-3255.6.patch
        153 kB
        Gunther Hagleitner
      3. HIVE-3255_6.patch
        153 kB
        Ashutosh Chauhan
      4. hive-3255_5.patch
        38 kB
        Deepesh Khandelwal
      5. hive-3255_4.patch
        37 kB
        Ashutosh Chauhan
      6. hive-3255_3.patch
        36 kB
        Ashutosh Chauhan
      7. hive-3255_2.patch
        35 kB
        Ashutosh Chauhan

        Issue Links

          Activity

          Ashutosh Chauhan created issue -
          Hide
          Ashutosh Chauhan added a comment -

          A WIP patch is available for review on https://reviews.facebook.net/D4113 in case some one wants to provide a feedback on design. Some function bodies are not done yet, but those are mostly self contained and shouldn't hinder in design review.

          Show
          Ashutosh Chauhan added a comment - A WIP patch is available for review on https://reviews.facebook.net/D4113 in case some one wants to provide a feedback on design. Some function bodies are not done yet, but those are mostly self contained and shouldn't hinder in design review.
          Hide
          Ashutosh Chauhan added a comment -

          Updating with complete patch. Patch is ready for review.

          Show
          Ashutosh Chauhan added a comment - Updating with complete patch. Patch is ready for review.
          Ashutosh Chauhan made changes -
          Field Original Value New Value
          Attachment hive-3255.patch [ 12549772 ]
          Hide
          Ashutosh Chauhan added a comment -
          Show
          Ashutosh Chauhan added a comment - Review request at https://reviews.apache.org/r/7941/
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.10.0 [ 12320745 ]
          Hide
          Ashutosh Chauhan added a comment -

          Updated patch on latest trunk.

          Show
          Ashutosh Chauhan added a comment - Updated patch on latest trunk.
          Ashutosh Chauhan made changes -
          Attachment hive-3255_2.patch [ 12553216 ]
          Hide
          Ashutosh Chauhan added a comment -

          Updated patch incorporating Mark and Prasad's comments.

          Show
          Ashutosh Chauhan added a comment - Updated patch incorporating Mark and Prasad's comments.
          Ashutosh Chauhan made changes -
          Attachment hive-3255_3.patch [ 12553702 ]
          Hide
          Mark Grover added a comment -

          Looks good to me (non-committer). Thanks Ashutosh!

          Show
          Mark Grover added a comment - Looks good to me (non-committer). Thanks Ashutosh!
          Hide
          Ashutosh Chauhan added a comment -

          Patch refreshed for latest trunk. Really appreciate if some committer can take a look.

          Show
          Ashutosh Chauhan added a comment - Patch refreshed for latest trunk. Really appreciate if some committer can take a look.
          Ashutosh Chauhan made changes -
          Attachment hive-3255_4.patch [ 12555988 ]
          Hide
          Arpit Gupta added a comment -

          I tested this patch out on a cluster running Hive.

          Ran a pig script that used hcat loader to read data from a partitioned table, processed it and then stored the partitioned results in another table.

          Hive metastore service was restarted as the data was being read and processed but before the data was stored in the table. The tests passed on the cluster when the following config was present on the server

          <property>
            <name>hive.cluster.delegation.token.store.class</name>
            <value>org.apache.hadoop.hive.thrift.DBTokenStore</value>
          </property>
          

          When i ran the same test on a metastore which did not have this enabled the test failed as expected.

          Show
          Arpit Gupta added a comment - I tested this patch out on a cluster running Hive. Ran a pig script that used hcat loader to read data from a partitioned table, processed it and then stored the partitioned results in another table. Hive metastore service was restarted as the data was being read and processed but before the data was stored in the table. The tests passed on the cluster when the following config was present on the server <property> <name>hive.cluster.delegation.token.store.class</name> <value>org.apache.hadoop.hive.thrift.DBTokenStore</value> </property> When i ran the same test on a metastore which did not have this enabled the test failed as expected.
          Hide
          Deepesh Khandelwal added a comment -

          The patch would have issues on Postgres 8.4 as "CREATE IF NOT EXISTS" is not supported prior to Postgres 9.1. Attaching the patch which changes the "CREATE IF NOT EXISTS" to simply "CREATE". This is consistent with other CREATE calls in the postgres sql files.

          Show
          Deepesh Khandelwal added a comment - The patch would have issues on Postgres 8.4 as "CREATE IF NOT EXISTS" is not supported prior to Postgres 9.1. Attaching the patch which changes the "CREATE IF NOT EXISTS" to simply "CREATE". This is consistent with other CREATE calls in the postgres sql files.
          Deepesh Khandelwal made changes -
          Assignee Ashutosh Chauhan [ ashutoshc ] Deepesh Khandelwal [ deepesh ]
          Deepesh Khandelwal made changes -
          Attachment hive-3255_5.patch [ 12583188 ]
          Hide
          Ashutosh Chauhan added a comment -

          Thanks Deepesh for the feedback. Incorporated it. Updated patch rebased on trunk.

          Show
          Ashutosh Chauhan added a comment - Thanks Deepesh for the feedback. Incorporated it. Updated patch rebased on trunk.
          Ashutosh Chauhan made changes -
          Attachment HIVE-3255_6.patch [ 12594487 ]
          Hide
          Gunther Hagleitner added a comment -

          +1 looks good. Will commit if tests pass.

          Show
          Gunther Hagleitner added a comment - +1 looks good. Will commit if tests pass.
          Hide
          Gunther Hagleitner added a comment -

          Re-uploading Ashutosh' patch with different name to trigger pre-commit build. (.6 not _6)

          Show
          Gunther Hagleitner added a comment - Re-uploading Ashutosh' patch with different name to trigger pre-commit build. (.6 not _6)
          Gunther Hagleitner made changes -
          Attachment HIVE-3255.6.patch [ 12594528 ]
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12594528/HIVE-3255.6.patch

          SUCCESS: +1 2646 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/207/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/207/console

          Messages:

          Executing org.apache.hive.ptest.execution.CleanupPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594528/HIVE-3255.6.patch SUCCESS: +1 2646 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/207/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/207/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Gunther Hagleitner added a comment -

          Committed to trunk. Thanks Ashutosh and Deepesh!

          Show
          Gunther Hagleitner added a comment - Committed to trunk. Thanks Ashutosh and Deepesh!
          Gunther Hagleitner made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.12.0 [ 12324312 ]
          Resolution Fixed [ 1 ]
          Yin Huai made changes -
          Link This issue breaks HIVE-4976 [ HIVE-4976 ]
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Ashutosh Chauhan made changes -
          Assignee Deepesh Khandelwal [ deepesh ] Ashutosh Chauhan [ ashutoshc ]

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development