Hive
  1. Hive
  2. HIVE-2764

Obtain delegation tokens for MR jobs in secure hbase setup

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: HBase Handler, Security
    • Labels:
      None

      Description

      As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.

        Issue Links

          Activity

          Hide
          Enis Soztutar added a comment -

          We had some discussion with Ashutosh about how to best approach this. As a part of this issue, we have to obtain delegation tokens from the storage handler (hbase in this case), and then pass this to the submitted job. However, HiveStorageHandler.configureTableJobProperties() does not expose the JobConf object, or an interface to configure the job's credentials. There seems a couple of options we can go for:
          1. At HiveStorageHandler.configureTableJobProperties(), create and save the job credentials somewhere, and at the time of job submit, merge all the credentials before submitting the job.
          2. Change the HiveStorageHandler.configureTableJobProperties() to get the actual Job to be submitted, so that Storage handlers have a real shot at configuring the job.
          3. Add a method like HiveStorageHandler.configureJobCredentials(), and call it from HIF.getSplits() and HiveHBaseTableOutputFormat.checkOutputSpecs().

          Number 1 and 3 seems hackish, and we will want to give the storagehandlers ability to configure the actual job. So we are leaning towards that. But this will also be merged with HIVE-2773. I am still trying to make this work, so any comments / suggestions are more than welcome.

          Show
          Enis Soztutar added a comment - We had some discussion with Ashutosh about how to best approach this. As a part of this issue, we have to obtain delegation tokens from the storage handler (hbase in this case), and then pass this to the submitted job. However, HiveStorageHandler.configureTableJobProperties() does not expose the JobConf object, or an interface to configure the job's credentials. There seems a couple of options we can go for: 1. At HiveStorageHandler.configureTableJobProperties(), create and save the job credentials somewhere, and at the time of job submit, merge all the credentials before submitting the job. 2. Change the HiveStorageHandler.configureTableJobProperties() to get the actual Job to be submitted, so that Storage handlers have a real shot at configuring the job. 3. Add a method like HiveStorageHandler.configureJobCredentials(), and call it from HIF.getSplits() and HiveHBaseTableOutputFormat.checkOutputSpecs(). Number 1 and 3 seems hackish, and we will want to give the storagehandlers ability to configure the actual job. So we are leaning towards that. But this will also be merged with HIVE-2773 . I am still trying to make this work, so any comments / suggestions are more than welcome.
          Hide
          Enis Soztutar added a comment -

          Attaching a patch for demonstration of yet another approach of adding the Credentials to TableDesc. This patch is still WIP.

          I have tested this with fully secure hadoop +hbase. Compiles with -Dhadoop.version=1.0.0

          Show
          Enis Soztutar added a comment - Attaching a patch for demonstration of yet another approach of adding the Credentials to TableDesc. This patch is still WIP. I have tested this with fully secure hadoop +hbase. Compiles with -Dhadoop.version=1.0.0
          Hide
          Francis Liu added a comment -

          Having making getCredentials() exposed as a TableDesc member sounds ok to me as it really is a special property of JobConf.

          Though it breaks the expectation of tableDesc being readonly and jobProperties being the only one mutable for configureTableProperties(). I wonder if we can remove jobProperties altogether and use tableDesc.jobProperties to set properties. As tableDesc.jobProperties is always a new HashMap when configureTableProperties() is called?

          Show
          Francis Liu added a comment - Having making getCredentials() exposed as a TableDesc member sounds ok to me as it really is a special property of JobConf. Though it breaks the expectation of tableDesc being readonly and jobProperties being the only one mutable for configureTableProperties(). I wonder if we can remove jobProperties altogether and use tableDesc.jobProperties to set properties. As tableDesc.jobProperties is always a new HashMap when configureTableProperties() is called?
          Hide
          Phabricator added a comment -

          enis requested code review of "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
          Reviewers: JIRA

          HIVE-2764. Obtain delegation tokens for MR jobs in secure hbase setup.Title:

          HIVE-2764. Obtain delegation tokens for MR jobs in secure hbase setup.

          As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.

          TEST PLAN
          Tested manually by setting up a kerberos cluster with hadoop
          + hbase security enabled.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          AFFECTED FILES
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/4839/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - enis requested code review of " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Reviewers: JIRA HIVE-2764 . Obtain delegation tokens for MR jobs in secure hbase setup.Title: HIVE-2764 . Obtain delegation tokens for MR jobs in secure hbase setup. As discussed in HCATALOG-244 , in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. TEST PLAN Tested manually by setting up a kerberos cluster with hadoop + hbase security enabled. REVISION DETAIL https://reviews.facebook.net/D2205 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/4839/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Phabricator added a comment -

          enis has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".

          Some comments about the patch:
          Changing the StorageHandler.configureJobProperties() to be called with the actual job is kind of intrusive, so I have given up on that approach, and go with the plan of obtaining the delegation tokens from the IF.getSplits() and OF.checkOutputSpecs() time. However, one problem with this approach is that although on the input side it works fine, on the output side, Hive does not use OF's properly.

          So, this patch adds a HiveOutputFormatImpl class, and changes the HOF to extend OF. The actual usage of OFs are not changed as a part of this patch, since it is out of the scope of this issue. However, not the HOFImpl class delgates the checkOutputSpecs() call to the underlying OutputFormat, so that if the OF wants to do any initialization, it can.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          Show
          Phabricator added a comment - enis has commented on the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Some comments about the patch: Changing the StorageHandler.configureJobProperties() to be called with the actual job is kind of intrusive, so I have given up on that approach, and go with the plan of obtaining the delegation tokens from the IF.getSplits() and OF.checkOutputSpecs() time. However, one problem with this approach is that although on the input side it works fine, on the output side, Hive does not use OF's properly. So, this patch adds a HiveOutputFormatImpl class, and changes the HOF to extend OF. The actual usage of OFs are not changed as a part of this patch, since it is out of the scope of this issue. However, not the HOFImpl class delgates the checkOutputSpecs() call to the underlying OutputFormat, so that if the OF wants to do any initialization, it can. REVISION DETAIL https://reviews.facebook.net/D2205
          Hide
          Phabricator added a comment -

          enis updated the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
          Reviewers: JIRA

          Realized that HiveHFileOutputFormat does not need delegation tokens. Duh!

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          AFFECTED FILES
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
          shims/src/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java

          Show
          Phabricator added a comment - enis updated the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Reviewers: JIRA Realized that HiveHFileOutputFormat does not need delegation tokens. Duh! REVISION DETAIL https://reviews.facebook.net/D2205 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java shims/src/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
          Hide
          Phabricator added a comment -

          ashutoshc has requested changes to the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".

          Added comments. Also, patch needs to be rebased.

          INLINE COMMENTS
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java:104 I understand from this patch that checkOutputSpecs() is never called. But, see an existing impl here. So, then before this patch, this code was never getting invoked ?
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java:214 I don't think you want to override this function and leave a no-op here. I think you still want super-class checkOS to be called. Isn't it ?
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:53 This is not supposed to get called, right ? In that case this should throw exception, instead of creating object and returning it.
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:45-49 I dont' see any use of this, you can as well keep it in local variable in checkOS()
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:75 Instead of throwing Runtime, you should throw IOException which method signature allows you to do.
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 This results in setting of NullOF in all the shims. We should no longer do that, since we are resetting it again in following line.
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:769 Is this null check necessary? At the time, checkOS is called, hiveOF would already have been initialized, no?
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java:27 Looks like this import statement inadvertently got added.
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 To maintain backward-compat you should first check whether hiveOF has the method checkOS, and call it only if it has.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          BRANCH
          hbase-dt-of

          Show
          Phabricator added a comment - ashutoshc has requested changes to the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Added comments. Also, patch needs to be rebased. INLINE COMMENTS hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java:104 I understand from this patch that checkOutputSpecs() is never called. But, see an existing impl here. So, then before this patch, this code was never getting invoked ? hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java:214 I don't think you want to override this function and leave a no-op here. I think you still want super-class checkOS to be called. Isn't it ? ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:53 This is not supposed to get called, right ? In that case this should throw exception, instead of creating object and returning it. ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:45-49 I dont' see any use of this, you can as well keep it in local variable in checkOS() ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:75 Instead of throwing Runtime, you should throw IOException which method signature allows you to do. ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 This results in setting of NullOF in all the shims. We should no longer do that, since we are resetting it again in following line. ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:769 Is this null check necessary? At the time, checkOS is called, hiveOF would already have been initialized, no? ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java:27 Looks like this import statement inadvertently got added. ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 To maintain backward-compat you should first check whether hiveOF has the method checkOS, and call it only if it has. REVISION DETAIL https://reviews.facebook.net/D2205 BRANCH hbase-dt-of
          Hide
          Phabricator added a comment -

          enis has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".

          INLINE COMMENTS
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java:104 Yes, that is correct. It was never invoked.
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 HadoopShims.setNullOutputFormat() not only sets the OF, but sets the null output committer, etc. So we still need that call.
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:769 checkOS() is called at the job submission time from the client, but FileSinkOperator.initializeOp() is called from the task context when the job is running. So we need to initialize the output format.
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 I though HOF was hive-private. You are saying that, we need this for maintaining binary BC? We are also changing the HOF to extent OF, so all HOF implementations have to be fixed (to include the OF methods) and recompiled.

          If for binary BC, I can use reflection to check whether the method is there.
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java:27 It is a rearrange for the import statement.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          BRANCH
          hbase-dt-of

          Show
          Phabricator added a comment - enis has commented on the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". INLINE COMMENTS hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java:104 Yes, that is correct. It was never invoked. ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 HadoopShims.setNullOutputFormat() not only sets the OF, but sets the null output committer, etc. So we still need that call. ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:769 checkOS() is called at the job submission time from the client, but FileSinkOperator.initializeOp() is called from the task context when the job is running. So we need to initialize the output format. ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 I though HOF was hive-private. You are saying that, we need this for maintaining binary BC? We are also changing the HOF to extent OF, so all HOF implementations have to be fixed (to include the OF methods) and recompiled. If for binary BC, I can use reflection to check whether the method is there. ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java:27 It is a rearrange for the import statement. REVISION DETAIL https://reviews.facebook.net/D2205 BRANCH hbase-dt-of
          Hide
          Phabricator added a comment -

          ashutoshc has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 Sorry, I wasn't clear. What I meant was we should remove setting of OF class inside setNullOF() method, because its redundant as we set it again later in this method.
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 Yes, please use reflection to check if method is there and then only call.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          BRANCH
          hbase-dt-of

          Show
          Phabricator added a comment - ashutoshc has commented on the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 Sorry, I wasn't clear. What I meant was we should remove setting of OF class inside setNullOF() method, because its redundant as we set it again later in this method. ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 Yes, please use reflection to check if method is there and then only call. REVISION DETAIL https://reviews.facebook.net/D2205 BRANCH hbase-dt-of
          Hide
          Phabricator added a comment -

          enis has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:53 Hadoop framework calls the getRR() although we do not emit any records. So we need some kind of an empty implementation.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          BRANCH
          hbase-dt-of

          Show
          Phabricator added a comment - enis has commented on the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:53 Hadoop framework calls the getRR() although we do not emit any records. So we need some kind of an empty implementation. REVISION DETAIL https://reviews.facebook.net/D2205 BRANCH hbase-dt-of
          Hide
          Phabricator added a comment -

          enis updated the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
          Reviewers: JIRA, ashutoshc

          incorporated review suggestions:

          • change HadoopShims.setNullOutputFormat() to .prepareJobOutput()
          • BC check for HiveOutputFormat.checkOutputSpecs() call.
          • HiveHFileOutputFormat.checkOS() now delegates to underlying OF.
          • minor changes per review.
          • running tests right now.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          AFFECTED FILES
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
          shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java

          Show
          Phabricator added a comment - enis updated the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Reviewers: JIRA, ashutoshc incorporated review suggestions: change HadoopShims.setNullOutputFormat() to .prepareJobOutput() BC check for HiveOutputFormat.checkOutputSpecs() call. HiveHFileOutputFormat.checkOS() now delegates to underlying OF. minor changes per review. running tests right now. REVISION DETAIL https://reviews.facebook.net/D2205 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Hide
          Phabricator added a comment -

          ashutoshc has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".

          Changes look good. But, I see plenty of failures while running tests.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          Show
          Phabricator added a comment - ashutoshc has commented on the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Changes look good. But, I see plenty of failures while running tests. REVISION DETAIL https://reviews.facebook.net/D2205
          Hide
          Phabricator added a comment -

          enis has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".

          Strange, all of the tests passed for me except for one. I rerun that and it passed as well. Can you rerun the tests or post the logs ?

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          Show
          Phabricator added a comment - enis has commented on the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Strange, all of the tests passed for me except for one. I rerun that and it passed as well. Can you rerun the tests or post the logs ? REVISION DETAIL https://reviews.facebook.net/D2205
          Hide
          Ashutosh Chauhan added a comment -

          In TestCliDriver, following tests failed:

          • alter_concatenate_indexed_table.q
          • alter_merge.q
          • alter_merge_2.q
          • alter_merge_stats.q
          • concatenate_inherit_table_location.q
          • create_merge_compressed.q
          • escape2.q

          I re-ran them again individually, and they again failed.

          Show
          Ashutosh Chauhan added a comment - In TestCliDriver, following tests failed: alter_concatenate_indexed_table.q alter_merge.q alter_merge_2.q alter_merge_stats.q concatenate_inherit_table_location.q create_merge_compressed.q escape2.q I re-ran them again individually, and they again failed.
          Hide
          Phabricator added a comment -

          enis updated the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
          Reviewers: JIRA, ashutoshc

          Fixed failing tests

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          AFFECTED FILES
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
          shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java

          Show
          Phabricator added a comment - enis updated the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Reviewers: JIRA, ashutoshc Fixed failing tests REVISION DETAIL https://reviews.facebook.net/D2205 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Hide
          Ashutosh Chauhan added a comment -

          Those tests passed. So, thats a progress. But following new ones failed.

          • rcfile_merge3.q
          • rcfile_createas1.q
          Show
          Ashutosh Chauhan added a comment - Those tests passed. So, thats a progress. But following new ones failed. rcfile_merge3.q rcfile_createas1.q
          Hide
          Phabricator added a comment -

          enis updated the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
          Reviewers: JIRA, ashutoshc

          Fixed one more issue with the tests. All tests pass now.

          REVISION DETAIL
          https://reviews.facebook.net/D2205

          AFFECTED FILES
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
          shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java

          Show
          Phabricator added a comment - enis updated the revision " HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup". Reviewers: JIRA, ashutoshc Fixed one more issue with the tests. All tests pass now. REVISION DETAIL https://reviews.facebook.net/D2205 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Hide
          Ashutosh Chauhan added a comment -

          Checked-in trunk. Thanks, Enis!

          Show
          Ashutosh Chauhan added a comment - Checked-in trunk. Thanks, Enis!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1364 (See https://builds.apache.org/job/Hive-trunk-h0.21/1364/)
          HIVE-2764: Obtain delegation tokens for MR jobs in secure hbase setup (Enis Soztutar via Ashutosh Chauhan) (Revision 1311418)

          Result = SUCCESS
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311418
          Files :

          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1364 (See https://builds.apache.org/job/Hive-trunk-h0.21/1364/ ) HIVE-2764 : Obtain delegation tokens for MR jobs in secure hbase setup (Enis Soztutar via Ashutosh Chauhan) (Revision 1311418) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311418 Files : /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Hide
          Ashutosh Chauhan added a comment -

          This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

          Show
          Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-2764: Obtain delegation tokens for MR jobs in secure hbase setup (Enis Soztutar via Ashutosh Chauhan) (Revision 1311418)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311418
          Files :

          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          • /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2764 : Obtain delegation tokens for MR jobs in secure hbase setup (Enis Soztutar via Ashutosh Chauhan) (Revision 1311418) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311418 Files : /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java

            People

            • Assignee:
              Enis Soztutar
              Reporter:
              Enis Soztutar
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development