Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2940

HBaseStorage store fails in secure cluster

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11, 0.10.1
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      To reproduce ths issue, please do the following in secure hadoop/hbase cluster:

      1. On a gateway node, run kinit to obtain kerberos credentials and run a Pig script that includes a HBaseStorage load/store.
      2. In the front-end, HBaseStorage obtains a delegation token from hbase server and adds it to the JobConf object.
      3. In the back-end, mappers connect to hbase using the delegation token w/o kerberos credentials.

      While load-from-hbase works perfectly fine, store-to-hbase fails. This is because at step 3, mappers attempt to obtain a delegation token from hbase in the back-end.

      setStoreLocation()
      // Not setting a udf property and getting the hbase delegation token
      // only once like in setLocation as setStoreLocation gets different Job
      // objects for each call and the last Job passed is the one that is
      // launched. So we end up getting multiple hbase delegation tokens.
      addHBaseDelegationToken(m_conf, job);
      

      The problem is that mappers in the back-end don't have kerberos credentials, so the call to addHBaseDelegationToken() fails with the following error:

      2012-09-30 14:33:42,310 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:testuser (auth:SIMPLE) cause:org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients
      	at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)
      

      This is not an issue with load because a delegation token is only obtained in the front-end for the first time when HBASE_TOKEN_SET is not set.

      setLocation()
      String delegationTokenSet = udfProps.getProperty(HBASE_TOKEN_SET);
      if (delegationTokenSet == null) {
          addHBaseDelegationToken(m_conf, job);
          udfProps.setProperty(HBASE_TOKEN_SET, "true");
      }
      

      The proposed fix is to modify addHBaseDelegationToken() so that tokens are obtained only if the current user has kerberos credentials, which is true in the front-end while false in the back-end.

        Attachments

        1. container_log
          207 kB
          Cheolsoo Park
        2. PIG-2940.patch
          2 kB
          Cheolsoo Park
        3. PIG-2940-2.patch
          0.7 kB
          Cheolsoo Park

        Issue Links

          Activity

            People

            • Assignee:
              cheolsoo Cheolsoo Park
              Reporter:
              cheolsoo Cheolsoo Park

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment