Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2940

HBaseStorage store fails in secure cluster



    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.11, 0.10.1
    • None
    • Reviewed


      To reproduce ths issue, please do the following in secure hadoop/hbase cluster:

      1. On a gateway node, run kinit to obtain kerberos credentials and run a Pig script that includes a HBaseStorage load/store.
      2. In the front-end, HBaseStorage obtains a delegation token from hbase server and adds it to the JobConf object.
      3. In the back-end, mappers connect to hbase using the delegation token w/o kerberos credentials.

      While load-from-hbase works perfectly fine, store-to-hbase fails. This is because at step 3, mappers attempt to obtain a delegation token from hbase in the back-end.

      // Not setting a udf property and getting the hbase delegation token
      // only once like in setLocation as setStoreLocation gets different Job
      // objects for each call and the last Job passed is the one that is
      // launched. So we end up getting multiple hbase delegation tokens.
      addHBaseDelegationToken(m_conf, job);

      The problem is that mappers in the back-end don't have kerberos credentials, so the call to addHBaseDelegationToken() fails with the following error:

      2012-09-30 14:33:42,310 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:testuser (auth:SIMPLE) cause:org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients
      	at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)

      This is not an issue with load because a delegation token is only obtained in the front-end for the first time when HBASE_TOKEN_SET is not set.

      String delegationTokenSet = udfProps.getProperty(HBASE_TOKEN_SET);
      if (delegationTokenSet == null) {
          addHBaseDelegationToken(m_conf, job);
          udfProps.setProperty(HBASE_TOKEN_SET, "true");

      The proposed fix is to modify addHBaseDelegationToken() so that tokens are obtained only if the current user has kerberos credentials, which is true in the front-end while false in the back-end.


        1. PIG-2940.patch
          2 kB
          Cheolsoo Park
        2. container_log
          207 kB
          Cheolsoo Park
        3. PIG-2940-2.patch
          0.7 kB
          Cheolsoo Park

        Issue Links



              cheolsoo Cheolsoo Park
              cheolsoo Cheolsoo Park
              0 Vote for this issue
              5 Start watching this issue