Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33182

Using Delegation Tokens to access HBase through Spark (Java)

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
    • Environment:

      Spark 2.3, RHEL 7.5, HDP 3

      Description

      I have a requirement to access a kerberized HBase through delegation tokens instead of keytab. I have generated the token using the token util API and also loaded it back into the UserGroupInformation. However, hasKerberosCredentials returns a false. I am looking for the  the right way to use a saved delegation token to access HBase . The code that I have currently looks like shown below. I see the error :   

      client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=4850 ms ago, cancelled=false, msg=Connection closed, details=row 'employee,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=myhost.com,16020,1602758016346, seqNum=-1

       

      UserGroupInformation ugi;
       ugi = UserGroupInformation.getLoginUser();
       Credentials creds = new Credentials ();
       creds = Credentials.readTokenStorageFile (new Path("file:///zen-volume-home/tokenFile"),conf);
       
       System.out.println("TOKEN *********" + creds.getToken(new org.apache.hadoop.io.Text("hbase")));
       ugi.addToken(creds.getToken(new org.apache.hadoop.io.Text("hbase")));
       ugi.addCredentials(creds);
      
      /* I do see the token getting printed. However, I am looking for information on how to use this ugi which has the token added to it, to access data on HBase.
      I have tried doing this :  */
      
      SQLContext sqlC = ugi.getLoginUser().doAs((new PrivilegedExceptionAction<SQLContext>() {
      		            public SQLContext run() throws Exception {
      		     
      		     sparkconf = new SparkConf().setAppName("Hbase With Spark");
      		     jsc = new JavaSparkContext(sparkconf);
      		     hbaseContext = new JavaHBaseContext(jsc, conf);
      		 sqlContext = new org.apache.spark.sql.SQLContext(jsc);		
      		        		
      		 String sqlMapping  = "name STRING all:name" ;	
      		 Dataset<Row> dfEmp = sqlContext.read().format("org.apache.hadoop.hbase.spark")
      		        				.option("hbase.columns.mapping", sqlMapping)		        				.option("hbase.table", "employee").load();
      		        		        
      		        		dfEmp.registerTempTable("empdata");  
      		        		dfEmp.show();
      		        		return sqlContext;
      		            }
      		        }));
      		        		

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rishis Rishi S Balajis
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: