[SPARK-26432] Not able to connect Hbase 2.1 service Getting NoSuchMethodException while trying to obtain token from Hbase 2.1 service.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.2, 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Sujith Chacko
• Reporter: Sujith Chacko
• Created: 23/Dec/18 17:34
• Updated: 28/Jan/19 18:12
• Resolved: 28/Jan/19 18:08

Description

Getting NoSuchMethodException :

org.apache.hadoop.hbase.security.token.TokenUtil(org.apache.hadoop.conf.Configuration)

while trying  connect hbase 2.1 service from spark.

This is mainly happening because in spark uses  a deprecated hbase api 

public static Token<AuthenticationTokenIdentifier> obtainToken(Configuration conf)  

for obtaining the token and the same has been removed from hbase 2.1 version.

 Test steps:

Steps to test Spark-Hbase connection

1. Create 2 tables in hbase shell
>Launch hbase shell
>Enter commands to create tables and load data
create 'table1','cf'
put 'table1','row1','cf:cid','20'

create 'table2','cf'
put 'table2','row1','cf:cid','30'

>Show values command
get 'table1','row1','cf:cid' will diplay value as 20
get 'table2','row1','cf:cid' will diplay value as 30

2.Run SparkHbasetoHbase class in testSpark.jar using spark-submit

spark-submit --master yarn-cluster --class com.mrs.example.spark.SparkHbasetoHbase --conf "spark.yarn.security.credentials.hbase.enabled"="true" --conf "spark.security.credentials.hbase.enabled"="true" --keytab /opt/client/user.keytab --principal sen testSpark.jar

The SparkHbasetoHbase class will update the value of table2 with sum of values of table1 & table2.

table2 = table1+table2

 

3.Verify the result in hbase shell

Expected Result: The value of table2 should be 50.

get 'table1','row1','cf:cid'  will diplay value as 50

Actual Result : Not updating the value as an error will be thrown when spark tries to connect with hbase service.

Attached the snapshot of error logs below for more details

Issue Links

• links to: GitHub Pull Request #23429