Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26432

Not able to connect Hbase 2.1 service Getting NoSuchMethodException while trying to obtain token from Hbase 2.1 service.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.2, 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Getting NoSuchMethodException :

      org.apache.hadoop.hbase.security.token.TokenUtil(org.apache.hadoop.conf.Configuration)

      while trying  connect hbase 2.1 service from spark.

      This is mainly happening because in spark uses  a deprecated hbase api 

      public static Token<AuthenticationTokenIdentifier> obtainToken(Configuration conf)  

      for obtaining the token and the same has been removed from hbase 2.1 version.

       Test steps:

      Steps to test Spark-Hbase connection

      1. Create 2 tables in hbase shell
      >Launch hbase shell
      >Enter commands to create tables and load data
      create 'table1','cf'
      put 'table1','row1','cf:cid','20'

      create 'table2','cf'
      put 'table2','row1','cf:cid','30'

      >Show values command
      get 'table1','row1','cf:cid' will diplay value as 20
      get 'table2','row1','cf:cid' will diplay value as 30

      2.Run SparkHbasetoHbase class in testSpark.jar using spark-submit

      spark-submit --master yarn-cluster --class com.mrs.example.spark.SparkHbasetoHbase --conf "spark.yarn.security.credentials.hbase.enabled"="true" --conf "spark.security.credentials.hbase.enabled"="true" --keytab /opt/client/user.keytab --principal sen testSpark.jar

      The SparkHbasetoHbase class will update the value of table2 with sum of values of table1 & table2.

      table2 = table1+table2

       

      3.Verify the result in hbase shell

      Expected Result: The value of table2 should be 50.

      get 'table1','row1','cf:cid'  will diplay value as 50

      Actual Result : Not updating the value as an error will be thrown when spark tries to connect with hbase service.

      Attached the snapshot of error logs below for more details

        Attachments

        1. hbase-dep-obtaintok.png
          73 kB
          Sujith Chacko

          Issue Links

            Activity

              People

              • Assignee:
                S71955 Sujith Chacko
                Reporter:
                S71955 Sujith Chacko
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: