Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9029

yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 3.0.1
    • Fix Version/s: None
    • Component/s: ATSv2
    • Labels:
      None
    • Environment:

      Ambari 2.7.1

      HDP 3.0.1

      Isilon OneFS 8.1.2

      Description

      Background on ats-hbase start success:: We had setStoragePolicy exception on OneFS for yarn-ats hbase region server and below messages respectively on HDP and OneFS cluster.

      HDP Host è /var/log/hadoop-yarn/embedded-yarn-ats-hbase/

      2018-11-02 05:07:21,027 WARN  [StoreOpener-1588230740-1] util.CommonFSUtils: Unable to set storagePolicy=HOT for path=hdfs://kb-hdp-1.west.isilon.com:8020/atsv2/hbase/data/data/hbase/meta/1588230740/info. DEBUG log level might have more details.

      java.lang.reflect.InvocationTargetException

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

       

      OneFS HDFS log for (unsecured)

      2018-11-01T10:25:19Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 

      user: yarn-ats exception: org.apache.hadoop.ipc.RpcNoSuchMethodException

      cause: Unknown RPC: setStoragePolicy

       

      OneFS HDFS log for (secured)

      2018-11-01T12:39:56Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 

      user: yarn-ats-hbase/krb-hdp301.solarch.emc.com@SOLARCH.EMC.COM exception:

      org.apache.hadoop.ipc.RpcNoSuchMethodException cause: Unknown RPC:

      setStoragePolicy

       

       

      As a resolution, we set below 2 properties in custom hbase-site.xml file and setstoragePolicy exception got resolved.

       Custom yarn-hbase-site.xml

      'hbase.wal.storage.policy' to ‘NONE’

      ‘hbase.hstore.block.storage.policy’ to ‘NONE’

       

       

      *********

      NOW, we have a new issue that is OneFS HDFS log and /var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-***.log gets flooded with below error messages.*OneFS HDFS Log

      -------------------

      2018-11-06T06:25:43Z <30.6> kbhusan-edl9tdc-1 hdfs[2542]: [hdfs] RPC V9

      complete user:

      yarn-ats-hbase/rduvnode343393.west.isilon.com@WEST.ISILON.COM exception:

      java.io.FileNotFoundException cause: File not found:

      /atsv2/hbase/data/WALs/rduvnode343393.west.isilon.com,17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta

      <FS Exception at CreateHandleFromPath():448 status:

      STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 Path:

      atsv2/hbase/data/WALs/rduvnode343393.west.isilon.com,17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta>

      ---------------

       /var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-**.log

      --------------

      2018-11-06 04:01:12,863 INFO  [LruBlockCacheStatsExecutor]

      hfile.LruBlockCache: totalSize=1.12 MB, freeSize=1.25 GB, max=1.25 GB,

      blockCount=24, accesses=287, hits=263, hitRatio=91.64%, ,

      cachingAccesses=287, cachingHits=263, cachingHitsRatio=91.64%,

      evictions=419, evicted=0, evictedPerRun=0.0

       

      2018-11-06 04:01:48,763 WARN 

      LeaseRenewer:[yarn-ats@kb-hdp301-2.west.isilon.com impl.LeaseRenewer:

      Failed to renew lease for [DFSClient_NONMAPREDUCE_-533598397_1] for 30

      seconds.  Will retry shortly ...

       

      org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):

      Client auth method 'SIMPLE', mechanism '' is not supported on zone:1

      auth:'kerberos_only'

       

              at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)

       

              at org.apache.hadoop.ipc.Client.call(Client.java:1443)

       

              at org.apache.hadoop.ipc.Client.call(Client.java:1353)

       

                             “

                             “

                             “

                             “

      2018-11-06 12:51:02,843 WARN  [Close-WAL-Writer-1] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file /atsv2/hbase/data/WALs/rduvnode343393.west.isilon.com,17020,1541488822956/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541488822956.1541492429802 failed, retry = 637

      org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File not found: /atsv2/hbase/data/WALs/rduvnode343393.west.isilon.com,17020,1541488822956/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541488822956.1541492429802

              at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)

              at org.apache.hadoop.ipc.Client.call(Client.java:1443)

              at org.apache.hadoop.ipc.Client.call(Client.java:1353)

       

      --------------

      On checking the ats-hbase app logs, found hbase region server is trying to flush the old edits to free the WAL and it fails due to authentication issue. 

      Why yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kbhusanurmath Kirankumar Bhusanurmath
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: