Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
3.0.1
-
None
-
None
Description
HDp3.0.1 deployed on DellEMC Isilon OneFS 8.1.2
The ats-hbase app is up and running but flooding the OneFS hdfs and ats-hbase region server logs with below message.
OneFS log
2018-11-16T04:29:44Z <30.6> kbhusan-y93o5ew-1 hdfs[2711]: [hdfs] RPC V9 complete user: yarn-ats exception: java.io.FileNotFoundException cause: File not found: /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.1542088499176 <FS Exception at CreateHandleFromPath():448 status: STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 Path: atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.1542088499176>
2018-11-16T04:29:45Z <30.6> kbhusan-y93o5ew-1 hdfs[2711]: [hdfs] RPC V9 complete user: yarn-ats exception: java.io.FileNotFoundException cause: File not found: /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.meta.1542185701850.meta <FS Exception at CreateHandleFromPath():448 status: STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 Path: atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.meta.1542185701850.meta>
hbase-yarn-ats-regionserver-RDUVNODE334518.log.
2018-11-13 03:23:30,178 WARN [Close-WAL-Writer-22] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1541663220372/rduvnode334518.west.isilon.com%2C17020%2C1541663220372.1541742426412 failed, retry = 16586
org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File not found: /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1541663220372/rduvnode334518.west.isilon.com%2C17020%2C1541663220372.1541742426412
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.complete(Unknown Source)
--------------------
(each logs are more then 250MB unable to upload