[HBASE-20404] Ugly cleanerchore complaint that dir is not empty

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.4, 2.0.0
• Fix Version/s: 3.0.0-alpha-1, 2.1.0, 1.3.3, 1.4.4, 2.0.1
• Component/s: master
• Assignee: Sean Busbey
• Reporter: Michael Stack
• Created: 12/Apr/18 23:25
• Updated: 01/Feb/19 20:05
• Resolved: 17/Apr/18 18:21

Description

I see these big dirty exceptions in my master log during a long-run.... Lets clean them up (Are they exceptions I as an operator can actually do something about? Are they 'problems'? Should they be LOG.warn?)

2018-04-12 16:02:09,911 WARN  [ForkJoinPool-1-worker-15] cleaner.CleanerChore: Could not delete dir under hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/1e24549061df3adc4858fbcaf1929553/meta; {}
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException): `/hbase/archive/data/default/IntegrationTestBigLinkedList/1e24549061df3adc4858fbcaf1929553/meta is non empty': Directory is not empty
  at org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:115)
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:2848)
  at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:1048)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:641)
  at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)

  at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
  at org.apache.hadoop.ipc.Client.call(Client.java:1435)
  at org.apache.hadoop.ipc.Client.call(Client.java:1345)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
  at com.sun.proxy.$Proxy26.delete(Unknown Source)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:568)
  at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
  at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
  at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
  at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
  at com.sun.proxy.$Proxy27.delete(Unknown Source)
  at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)

...

Looks like log format is off too...