[HDFS-11913] Ozone: TestKeySpaceManager#testDeleteVolume fails

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDFS-7240
• Component/s: ozone
• Target Version/s: HDFS-7240
• Hadoop Flags: Reviewed
• Assignee: Weiwei Yang
• Reporter: Weiwei Yang
• Created: 02/Jun/17 04:31
• Updated: 26/Apr/18 21:50
• Resolved: 02/Jun/17 17:48

Description

HDFS-11774 introduces an UT failure, TestKeySpaceManager#testDeleteVolume, error as below

java.util.NoSuchElementException
 at org.fusesource.leveldbjni.internal.JniDBIterator.peekNext(JniDBIterator.java:84)
 at org.fusesource.leveldbjni.internal.JniDBIterator.next(JniDBIterator.java:98)
 at org.fusesource.leveldbjni.internal.JniDBIterator.next(JniDBIterator.java:45)
 at org.apache.hadoop.ozone.ksm.MetadataManagerImpl.isVolumeEmpty(MetadataManagerImpl.java:221)
 at org.apache.hadoop.ozone.ksm.VolumeManagerImpl.deleteVolume(VolumeManagerImpl.java:294)
 at org.apache.hadoop.ozone.ksm.KeySpaceManager.deleteVolume(KeySpaceManager.java:340)
 at org.apache.hadoop.ozone.protocolPB.KeySpaceManagerProtocolServerSideTranslatorPB.deleteVolume(KeySpaceManagerProtocolServerSideTranslatorPB.java:200)
 at org.apache.hadoop.ozone.protocol.proto.KeySpaceManagerProtocolProtos$KeySpaceManagerService$2.callBlockingMethod(KeySpaceManagerProtocolProtos.java:22742)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)

this is caused by a buggy code in MetadataManagerImpl#isVolumeEmpty, there are 2 issues need to be fixed

1. Iterate next element will throw this exception if it doesn't have next. This always fail when a volume is empty.
2. The code was checking if the first bucket name start with "/volume_name", this will return a wrong value if I have several empty volumes with same prefix, e.g "/volA/", "/volAA/". Such case isVolumeEmpty will return false as the next element from "/volA/" is not a bucket, it's another volume "/volAA/" but matches the prefix.

For now an empty volume with name "/volA/" is probably not valid, but if we make sure our bucket key starts with "/volA/" instead of just "/volA" is a good idea to leave us away from weird problems.