[HDFS-13284] Adjust criteria for LowRedundancyBlocks.QUEUE_VERY_LOW_REDUNDANCY

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: hdfs, namenode
• Assignee: Lukas Majercak
• Reporter: Lukas Majercak
• Created: 14/Mar/18 22:51
• Updated: 14/May/18 22:49

Description

LowRedundancyBlocks currently has 5 priority queues:

QUEUE_HIGHEST_PRIORITY = 0                         - reserved for last replica blocks
QUEUE_VERY_LOW_REDUNDANCY = 1             - if ((curReplicas * 3) < expectedReplicas)
QUEUE_LOW_REDUNDANCY = 2                         - the rest
QUEUE_REPLICAS_BADLY_DISTRIBUTED = 3
QUEUE_WITH_CORRUPT_BLOCKS = 4

The problem lies in  QUEUE_VERY_LOW_REDUNDANCY. Currently, a block that has curReplicas=2 and expectedReplicas=4 is treated the same as a block with curReplicas=3 and expectedReplicas=4. A block with 2/3 replicas is also put into QUEUE_LOW_REDUNDANCY. 

The proposal is to change the if ((curReplicas * 3) < expectedReplicas) check to if ((curReplicas * 2) <= expectedReplicas || curReplicas == 2)