[HDFS-14305] Serial number in BlockTokenSecretManager could overlap between different namenodes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: namenode, security
• Labels: multi-sbnn
• Target Version/s: 3.5.0
• Hadoop Flags: Reviewed

Release Note

NameNodes rely on independent block token key ranges to communicate block token identities to DataNodes and clients in a way that does not create conflicts between the tokens issued by multiple NameNodes. HDFS-6440 introduced the potential for overlaps in key ranges; this fixes the issue by creating 64 possible key ranges that NameNodes assign themselves to, allowing for up to 64 NameNodes to run safely. This limitation only applies within a single Namespace; there may be more than 64 NameNodes total spread among multiple federated Namespaces.

• Assignee: Konstantin Shvachko
• Reporter: Chao Sun
• Created: 21/Feb/19 18:19
• Updated: 05/Jan/24 21:00
• Resolved: 05/Jan/24 21:00

Description

Currently, a BlockTokenSecretManager starts with a random integer as the initial serial number, and then use this formula to rotate it:

    this.intRange = Integer.MAX_VALUE / numNNs;
    this.nnRangeStart = intRange * nnIndex;
    this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
 

while numNNs is the total number of NameNodes in the cluster, and nnIndex is the index of the current NameNode specified in the configuration dfs.ha.namenodes.<nameservice>.

However, with this approach, different NameNode could have overlapping ranges for serial number. For simplicity, let's assume Integer.MAX_VALUE is 100, and we have 2 NameNodes nn1 and nn2 in configuration. Then the ranges for these two are:

nn1 -> [-49, 49]
nn2 -> [1, 99]

This is because the initial serial number could be any negative integer.

Moreover, when the keys are updated, the serial number will again be updated with the formula:

this.serialNo = (this.serialNo % intRange) + (nnRangeStart);

which means the new serial number could be updated to a range that belongs to a different NameNode, thus increasing the chance of collision again.

When the collision happens, DataNodes could overwrite an existing key which will cause clients to fail because of InvalidToken error.

Issue Links

• is caused by: HDFS-6440 Support more than 2 NameNodes
• supercedes: HDFS-14793 BlockTokenSecretManager should LOG block token range it operates on.