[HDFS-13112] Token expiration edits may cause log corruption or deadlock

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.0-beta, 0.23.8
• Fix Version/s: 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
• Component/s: namenode
• Target Version/s: 3.0.1, 2.7.6
• Hadoop Flags: Reviewed
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 06/Feb/18 17:32
• Updated: 19/Mar/18 18:31
• Resolved: 15/Feb/18 21:51

Description

HDFS-4477 specifically did not acquire the fsn lock during token cancellation based on the belief that edit logs are thread-safe. However, log rolling is not thread-safe. Failure to externally synchronize on the fsn lock during a roll will cause problems.

For sync edit logging, it may cause corruption by interspersing edits with the end/start segment edits. Async edit logging may encounter a deadlock if the log queue overflows. Luckily, losing the race is extremely rare. In ~5 years, we've never encountered it. However, HDFS-13051 lost the race with async edits.

Issue Links

• is caused by: HDFS-4477 Secondary namenode may retain old tokens
• is related to: HDFS-13051 dead lock occurs when rolleditlog rpc call happen and editPendingQ is full
• requires: HADOOP-15212 Add independent secret manager method for logging expired tokens