[HDFS-14618] Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe).

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Labels: fix-provided patch-available
• Hadoop Flags: Reviewed
• Flags: Patch
• Assignee: Paul Ward
• Reporter: Paul Ward
• Created: 28/Jun/19 18:50
• Updated: 02/Oct/19 22:09
• Resolved: 28/Jun/19 23:07

Description

I submitted a CR for this issue at:

https://github.com/apache/hadoop/pull/1030

The field timedOutItems (an ArrayList, i.e., not thread safe):

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReconstructionBlocks.java#L70

is protected by synchronization on itself (timedOutItems):

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReconstructionBlocks.java#L167-L168

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReconstructionBlocks.java#L267-L268

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReconstructionBlocks.java#L178

However, in one place:

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReconstructionBlocks.java#L133-L135

it is (trying to be) protected by synchronized using pendingReconstructions — but this cannot protect timedOutItems.

Synchronized on different objects does not ensure mutual exclusion with the other locations.

I.e., 2 code locations, one synchronized by pendingReconstructions and the other by timedOutItems can still executed concurrently.

This CR adds the synchronized on timedOutItems.

Note that this CR keeps the synchronized on pendingReconstructions, which is needed for a different purpose (protect pendingReconstructions)

Issue Links

• links to: PR1030