[HDFS-11900] Hedged reads thread pool creation not synchronized

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 3.2.0, 3.1.1
• Component/s: hdfs-client
• Target Version/s: 3.2.0
• Assignee: John Zhuge
• Reporter: John Zhuge
• Created: 29/May/17 09:44
• Updated: 13/Apr/18 17:34
• Resolved: 22/Mar/18 18:32

Description

Non-static synchronized method initThreadsNumForHedgedReads can't synchronize the access to the static class variable HEDGED_READ_THREAD_POOL.

  private static ThreadPoolExecutor HEDGED_READ_THREAD_POOL;
...
  private synchronized void initThreadsNumForHedgedReads(int num) {

2 DFS clients may update the same static variable in a race because the lock is on each DFS client object, not on the shared DFSClient class object.

There are 2 possible fixes:
1. "Global thread pool": Change initThreadsNumForHedgedReads to static
2. "Per-client thread pool": Change HEDGED_READ_THREAD_POOL to non-static

From the description for property dfs.client.hedged.read.threadpool.size:

to a positive number. The threadpool size is how many threads to dedicate
to the running of these 'hedged', concurrent reads in your client.

it seems to indicate the thread pool is per DFS client.

Let's assume we go with #1 "Global thread pool". One DFS client has the property set to 10 in its config, while the other client has the property set to 5 in its config, what is supposed to the size of the global thread pool? 5? 10? Or 15?

The 2nd fix seems more reasonable to me.

Issue Links

• is related to: HDFS-11898 DFSClient#isHedgedReadsEnabled() should be per client flag