[HDFS-16550] [SBN read] Improper cache-size for journal node may cause cluster crash

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: journal-node
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Tao Li
• Reporter: Tao Li
• Created: 21/Apr/22 02:10
• Updated: 11/Feb/24 23:53
• Resolved: 30/Nov/22 15:54

Description

When we introduced SBN Read, we encountered a situation during upgrade the JournalNodes.

Cluster Info: 
Active: nn0
Standby: nn1

1. Rolling restart journal node. (related config: fs.journalnode.edit-cache-size.bytes=1G, -Xms1G, -Xmx=1G)

2. The cluster runs for a while, edits cache usage is increasing and memory is used up.

3. Active namenode(nn0) shutdown because of “Timed out waiting 120000ms for a quorum of nodes to respond”.

4. Transfer nn1 to Active state.

5. New Active namenode(nn1) also shutdown because of “Timed out waiting 120000ms for a quorum of nodes to respond” too.

6. The cluster crashed.

 

Related code:

JournaledEditsCache(Configuration conf) {
  capacity = conf.getInt(DFSConfigKeys.DFS_JOURNALNODE_EDIT_CACHE_SIZE_KEY,
      DFSConfigKeys.DFS_JOURNALNODE_EDIT_CACHE_SIZE_DEFAULT);
  if (capacity > 0.9 * Runtime.getRuntime().maxMemory()) {
    Journal.LOG.warn(String.format("Cache capacity is set at %d bytes but " +
        "maximum JVM memory is only %d bytes. It is recommended that you " +
        "decrease the cache size or increase the heap size.",
        capacity, Runtime.getRuntime().maxMemory()));
  }
  Journal.LOG.info("Enabling the journaled edits cache with a capacity " +
      "of bytes: " + capacity);
  ReadWriteLock lock = new ReentrantReadWriteLock(true);
  readLock = new AutoCloseableLock(lock.readLock());
  writeLock = new AutoCloseableLock(lock.writeLock());
  initialize(INVALID_TXN_ID);
} 

Currently, fs.journalNode.edit-cache-size-bytes can be set to a larger size than the memory requested by the process. If fs.journalNode.edit-cache-sie.bytes > 0.9 * Runtime.getruntime().maxMemory(), only warn logs are printed during journalnode startup. This can easily be overlooked by users. However, as the cluster runs to a certain period of time, it is likely to cause the cluster to crash.

 

NN log:

IMO, we should not set the cache size to a fixed value, but to the ratio of maximum memory, which is 0.2 by default.
This avoids the problem of too large cache size. In addition, users can actively adjust the heap size when they need to increase the cache size.

Issue Links

• links to: GitHub Pull Request #4209