[HADOOP-17731] IPC client is stuck, waiting for connections map to be empty

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.10.1
• Component/s: ipc
• Assignee: Unassigned
• Reporter: Kiran Kumar Maturi
• Created: 24/May/21 11:27
• Updated: 24/May/21 16:07

Description

In our cluster we have observed when close command is issued to Datanode, It is stuck indefinitely waiting for connections map to be empty for some reason the map is not empty even after the connections have closed. code link.

Instead of infinite wait a timed wait (based on retries or timeout) would enable to shutdown the datanode in timely manner.

for (Connection conn : connections.values()) {
      conn.interrupt();
      conn.interruptConnectingThread();
    }
    
    // wait until all connections are closed
    while (!connections.isEmpty()) {
      try {
        Thread.sleep(STOP_SLEEP_TIME_MS);
      } catch (InterruptedException e) {
      }
    }