[HDFS-10688] BPServiceActor may run into a tight loop for sending block report when hitting IOException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: datanode
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Chen Liang
• Reporter: Jing Zhao
• Created: 25/Jul/16 17:51
• Updated: 30/Aug/16 01:14
• Resolved: 26/Jul/16 01:45

Description

Currently in BPServiceActor#offerService, when datanode runs into a local IOException, the DataNode only logs the exception and runs into the while loop again:

      } catch(RemoteException re) {
        .......
        LOG.warn("RemoteException in offerService", re);
        try {
          long sleepTime = Math.min(1000, dnConf.heartBeatInterval);
          Thread.sleep(sleepTime);
        } catch (InterruptedException ie) {
          Thread.currentThread().interrupt();
        }
      } catch (IOException e) {
        LOG.warn("IOException in offerService", e);
      }

This tight loop may cause some issue. For example, in a production cluster, we saw a DataNode hit exception when doing kerberos realm lookup. This tight loop finally caused the DataNode to send hundreds of DNS lookup queries per second.