[HDFS-8163] Using monotonicNow for block report scheduling causes test failures on recently restarted systems

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.1
• Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Arpit Agarwal
• Reporter: Arpit Agarwal
• Created: 16/Apr/15 18:56
• Updated: 05/Jan/17 22:41
• Resolved: 21/Apr/15 18:21

Description

BPServiceActor#blockReport has the following check:

  List<DatanodeCommand> blockReport() throws IOException {
    // send block report if timer has expired.
    final long startTime = monotonicNow();
    if (startTime - lastBlockReport <= dnConf.blockReportInterval) {
      return null;
    }

Many tests trigger an immediate block report via BPServiceActor#triggerBlockReportForTests which sets lastBlockReport = 0. However if the machine was restarted recently then startTime may be less than dnConf.blockReportInterval and the block report is not sent.

Time#monotonicNow uses System#nanoTime which represents time elapsed since an arbitrary origin. The time should be used only for comparison with other values returned by System#nanoTime.

Issue Links

• is broken by: HDFS-6841 Use Time.monotonicNow() wherever applicable instead of Time.now()
• relates to: HDFS-8179 DFSClient#getServerDefaults returns null within 1 hour of system start