[MAPREDUCE-7079] JobHistory#ServiceStop implementation is incorrect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2, 2.10.1
• Assignee: Ahmed Hussein
• Reporter: Jason Darrell Lowe
• Created: 12/Apr/18 16:02
• Updated: 29/Jan/20 17:43
• Resolved: 29/Jan/20 17:43

Description

JobHistory.serviceStop skips waiting for the thread pool to terminate. The problem is due to incorrect while condition that will evaluate to false on the iteration of the loop.

     scheduledExecutor.shutdown();
      boolean interrupted = false;
      long currentTime = System.currentTimeMillis();
      while (!scheduledExecutor.isShutdown()
          && System.currentTimeMillis() > currentTime + 1000l && !interrupted) {
        try {
          Thread.sleep(20);
        } catch (InterruptedException e) {
          interrupted = true;
        }
      }

The expression "System.currentTimeMillis() > currentTime + 1000L" is false because currentTime was just initialized with System.currentTimeMillis(). As a result the the thread won't wait until the executor is terminated. Instead, it will force a shutdown immediately.

TestMRIntermediateDataEncryption is failing in precommit builds

TestMRIntermediateDataEncryption is either timing out or tearing down the JVM which causes the unit tests in jobclient to not pass cleanly during precommit builds. From sample precommit console output, note the lack of a test results line when the test is run:

[INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976 s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat
[INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
[INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659 s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
[...]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:14 h
[INFO] Finished at: 2018-04-12T04:27:06+00:00
[INFO] Final Memory: 24M/594M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "parallel-tests" could not be activated because it does not exist.
[WARNING] The requested profile "native" could not be activated because it does not exist.
[WARNING] The requested profile "yarn-ui" could not be activated because it does not exist.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork -> [Help 1]

Issue Links

• causes: MAPREDUCE-7099 Daily test result fails in MapReduce JobClient though there isn't any error
• requires: HADOOP-16810 Increase entropy to improve cryptographic randomness on precommit Linux VMs