Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7079

JobHistory#ServiceStop implementation is incorrect

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.1.4, 3.2.2, 2.10.1
    • Component/s: None
    • Labels:
      None

      Description

      JobHistory.serviceStop skips waiting for the thread pool to terminate. The problem is due to incorrect while condition that will evaluate to false on the iteration of the loop.

           scheduledExecutor.shutdown();
            boolean interrupted = false;
            long currentTime = System.currentTimeMillis();
            while (!scheduledExecutor.isShutdown()
                && System.currentTimeMillis() > currentTime + 1000l && !interrupted) {
              try {
                Thread.sleep(20);
              } catch (InterruptedException e) {
                interrupted = true;
              }
            }
      

      The expression "System.currentTimeMillis() > currentTime + 1000L" is false because currentTime was just initialized with System.currentTimeMillis(). As a result the the thread won't wait until the executor is terminated. Instead, it will force a shutdown immediately.

      TestMRIntermediateDataEncryption is failing in precommit builds

      TestMRIntermediateDataEncryption is either timing out or tearing down the JVM which causes the unit tests in jobclient to not pass cleanly during precommit builds. From sample precommit console output, note the lack of a test results line when the test is run:

      [INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
      [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976 s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat
      [INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
      [INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
      [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659 s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
      [...]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 02:14 h
      [INFO] Finished at: 2018-04-12T04:27:06+00:00
      [INFO] Final Memory: 24M/594M
      [INFO] ------------------------------------------------------------------------
      [WARNING] The requested profile "parallel-tests" could not be activated because it does not exist.
      [WARNING] The requested profile "native" could not be activated because it does not exist.
      [WARNING] The requested profile "yarn-ui" could not be activated because it does not exist.
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork -> [Help 1]
      

        Attachments

        1. 2020-01-10-MRApp-stack-dump.txt
          46 kB
          Ahmed Hussein
        2. 2020-01-10-org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-version-14.txt
          1.48 MB
          Ahmed Hussein
        3. MAPREDUCE-7079.001.patch
          12 kB
          Ahmed Hussein
        4. MAPREDUCE-7079.002.patch
          13 kB
          Ahmed Hussein
        5. MAPREDUCE-7079.003.patch
          16 kB
          Ahmed Hussein
        6. MAPREDUCE-7079.004.patch
          16 kB
          Ahmed Hussein
        7. MAPREDUCE-7079.005.patch
          16 kB
          Ahmed Hussein
        8. MAPREDUCE-7079.006.patch
          16 kB
          Ahmed Hussein
        9. MAPREDUCE-7079.007.patch
          16 kB
          Ahmed Hussein
        10. MAPREDUCE-7079.008.patch
          16 kB
          Ahmed Hussein
        11. MAPREDUCE-7079.009.patch
          16 kB
          Ahmed Hussein
        12. MAPREDUCE-7079.010.patch
          16 kB
          Ahmed Hussein

          Issue Links

            Activity

              People

              • Assignee:
                ahussein Ahmed Hussein
                Reporter:
                jlowe Jason Darrell Lowe
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: