Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7079

JobHistory#ServiceStop implementation is incorrect

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0, 3.1.4, 3.2.2, 2.10.1
    • None
    • None

    Description

      JobHistory.serviceStop skips waiting for the thread pool to terminate. The problem is due to incorrect while condition that will evaluate to false on the iteration of the loop.

           scheduledExecutor.shutdown();
            boolean interrupted = false;
            long currentTime = System.currentTimeMillis();
            while (!scheduledExecutor.isShutdown()
                && System.currentTimeMillis() > currentTime + 1000l && !interrupted) {
              try {
                Thread.sleep(20);
              } catch (InterruptedException e) {
                interrupted = true;
              }
            }
      

      The expression "System.currentTimeMillis() > currentTime + 1000L" is false because currentTime was just initialized with System.currentTimeMillis(). As a result the the thread won't wait until the executor is terminated. Instead, it will force a shutdown immediately.

      TestMRIntermediateDataEncryption is failing in precommit builds

      TestMRIntermediateDataEncryption is either timing out or tearing down the JVM which causes the unit tests in jobclient to not pass cleanly during precommit builds. From sample precommit console output, note the lack of a test results line when the test is run:

      [INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
      [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976 s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat
      [INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
      [INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
      [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659 s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
      [...]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 02:14 h
      [INFO] Finished at: 2018-04-12T04:27:06+00:00
      [INFO] Final Memory: 24M/594M
      [INFO] ------------------------------------------------------------------------
      [WARNING] The requested profile "parallel-tests" could not be activated because it does not exist.
      [WARNING] The requested profile "native" could not be activated because it does not exist.
      [WARNING] The requested profile "yarn-ui" could not be activated because it does not exist.
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork -> [Help 1]
      

      Attachments

        1. 2020-01-10-MRApp-stack-dump.txt
          46 kB
          Ahmed Hussein
        2. 2020-01-10-org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-version-14.txt
          1.48 MB
          Ahmed Hussein
        3. MAPREDUCE-7079.001.patch
          12 kB
          Ahmed Hussein
        4. MAPREDUCE-7079.002.patch
          13 kB
          Ahmed Hussein
        5. MAPREDUCE-7079.003.patch
          16 kB
          Ahmed Hussein
        6. MAPREDUCE-7079.004.patch
          16 kB
          Ahmed Hussein
        7. MAPREDUCE-7079.005.patch
          16 kB
          Ahmed Hussein
        8. MAPREDUCE-7079.006.patch
          16 kB
          Ahmed Hussein
        9. MAPREDUCE-7079.007.patch
          16 kB
          Ahmed Hussein
        10. MAPREDUCE-7079.008.patch
          16 kB
          Ahmed Hussein
        11. MAPREDUCE-7079.009.patch
          16 kB
          Ahmed Hussein
        12. MAPREDUCE-7079.010.patch
          16 kB
          Ahmed Hussein

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ahussein Ahmed Hussein
            jlowe Jason Darrell Lowe
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment