Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.4
    • Component/s: None
    • Labels:
      None

      Description

      I've been trying to port the hive ut to tez local mode. However, local mode seems to leak threads which causes tests to crash after a while (oom). See attached stack trace - there are a lot of "TezChild" threads still hanging around.

      (Siddharth Seth as discussed offline)

      1. stack5.txt
        867 kB
        Gunther Hagleitner
      2. TEZ-1962.1.txt
        9 kB
        Siddharth Seth
      3. TEZ-1962.1.branch_0.6.txt
        9 kB
        Siddharth Seth

        Activity

        Hide
        hitesh Hitesh Shah added a comment -

        Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

        Show
        hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.
        Hide
        sseth Siddharth Seth added a comment -

        Committed to branch-0.6 and branch-0.5.

        Show
        sseth Siddharth Seth added a comment - Committed to branch-0.6 and branch-0.5.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12692641/TEZ-1962.1.branch_0.6.txt
        against master revision 2544b05.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/40//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692641/TEZ-1962.1.branch_0.6.txt against master revision 2544b05. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/40//console This message is automatically generated.
        Hide
        sseth Siddharth Seth added a comment -

        Rebased patch for branch-0.6 and branch-0.5.

        Committed to master. Running tests locally for this branch, and will commit after that.

        Show
        sseth Siddharth Seth added a comment - Rebased patch for branch-0.6 and branch-0.5. Committed to master. Running tests locally for this branch, and will commit after that.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12692579/TEZ-1962.1.txt
        against master revision b723a05.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        -1 findbugs. The patch appears to introduce 254 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/37//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/37//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/37//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/37//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/37//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/37//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692579/TEZ-1962.1.txt against master revision b723a05. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 findbugs . The patch appears to introduce 254 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/37//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/37//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/37//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/37//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/37//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/37//console This message is automatically generated.
        Hide
        hitesh Hitesh Shah added a comment - - edited

        I think you need to run "mvn clean -Phadoop24 -P!hadoop26" - not sure why target is not getting excluded though.

        Show
        hitesh Hitesh Shah added a comment - - edited I think you need to run "mvn clean -Phadoop24 -P!hadoop26" - not sure why target is not getting excluded though.
        Hide
        sseth Siddharth Seth added a comment -

        There appear to be 9 release audit warnings after applying the patch.

        {color:red}-1 overall{color}.
        
            {color:green}+1 @author{color}.  The patch does not contain any @author tags.
        
            {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified tests.
                                Please justify why no new tests are needed for this patch.
                                Also please list what manual steps were performed to verify this patch.
        
            {color:green}+1 javac{color}.  The applied patch does not increase the total number of javac compiler warnings.
        
            {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.
        
            {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version ) warnings.
        
                {color:red}-1 release audit{color}.  The applied patch generated 9 release audit warnings.
        

        I don't think the 9 rat warnings have anything to do with the patch. From a local rat test

        Unapproved licenses:
        
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/inputFiles.lst
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/org.apache.tez.dag.history.logging.ats.TestATSHistoryLoggingService-output.txt
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/org.apache.tez.dag.history.logging.ats.TestATSHistoryLoggingService.txt
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/org.apache.tez.dag.history.logging.ats.TestHistoryEventTimelineConversion.txt
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/TEST-org.apache.tez.dag.history.logging.ats.TestATSHistoryLoggingService.xml
          /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/TEST-org.apache.tez.dag.history.logging.ats.TestHistoryEventTimelineConversion.xml
        

        Thanks for the review. Will commit once git is back up.

        Show
        sseth Siddharth Seth added a comment - There appear to be 9 release audit warnings after applying the patch. {color:red}-1 overall{color}. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version ) warnings. {color:red}-1 release audit{color}. The applied patch generated 9 release audit warnings. I don't think the 9 rat warnings have anything to do with the patch. From a local rat test Unapproved licenses: /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/maven-status/maven-compiler-plugin/compile/ default -compile/createdFiles.lst /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/maven-status/maven-compiler-plugin/compile/ default -compile/inputFiles.lst /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/maven-status/maven-compiler-plugin/testCompile/ default -testCompile/createdFiles.lst /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/maven-status/maven-compiler-plugin/testCompile/ default -testCompile/inputFiles.lst /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/org.apache.tez.dag.history.logging.ats.TestATSHistoryLoggingService-output.txt /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/org.apache.tez.dag.history.logging.ats.TestATSHistoryLoggingService.txt /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/org.apache.tez.dag.history.logging.ats.TestHistoryEventTimelineConversion.txt /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/TEST-org.apache.tez.dag.history.logging.ats.TestATSHistoryLoggingService.xml /Users/sseth/work2/projects/tez/commit/incubator-tez/tez-plugins/tez-yarn-timeline-history/target/surefire-reports/TEST-org.apache.tez.dag.history.logging.ats.TestHistoryEventTimelineConversion.xml Thanks for the review. Will commit once git is back up.
        Hide
        hitesh Hitesh Shah added a comment -

        Please run test patch locally and update jira with results as git is down ( causes the precommit build to fail )

        Show
        hitesh Hitesh Shah added a comment - Please run test patch locally and update jira with results as git is down ( causes the precommit build to fail )
        Hide
        hitesh Hitesh Shah added a comment -

        +1

        Show
        hitesh Hitesh Shah added a comment - +1
        Hide
        sseth Siddharth Seth added a comment -

        Patch to fix this.

        The main reason here is a NPE in a log line in case of an Interrupt. The exception causes TezChild.run to fall off without shutting down the executor and TaskReporter threads.

        The patch fixes the NPE, adds some checks to ensure shutdown is called, and changes LocalContainerLauncher to invoke a TezChild shutdown in case of an error from TezChild.

        I'm going to open a couple of follow up jiras to change the way tasks are cancelled.

        Tested locally, and there's no hung threads after this.

        Hitesh Shah - please review.

        Show
        sseth Siddharth Seth added a comment - Patch to fix this. The main reason here is a NPE in a log line in case of an Interrupt. The exception causes TezChild.run to fall off without shutting down the executor and TaskReporter threads. The patch fixes the NPE, adds some checks to ensure shutdown is called, and changes LocalContainerLauncher to invoke a TezChild shutdown in case of an error from TezChild. I'm going to open a couple of follow up jiras to change the way tasks are cancelled. Tested locally, and there's no hung threads after this. Hitesh Shah - please review.

          People

          • Assignee:
            sseth Siddharth Seth
            Reporter:
            hagleitn Gunther Hagleitner
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development