Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1934

TestAMRecovery may fail due to the execution order is not determined

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      task_1 is not guaranteed to been scheduled before task_0, so task_1 may finished before task_0. While in the current TestAMRecovery, the finish of task_1 is treated as the finished signal of vertex ( only 2 tasks in this vertex)

        Activity

        Hide
        zjffdu Jeff Zhang added a comment -

        Hitesh Shah, Please help review it.

        Show
        zjffdu Jeff Zhang added a comment - Hitesh Shah , Please help review it.
        Hide
        hitesh Hitesh Shah added a comment -

        Mostly looks good. Does the "onSourceTaskCompleted" function need to be synchronized - can it be called concurrently for diff tasks finishing?

        Show
        hitesh Hitesh Shah added a comment - Mostly looks good. Does the "onSourceTaskCompleted" function need to be synchronized - can it be called concurrently for diff tasks finishing?
        Hide
        hitesh Hitesh Shah added a comment -

        Triggered test patch for this jira.

        Show
        hitesh Hitesh Shah added a comment - Triggered test patch for this jira.
        Hide
        zjffdu Jeff Zhang added a comment -

        onSourceTaskCompleted is only called in the main dispatcher thread. it should be fine without synchronized

        Show
        zjffdu Jeff Zhang added a comment - onSourceTaskCompleted is only called in the main dispatcher thread. it should be fine without synchronized
        Hide
        hitesh Hitesh Shah added a comment - - edited

        +1 (pending test-patch results).

        Show
        hitesh Hitesh Shah added a comment - - edited +1 (pending test-patch results).
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12691636/TEZ-1934-1.patch
        against master revision 61bb0f8.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        -1 findbugs. The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/26//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/26//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12691636/TEZ-1934-1.patch against master revision 61bb0f8. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 findbugs . The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/26//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/26//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/26//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12691636/TEZ-1934-1.patch
        against master revision 61bb0f8.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javac. The applied patch generated 179 javac compiler warnings (more than the master's current 171 warnings).

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.
        See https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/diffJavadocWarnings.txt for details.

        -1 findbugs. The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/27//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
        Javac warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/diffJavacWarnings.txt
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/27//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12691636/TEZ-1934-1.patch against master revision 61bb0f8. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 179 javac compiler warnings (more than the master's current 171 warnings). -1 javadoc . The javadoc tool appears to have generated 2 warning messages. See https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/diffJavadocWarnings.txt for details. -1 findbugs . The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/27//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html Javac warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/27//console This message is automatically generated.
        Hide
        zjffdu Jeff Zhang added a comment -

        Hitesh Shah, New javac compiler warnings and javadoc warnings are generated, where can I see these warnings ? The link https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/diffJavadocWarnings.txt looks broken.

        Show
        zjffdu Jeff Zhang added a comment - Hitesh Shah , New javac compiler warnings and javadoc warnings are generated, where can I see these warnings ? The link https://builds.apache.org/job/PreCommit-TEZ-Build/27//artifact/patchprocess/diffJavadocWarnings.txt looks broken.
        Hide
        hitesh Hitesh Shah added a comment -

        Not sure why one build worked fine and the latter didnt. Triggered test patch again.

        Show
        hitesh Hitesh Shah added a comment - Not sure why one build worked fine and the latter didnt. Triggered test patch again.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12691636/TEZ-1934-1.patch
        against master revision 61bb0f8.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        -1 findbugs. The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/32//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/32//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12691636/TEZ-1934-1.patch against master revision 61bb0f8. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 findbugs . The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/32//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/32//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/32//console This message is automatically generated.
        Hide
        hitesh Hitesh Shah added a comment -

        +1. Looks good to commit ( after git comes back online )

        Show
        hitesh Hitesh Shah added a comment - +1. Looks good to commit ( after git comes back online )
        Hide
        zjffdu Jeff Zhang added a comment -

        committed to master

        Show
        zjffdu Jeff Zhang added a comment - committed to master
        Hide
        zjffdu Jeff Zhang added a comment -

        backport it to branch-0.5

        commit 119003effd4acfcba6f26397fddc8f4d8b0f4671 (HEAD, origin/branch-0.5, branch-0.5)
        Author: Jeff Zhang <zjffdu@apache.org>
        Date: Fri Jan 16 09:47:46 2015 +0800

        TEZ-1934. TestAMRecovery may fail due to the execution order is not determined. (zjffdu)

        (cherry picked from commit 880d4f38f4ccb2de987a7c9ca4cbbbe67b0a833e)

        Conflicts:
        CHANGES.txt

        Show
        zjffdu Jeff Zhang added a comment - backport it to branch-0.5 commit 119003effd4acfcba6f26397fddc8f4d8b0f4671 (HEAD, origin/branch-0.5, branch-0.5) Author: Jeff Zhang <zjffdu@apache.org> Date: Fri Jan 16 09:47:46 2015 +0800 TEZ-1934 . TestAMRecovery may fail due to the execution order is not determined. (zjffdu) (cherry picked from commit 880d4f38f4ccb2de987a7c9ca4cbbbe67b0a833e) Conflicts: CHANGES.txt
        Hide
        hitesh Hitesh Shah added a comment -

        Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

        Show
        hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

          People

          • Assignee:
            zjffdu Jeff Zhang
            Reporter:
            zjffdu Jeff Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development