Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1942

Number of tasks show in Tez UI with auto-reduce parallelism is misleading

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.5.2
    • Fix Version/s: 0.5.4
    • Component/s: None
    • Labels:
      None

      Description

      Ran a simple hive query (with tez) and "--hiveconf hive.tez.auto.reducer.parallelism=true" . This internally turns on tez's auto reduce parallelism.

      • Job started off with 1009 reduce tasks
      • Tez reduces the number of reducers to 253
      • Job completes successfully, but TEZ UI shows 1009 as the number of reducers (and 253 tasks as successful tasks). This can be a little misleading.

      I will attach the screenshots soon.

      1. output.json
        22 kB
        Rajesh Balamohan
      2. result_with_direct_vertex.png
        97 kB
        Prakash Ramachandran
      3. result_with_primary_filter.png
        137 kB
        Prakash Ramachandran
      4. Screen Shot 2015-01-14 at 9.18.21 AM.png
        101 kB
        Rajesh Balamohan
      5. Screen Shot 2015-01-14 at 9.18.54 AM.png
        87 kB
        Rajesh Balamohan
      6. TEZ-1942.1.branch-0.5.patch
        19 kB
        Prakash Ramachandran
      7. TEZ-1942.1.patch
        5 kB
        Prakash Ramachandran
      8. TEZ-1942.2.patch
        7 kB
        Prakash Ramachandran
      9. TEZ-1942.3.patch
        20 kB
        Prakash Ramachandran

        Activity

        Hide
        hitesh Hitesh Shah added a comment -

        Rajesh Balamohan Mind attaching the output for timelinehost:8188/ws/v1/timeline/TEZ_VERTEX_ID/<vertexId>/ for the vertex in question.

        Show
        hitesh Hitesh Shah added a comment - Rajesh Balamohan Mind attaching the output for timelinehost:8188/ws/v1/timeline/TEZ_VERTEX_ID/<vertexId>/ for the vertex in question.
        Show
        rajesh.balamohan Rajesh Balamohan added a comment - Attaching the output of http://machine:8188/ws/v1/timeline/TEZ_VERTEX_ID/vertex_1421164610335_0020_1_01/
        Hide
        pramachandran Prakash Ramachandran added a comment - - edited

        did some analysis. looks like the results returned is changing based on the query parameters.
        when queried for "get all vertexes for this dag" it returns 1009 (numTasks in the screenshot)
        see screenshot result_with_primary_filter.png

        http://machine:8188/ws/v1/timeline/TEZ_VERTEX_ID?limit=11&primaryFilter=TEZ_DAG_ID%3Adag_1421164610335_0020_1
        

        when queried for "get for a particular vertex" it returns 253
        see screenshot result_with_direct_vertex.png

        http://machine:8188/ws/v1/timeline/TEZ_VERTEX_ID/vertex_1421164610335_0020_1_01/
        
        Show
        pramachandran Prakash Ramachandran added a comment - - edited did some analysis. looks like the results returned is changing based on the query parameters. when queried for "get all vertexes for this dag" it returns 1009 (numTasks in the screenshot) see screenshot result_with_primary_filter.png http: //machine:8188/ws/v1/timeline/TEZ_VERTEX_ID?limit=11&primaryFilter=TEZ_DAG_ID%3Adag_1421164610335_0020_1 when queried for "get for a particular vertex" it returns 253 see screenshot result_with_direct_vertex.png http: //machine:8188/ws/v1/timeline/TEZ_VERTEX_ID/vertex_1421164610335_0020_1_01/
        Hide
        hitesh Hitesh Shah added a comment -

        Prakash Ramachandran Does a YARN jira need to be filed for this timeline issue?

        Show
        hitesh Hitesh Shah added a comment - Prakash Ramachandran Does a YARN jira need to be filed for this timeline issue?
        Hide
        pramachandran Prakash Ramachandran added a comment - - edited

        raised YARN-3062 for the same.

        Show
        pramachandran Prakash Ramachandran added a comment - - edited raised YARN-3062 for the same.
        Hide
        hitesh Hitesh Shah added a comment -

        Prakash Ramachandran Looks like we need to add primary filters to the entities on every call to timeline as per the conversation on YARN-3062. Seems like a very lame solution but probably the only way to get the UI to work correctly against timeline data.

        Would you like to take a crack at this?

        Show
        hitesh Hitesh Shah added a comment - Prakash Ramachandran Looks like we need to add primary filters to the entities on every call to timeline as per the conversation on YARN-3062 . Seems like a very lame solution but probably the only way to get the UI to work correctly against timeline data. Would you like to take a crack at this?
        Hide
        hitesh Hitesh Shah added a comment -

        An initial fix might be to fix VertexInit, VertexFinished and VertexParallelismUpdated events.

        Show
        hitesh Hitesh Shah added a comment - An initial fix might be to fix VertexInit, VertexFinished and VertexParallelismUpdated events.
        Hide
        pramachandran Prakash Ramachandran added a comment -

        Hitesh Shah review please

        Show
        pramachandran Prakash Ramachandran added a comment - Hitesh Shah review please
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12692431/TEZ-1942.1.patch
        against master revision 61bb0f8.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        -1 findbugs. The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in :
        org.apache.tez.dag.history.logging.ats.TestHistoryEventTimelineConversion

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/29//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/29//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692431/TEZ-1942.1.patch against master revision 61bb0f8. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 findbugs . The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in : org.apache.tez.dag.history.logging.ats.TestHistoryEventTimelineConversion Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/29//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/29//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/29//console This message is automatically generated.
        Hide
        pramachandran Prakash Ramachandran added a comment -

        fixed UT.

        Show
        pramachandran Prakash Ramachandran added a comment - fixed UT.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12692464/TEZ-1942.2.patch
        against master revision 61bb0f8.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        -1 findbugs. The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/30//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/30//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692464/TEZ-1942.2.patch against master revision 61bb0f8. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 findbugs . The patch appears to introduce 260 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/30//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/30//artifact/patchprocess/newPatchFindbugsWarningstez-tests.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/30//console This message is automatically generated.
        Hide
        hitesh Hitesh Shah added a comment -

        Source changes look good - however the unit test changes do not match the no. of changes done to the source. Could you add/modify the required unit tests to match the code changes.

        Show
        hitesh Hitesh Shah added a comment - Source changes look good - however the unit test changes do not match the no. of changes done to the source. Could you add/modify the required unit tests to match the code changes.
        Hide
        pramachandran Prakash Ramachandran added a comment -

        patch 3

        • added test cases
        Show
        pramachandran Prakash Ramachandran added a comment - patch 3 added test cases
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12692760/TEZ-1942.3.patch
        against master revision cdf253f.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        -1 findbugs. The patch appears to introduce 68 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/46//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/46//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692760/TEZ-1942.3.patch against master revision cdf253f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 findbugs . The patch appears to introduce 68 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/46//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/46//console This message is automatically generated.
        Hide
        hitesh Hitesh Shah added a comment -

        +1. Committing shortly.

        Show
        hitesh Hitesh Shah added a comment - +1. Committing shortly.
        Hide
        hitesh Hitesh Shah added a comment -

        Committed to master and branch 0.6.

        Prakash Ramachandran This will need a re-based patch for branch 0.5 - would you mind creating one?

        Show
        hitesh Hitesh Shah added a comment - Committed to master and branch 0.6. Prakash Ramachandran This will need a re-based patch for branch 0.5 - would you mind creating one?
        Hide
        pramachandran Prakash Ramachandran added a comment -

        rebased patch for branch-0.5 Hitesh Shah review please.

        Show
        pramachandran Prakash Ramachandran added a comment - rebased patch for branch-0.5 Hitesh Shah review please.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12693092/TEZ-1942.1.branch-0.5.patch
        against master revision c684653.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/56//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693092/TEZ-1942.1.branch-0.5.patch against master revision c684653. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/56//console This message is automatically generated.
        Hide
        hitesh Hitesh Shah added a comment -

        +1 - committing shortly.

        Show
        hitesh Hitesh Shah added a comment - +1 - committing shortly.
        Hide
        hitesh Hitesh Shah added a comment -

        Committed to branch 0.5 too. Thanks Prakash Ramachandran

        Show
        hitesh Hitesh Shah added a comment - Committed to branch 0.5 too. Thanks Prakash Ramachandran
        Hide
        hitesh Hitesh Shah added a comment -

        Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

        Show
        hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

          People

          • Assignee:
            pramachandran Prakash Ramachandran
            Reporter:
            rajesh.balamohan Rajesh Balamohan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development