Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1942

Number of tasks show in Tez UI with auto-reduce parallelism is misleading

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.5.2
    • Fix Version/s: 0.5.4
    • Component/s: None
    • Labels:
      None

      Description

      Ran a simple hive query (with tez) and "--hiveconf hive.tez.auto.reducer.parallelism=true" . This internally turns on tez's auto reduce parallelism.

      • Job started off with 1009 reduce tasks
      • Tez reduces the number of reducers to 253
      • Job completes successfully, but TEZ UI shows 1009 as the number of reducers (and 253 tasks as successful tasks). This can be a little misleading.

      I will attach the screenshots soon.

        Attachments

        1. output.json
          22 kB
          Rajesh Balamohan
        2. result_with_direct_vertex.png
          97 kB
          Prakash Ramachandran
        3. result_with_primary_filter.png
          137 kB
          Prakash Ramachandran
        4. Screen Shot 2015-01-14 at 9.18.21 AM.png
          101 kB
          Rajesh Balamohan
        5. Screen Shot 2015-01-14 at 9.18.54 AM.png
          87 kB
          Rajesh Balamohan
        6. TEZ-1942.1.branch-0.5.patch
          19 kB
          Prakash Ramachandran
        7. TEZ-1942.1.patch
          5 kB
          Prakash Ramachandran
        8. TEZ-1942.2.patch
          7 kB
          Prakash Ramachandran
        9. TEZ-1942.3.patch
          20 kB
          Prakash Ramachandran

          Activity

            People

            • Assignee:
              pramachandran Prakash Ramachandran
              Reporter:
              rajesh.balamohan Rajesh Balamohan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: