Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1942

Number of tasks show in Tez UI with auto-reduce parallelism is misleading

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.5.2
    • 0.5.4
    • None
    • None

    Description

      Ran a simple hive query (with tez) and "--hiveconf hive.tez.auto.reducer.parallelism=true" . This internally turns on tez's auto reduce parallelism.

      • Job started off with 1009 reduce tasks
      • Tez reduces the number of reducers to 253
      • Job completes successfully, but TEZ UI shows 1009 as the number of reducers (and 253 tasks as successful tasks). This can be a little misleading.

      I will attach the screenshots soon.

      Attachments

        1. Screen Shot 2015-01-14 at 9.18.54 AM.png
          87 kB
          Rajesh Balamohan
        2. Screen Shot 2015-01-14 at 9.18.21 AM.png
          101 kB
          Rajesh Balamohan
        3. output.json
          22 kB
          Rajesh Balamohan
        4. result_with_direct_vertex.png
          97 kB
          Prakash Ramachandran
        5. result_with_primary_filter.png
          137 kB
          Prakash Ramachandran
        6. TEZ-1942.1.patch
          5 kB
          Prakash Ramachandran
        7. TEZ-1942.2.patch
          7 kB
          Prakash Ramachandran
        8. TEZ-1942.3.patch
          20 kB
          Prakash Ramachandran
        9. TEZ-1942.1.branch-0.5.patch
          19 kB
          Prakash Ramachandran

        Activity

          People

            pramachandran Prakash Ramachandran
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: