Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4921

TaskSetManager mistakenly returns PROCESS_LOCAL for NO_PREF tasks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 1.2.0
    • None
    • Spark Core
    • None

    Description

      During research for HIVE-9153, we found that TaskSetManager returns PROCESS_LOCAL for NO_PREF tasks, which may caused performance degradation. Changing the return value to NO_PREF, as demonstrated in the attached patch, seemingly improves the performance.

      Attachments

        1. NO_PREF.patch
          0.7 kB
          Xuefu Zhang

        Issue Links

          is depended upon by

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-3145 Hive on Spark umbrella

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HIVE-9153 Perf enhancement on CombineHiveInputFormat and HiveInputFormat

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #3816 [Github] Pull Request #3816 (sryza)

          Activity

            People

              Unassigned Unassigned
              xuefuz Xuefu Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: