Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1665

DAGScheduler should provide a priority range instead of an exact priority

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.2
    • None
    • None
    • Reviewed

    Description

      Specifying the exact priority leaks task attempt concepts into the DAGScheduler. Eg. the DAGScheduler increases the priority of attempt-re-executions arbitrarily. Providing a range allows these decisions to be made at the task attempt level. e.g. when there are no competing tasks then there is no reason to increase the priority. Similarly for speculation, the speculative task priority could be lower than normal tasks because we may want the first attempt of other tasks to run ahead of a speculative task. These relative priorities within a vertex can be made locally instead of leaking into the DAGScheduler.

      Attachments

        Issue Links

        is depended upon by

        Sub-task - The sub-task of the issue TEZ-14 Support MR like speculation capabilities based on latency deviation from the mean

        • Major - Major loss of function.
        • Closed

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bikassaha Bikas Saha
            bikassaha Bikas Saha
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment