Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9737

Performance degradation, Distributed Opportunistic Scheduling

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.2
    • None

    Description

      Opportunistic scheduling is supposed to provide lower scheduling time, and thus higher task throughput and lower job completion times for short jobs/tasks.

      Through my experiments I have found distributed scheduling can degrade performance.

      I ran a gridmix trace of 100 short jobs, each with 50 tasks. Average task run time was 1523ms.

      Findings:

      Attachments

        1. yarn-site.xml
          3 kB
          Babble Shack
        2. yarn-client.yarn-site.xml
          5 kB
          Babble Shack
        3. task_throughput_bp_100j_30t.svg
          42 kB
          Babble Shack
        4. task_throughput_bp_100j_30t.png
          9 kB
          Babble Shack
        5. task_throughput_boxplot_100j_50t_1500.svg
          43 kB
          Babble Shack
        6. jet_boxplot_j100_50t_1500.svg
          43 kB
          Babble Shack
        7. jet_boxplot_j100_50t_1500_with_outliers.svg
          47 kB
          Babble Shack
        8. jct_cdf_100j_50t_1500_with_outliers.svg
          52 kB
          Babble Shack
        9. jct_cdf_100j_100t_1500.svg
          38 kB
          Babble Shack
        10. jct_100j_30t.svg
          59 kB
          Babble Shack
        11. jct_100j_30t.png
          27 kB
          Babble Shack

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            Babbleshack Babble Shack

            Dates

              Created:
              Updated:

              Slack

                Issue deployment