Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5983

[Umbrella] Support for FPGA as a Resource in YARN

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • yarn
    • None

    Description

      As various big data workload running on YARN, CPU will no longer scale eventually and heterogeneous systems will become more important. ML/DL is a rising star in recent years, applications focused on these areas have to utilize GPU or FPGA to boost performance. Also, hardware vendors such as Intel also invest in such hardware. It is most likely that FPGA will become popular in data centers like CPU in the near future.

      So YARN as a resource managing and scheduling system, would be great to evolve to support this. This JIRA proposes FPGA to be a first-class citizen. The changes roughly includes:
      1. FPGA resource detection and heartbeat
      2. Scheduler changes (YARN-3926 invlolved)
      3. FPGA related preparation and isolation before launch container
      We know that YARN-3926 is trying to extend current resource model. But still we can leave some FPGA related discussion here

      Attachments

        1. YARN-5983_end-to-end_test_report.pdf
          1.85 MB
          Zhankun Tang
        2. YARN-5983-implementation-notes.pdf
          301 kB
          Zhankun Tang
        3. YARN-5983-Support-FPGA-resource-on-NM-side_v1.pdf
          381 kB
          Zhankun Tang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tangzhankun Zhankun Tang
            tangzhankun Zhankun Tang
            Votes:
            1 Vote for this issue
            Watchers:
            32 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment