Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5131 Fine-grained Resource Configuration
  3. FLINK-5135

ResourceProfile for slot request should be expanded to correspond with ResourceSpec

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: JobManager, ResourceManager
    • Labels:
      None

      Description

      The JobManager requests slot by ResourceProfile from ResourceManager before submitting tasks. Currently the ResourceProfile only contains cpu cores and memory properties. The memory should be expanded to different types such as heap memory, direct memory and native memory which corresponds with memory in ResourceSpec.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user wangzhijiang999 opened a pull request:

          https://github.com/apache/flink/pull/2907

          FLINK-5135[JM]ResourceProfile for slot request should be expanded to correspond with ResourceSpec

          Currently the *ResourceProfile* only contains cpu cores and memory properties. The memory should be expanded to different types such as heap memory, direct memory and native memory which corresponds with memory in *ResourceSpec*.

          It contains the related un-merging codes for FLINK-5132 FLINK-5133 FLINK-5134. And the modifications for FLINK-5135 only includes *SlotPool* and *ResourceProfile*.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/wangzhijiang999/flink jira-5135

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/2907.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2907


          commit 73e7cca3b3540b877bf1c03d18fc77d6b69dc867
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2016-11-28T10:07:15Z

          FLINK-5132FLINK-5133FLINK-5134[Core,DataStream API,DataSet API] Fine-grained Resource Configuration

          Summary: above

          Test Plan: NA

          Reviewers: yushi.wxg

          Subscribers: #blink

          Differential Revision: http://phabricator.taobao.net/D6593

          commit 02b0e9d71db86921485ee27dbbb588f6c48fd19e
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2016-11-30T09:13:32Z

          FLINK-5132[DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream

          commit ceed4cb5a9798fb628e82511c08b70636acae9b0
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2016-11-30T09:43:22Z

          FLINK-5132[DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream

          commit 6021b056f4de8bb2e74ff6709326b58e69547404
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2016-11-30T09:55:10Z

          FLINK-5132[DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream

          commit 3ccd36033ff9b55c5f7ec7b4afbbefd261c85d61
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2016-11-30T10:01:16Z

          FLINK-5135[JM]ResourceProfile for slot request should be expanded to correspond with ResourceSpec


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user wangzhijiang999 opened a pull request: https://github.com/apache/flink/pull/2907 FLINK-5135 [JM] ResourceProfile for slot request should be expanded to correspond with ResourceSpec Currently the * ResourceProfile * only contains cpu cores and memory properties. The memory should be expanded to different types such as heap memory, direct memory and native memory which corresponds with memory in * ResourceSpec *. It contains the related un-merging codes for FLINK-5132 FLINK-5133 FLINK-5134 . And the modifications for FLINK-5135 only includes * SlotPool * and * ResourceProfile *. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangzhijiang999/flink jira-5135 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2907.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2907 commit 73e7cca3b3540b877bf1c03d18fc77d6b69dc867 Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2016-11-28T10:07:15Z FLINK-5132 FLINK-5133 FLINK-5134 [Core,DataStream API,DataSet API] Fine-grained Resource Configuration Summary: above Test Plan: NA Reviewers: yushi.wxg Subscribers: #blink Differential Revision: http://phabricator.taobao.net/D6593 commit 02b0e9d71db86921485ee27dbbb588f6c48fd19e Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2016-11-30T09:13:32Z FLINK-5132 [DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream commit ceed4cb5a9798fb628e82511c08b70636acae9b0 Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2016-11-30T09:43:22Z FLINK-5132 [DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream commit 6021b056f4de8bb2e74ff6709326b58e69547404 Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2016-11-30T09:55:10Z FLINK-5132 [DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream commit 3ccd36033ff9b55c5f7ec7b4afbbefd261c85d61 Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2016-11-30T10:01:16Z FLINK-5135 [JM] ResourceProfile for slot request should be expanded to correspond with ResourceSpec
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user wangzhijiang999 closed the pull request at:

          https://github.com/apache/flink/pull/2907

          Show
          githubbot ASF GitHub Bot added a comment - Github user wangzhijiang999 closed the pull request at: https://github.com/apache/flink/pull/2907
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zhijiangW opened a pull request:

          https://github.com/apache/flink/pull/3457

          FLINK-5135[runtime] Expand the memory types in ResourceProfile related with ResourceSpec

          This is a part of fine-grained resource configuration in `flip6`

          The `JobManager` requests slot with `ResourceProfile` from `ResourceManager` before deployment. Currently the `ResourceProfile` only contains cpu cores and memory fields. The memory should be expanded to different types like heap memory, direct memory and native memory corresponding with `ResourceSpec`.

          Further, the direct memory in `ResourceProfile` would be considered in `ContaineredTaskManagerParameters` used to set JVM options before launching the `TaskManager` process.

          And some other related processes in runtime would be submitted later:
          1. `SlotPool` requests slot to get preferred resource from `JobVertex`, which is blocked by [PR-3455](https://github.com/apache/flink/pull/3455)
          2. Sets the `ResourceProfile` to `TaskManager` by `ResourceManager`

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zhijiangW/flink FLINK-5135

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3457.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3457


          commit c1dcf48f2d5b3d444fba0f1cd23c7c0070621796
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2017-03-02T10:23:42Z

          FLINK-5135[runtime]Expand the fields in ResourceProfile based on ResourceSpec


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zhijiangW opened a pull request: https://github.com/apache/flink/pull/3457 FLINK-5135 [runtime] Expand the memory types in ResourceProfile related with ResourceSpec This is a part of fine-grained resource configuration in `flip6` The `JobManager` requests slot with `ResourceProfile` from `ResourceManager` before deployment. Currently the `ResourceProfile` only contains cpu cores and memory fields. The memory should be expanded to different types like heap memory, direct memory and native memory corresponding with `ResourceSpec`. Further, the direct memory in `ResourceProfile` would be considered in `ContaineredTaskManagerParameters` used to set JVM options before launching the `TaskManager` process. And some other related processes in runtime would be submitted later: 1. `SlotPool` requests slot to get preferred resource from `JobVertex`, which is blocked by [PR-3455] ( https://github.com/apache/flink/pull/3455 ) 2. Sets the `ResourceProfile` to `TaskManager` by `ResourceManager` You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhijiangW/flink FLINK-5135 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3457.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3457 commit c1dcf48f2d5b3d444fba0f1cd23c7c0070621796 Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2017-03-02T10:23:42Z FLINK-5135 [runtime] Expand the fields in ResourceProfile based on ResourceSpec
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3457

          Looks good to me. merging this...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3457 Looks good to me. merging this...
          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed via 2592a19ccd4f83e4f57a835b03c7846d6edde927

          Show
          StephanEwen Stephan Ewen added a comment - Fixed via 2592a19ccd4f83e4f57a835b03c7846d6edde927
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3457

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3457

            People

            • Assignee:
              zjwang zhijiang
              Reporter:
              zjwang zhijiang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development