Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5131 Fine-grained Resource Configuration
  3. FLINK-5132

Introduce the ResourceSpec for grouping different resource factors in API

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Core
    • Labels:
      None

      Description

      This is part of the fine-grained resource configuration.

      The current resource factors include cpu cores, heap memory, direct memory, native memory and state size.
      The ResourceSpec will provide some basic constructions for grouping different resource factors as needed and the construction can also be expanded easily for further requirements.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user wangzhijiang999 opened a pull request:

          https://github.com/apache/flink/pull/2906

          FLINK-5132FLINK-5133FLINK-5134[Core,DataStream API,DataSet API] Fine-grained Resource Configuration

          Introduce the *ResourceSpec* structure to describe the different resource factors.
          Add new *setResource* API for both *DataStream* and *DataSet*.
          In the internal construction of *StreamGraph* and *JobGraph* for DataStream, set the corresponding resource and merge resource for the *JobVertex, and the similar way in **Plan, **OptimizedPlan* for *DataSet*.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/wangzhijiang999/flink jira-5132

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/2906.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2906


          commit 73e7cca3b3540b877bf1c03d18fc77d6b69dc867
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2016-11-28T10:07:15Z

          FLINK-5132FLINK-5133FLINK-5134[Core,DataStream API,DataSet API] Fine-grained Resource Configuration

          Summary: above

          Test Plan: NA

          Reviewers: yushi.wxg

          Subscribers: #blink

          Differential Revision: http://phabricator.taobao.net/D6593

          commit 02b0e9d71db86921485ee27dbbb588f6c48fd19e
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2016-11-30T09:13:32Z

          FLINK-5132[DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream

          commit ceed4cb5a9798fb628e82511c08b70636acae9b0
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2016-11-30T09:43:22Z

          FLINK-5132[DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user wangzhijiang999 opened a pull request: https://github.com/apache/flink/pull/2906 FLINK-5132 FLINK-5133 FLINK-5134 [Core,DataStream API,DataSet API] Fine-grained Resource Configuration Introduce the * ResourceSpec * structure to describe the different resource factors. Add new * setResource * API for both * DataStream * and * DataSet *. In the internal construction of * StreamGraph * and * JobGraph * for DataStream, set the corresponding resource and merge resource for the * JobVertex , and the similar way in **Plan , **OptimizedPlan * for * DataSet *. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangzhijiang999/flink jira-5132 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2906.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2906 commit 73e7cca3b3540b877bf1c03d18fc77d6b69dc867 Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2016-11-28T10:07:15Z FLINK-5132 FLINK-5133 FLINK-5134 [Core,DataStream API,DataSet API] Fine-grained Resource Configuration Summary: above Test Plan: NA Reviewers: yushi.wxg Subscribers: #blink Differential Revision: http://phabricator.taobao.net/D6593 commit 02b0e9d71db86921485ee27dbbb588f6c48fd19e Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2016-11-30T09:13:32Z FLINK-5132 [DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream commit ceed4cb5a9798fb628e82511c08b70636acae9b0 Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2016-11-30T09:43:22Z FLINK-5132 [DataSet API,DataStream API] Introduce the ResourceSpec API for DataSet and DataStream
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user wangzhijiang999 closed the pull request at:

          https://github.com/apache/flink/pull/2906

          Show
          githubbot ASF GitHub Bot added a comment - Github user wangzhijiang999 closed the pull request at: https://github.com/apache/flink/pull/2906
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user wangzhijiang999 opened a pull request:

          https://github.com/apache/flink/pull/3114

          FLINK-5132Introduce the ResourceSpec for grouping different resourc…

          This is part of the fine-grained resource configuration.

          The current resource factors include cpu cores, heap memory, direct memory, native memory and state size.
          The *ResourceSpec* will provide some basic constructions for grouping different resource factors as needed and the construction can also be expanded easily for further requirements.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/wangzhijiang999/flink FLINK-5132

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3114.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3114


          commit 8689e0d4f81de9f2b704d425aa56ca3f5e792f29
          Author: 淘江 <taojiang.wzj@alibaba-inc.com>
          Date: 2017-01-13T10:40:47Z

          FLINK-5132Introduce the ResourceSpec for grouping different resource factors in API


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user wangzhijiang999 opened a pull request: https://github.com/apache/flink/pull/3114 FLINK-5132 Introduce the ResourceSpec for grouping different resourc… This is part of the fine-grained resource configuration. The current resource factors include cpu cores, heap memory, direct memory, native memory and state size. The * ResourceSpec * will provide some basic constructions for grouping different resource factors as needed and the construction can also be expanded easily for further requirements. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangzhijiang999/flink FLINK-5132 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3114 commit 8689e0d4f81de9f2b704d425aa56ca3f5e792f29 Author: 淘江 <taojiang.wzj@alibaba-inc.com> Date: 2017-01-13T10:40:47Z FLINK-5132 Introduce the ResourceSpec for grouping different resource factors in API
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3114

          Looks good, merging this.
          As a followup, I am wondering if it makes sense to check already in the constructors whether values are valid (for example that the amount of memory it not negative).

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3114 Looks good, merging this. As a followup, I am wondering if it makes sense to check already in the constructors whether values are valid (for example that the amount of memory it not negative).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3114

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3114
          Hide
          StephanEwen Stephan Ewen added a comment -

          Implemented in fd872a14974cad14c63478e44edb4e508dfb4069

          Show
          StephanEwen Stephan Ewen added a comment - Implemented in fd872a14974cad14c63478e44edb4e508dfb4069
          Hide
          zjwang zhijiang added a comment -

          Thank you, Stephan Ewen. Yeah, it also makes sense to check valid values in the constructors, currently it provides the isValid method and will be called when setting to the operator.

          Show
          zjwang zhijiang added a comment - Thank you, Stephan Ewen . Yeah, it also makes sense to check valid values in the constructors, currently it provides the isValid method and will be called when setting to the operator.

            People

            • Assignee:
              zjwang zhijiang
              Reporter:
              zjwang zhijiang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development