Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5131

Fine-grained Resource Configuration

    XMLWordPrintableJSON

Details

    Description

      Normally the UDF just creates short-life small objects and these can be recycled quickly by JVM, so most of the memory resource is controlled and managed by TaskManager framework. But for some special cases, the UDF may consume much resource to create long-live big objects, so it is necessary to provide the options for professional users to define the resource usages if needed.

      The basic approach is the following:

      • Introduce the ResourceSpec structure to describe the different resource factors (cpu cores, heap memory, direct memory, native memory, etc) and provide some basic construction methods for resource group.
      • The ResourceSpec can be setted onto the internal transformation in DataStream and base operator in DataSet separately.
      • The StateBackend should provide the method to estimate the memory usage based on hint of state size in ResourceSpec.
      • In job graph generation, the ResourceSpec will be aggregated for chained operators.
      • When JobManager requests slot for submitting task from ResourceManager, the ResourceProfile will be expanded to correspond with ResourceSpec.
      • The ResourceManager requests resource for container from cluster, it should consider extra framework memory except for slot ResourceProfile.
      • The framework memory is mainly used by NetworkBufferPool and MemoryManager in TaskManager, and it can be configured in job level.
      • Apart from resource, The JVM options attached with container should be supported and could also be configured in job level.

      This feature will be implemented directly into flip-6 branch.

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-9096 flip 6 allocates different resources for different jobs

          • Major - Major loss of function.
          • Closed
          incorporates

          Sub-task - The sub-task of the issue FLINK-5132 Introduce the ResourceSpec for grouping different resource factors in API

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-5133 Support to set resource for operator in DataStream and DataSet

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-5134 Aggregate ResourceSpec for chained operators when generating job graph

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-5135 ResourceProfile for slot request should be expanded to correspond with ResourceSpec

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-5782 Support GPU calculations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is superceded by

          New Feature - A new feature of the product, which has yet to be developed. FLINK-14058 FLIP-53 Fine-grained Operator Resource Management

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Fine-grained Resource Configuration

          Activity

            People

              Unassigned Unassigned
              zjwang Zhijiang
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: