Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13980

FLIP-49 Unified Memory Configuration for TaskExecutors

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Hide
      With FLINK-13980, a new memory model has been introduced for the task executor. New configuration options have been introduced to control the memory consumption of the task executor process. This affects all types of deployments: standalone, YARN, Mesos, and the new active Kubernetes integration. The memory model of the job manager process has not been changed yet but it is planned to be updated as well.

      If you try to reuse your previous Flink configuration without any adjustments, the new memory model can result in differently computed memory parameters for the JVM and, thus, performance changes.

      Please note that the following options have been removed and have no effect anymore: "taskmanager.memory.fraction", "taskmanager.memory.off-heap", "taskmanager.memory.preallocate".

      The following options, if used, are interpreted as other new options in order to maintain backwards compatibility where it makes sense: "taskmanager.heap.size", "taskmanager.memory.size", "taskmanager.network.memory.min", "taskmanager.network.memory.max", "taskmanager.network.memory.fraction".


      The container cut-off configuration options, `containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`, have no effect for task executor processes anymore but they still have the same semantics for the JobManager process.

      More details, please refer to Flink user documents.
      Show
      With FLINK-13980 , a new memory model has been introduced for the task executor. New configuration options have been introduced to control the memory consumption of the task executor process. This affects all types of deployments: standalone, YARN, Mesos, and the new active Kubernetes integration. The memory model of the job manager process has not been changed yet but it is planned to be updated as well. If you try to reuse your previous Flink configuration without any adjustments, the new memory model can result in differently computed memory parameters for the JVM and, thus, performance changes. Please note that the following options have been removed and have no effect anymore: "taskmanager.memory.fraction", "taskmanager.memory.off-heap", "taskmanager.memory.preallocate". The following options, if used, are interpreted as other new options in order to maintain backwards compatibility where it makes sense: "taskmanager.heap.size", "taskmanager.memory.size", "taskmanager.network.memory.min", "taskmanager.network.memory.max", "taskmanager.network.memory.fraction". The container cut-off configuration options, `containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`, have no effect for task executor processes anymore but they still have the same semantics for the JobManager process. More details, please refer to Flink user documents.

    Description

      This is the umbrella issue of 'FLIP-49: Unified Memory Configuration for TaskExecutors'.

      Attachments

        Issue Links

        causes

        Bug - A problem which impairs or prevents the functions of the product. FLINK-15103 Performance regression on 3.12.2019 in various benchmarks

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. FLINK-15925 TaskExecutors don't work out-of-the-box on Windows

        • Major - Major loss of function.
        • Closed
        contains

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-9904 Allow users to control MaxDirectMemorySize

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed
        duplicates

        Sub-task - The sub-task of the issue FLINK-12762 Memory configuration implementation

        • Major - Major loss of function.
        • Closed
        is blocked by

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15763 Check or set necessary resource configuration options to defaults for local execution ignoring FLIP-49

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved
        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-13477 Containerized TaskManager killed because of lack of memory overhead

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed
        is related to

        Sub-task - The sub-task of the issue FLINK-14489 Check and make sure state backend fits into TaskExecutor memory configuration

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. FLINK-15082 Mesos App Master does not respect taskmanager.memory.total-process.size

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. FLINK-15300 Shuffle memory fraction sanity check does not account for its min/max limit

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. FLINK-18153 flink-daemon does not detect errors if JVM crashes immediately

        • Major - Major loss of function.
        • Closed

        New Feature - A new feature of the product, which has yet to be developed. FLINK-14883 Resource management on state backends

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15774 Consider adding an explicit network memory size config option

        • Not a Priority -
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15145 Tune default values for FLIP-49 TM memory configurations with real production jobs.

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15597 Relax sanity check of JVM memory overhead to be within its min/max

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-14431 Update TaskManager's memory information to match its memory composition

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15198 Remove deprecated mesos.resourcemanager.tasks.mem in 1.11

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15517 Use back 'network' in 'shuffle' memory config option names

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15530 Replace process memory with flink memory for TMs in default flink-conf.yaml

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15692 Enable limiting RocksDB memory consumption by default

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-16406 Increase default value for JVM Metaspace to minimise its OutOfMemoryError

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15989 Rewrap OutOfMemoryError in allocateUnpooledOffHeap with better message

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-16278 Document new settings for MaxMetaSpace in 1.10

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-16614 FLIP-116 Unified Memory Configuration for Job Manager

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-17503 Make memory configuration logging more user-friendly

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-20171 Improve error message for Flink process memory configuration

        • Major - Major loss of function.
        • Closed
        relates to

        Sub-task - The sub-task of the issue FLINK-15143 Create document for FLIP-49 TM memory model and configuration guide

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed

        Sub-task - The sub-task of the issue FLINK-17973 Test memory configuration of Flink cluster

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. FLINK-14894 HybridOffHeapUnsafeMemorySegmentTest#testByteBufferWrap failed on Travis

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-15144 Document performance related changes caused by FLIP-49

        • Blocker - Blocks development and/or testing work, production could not run
        • Closed
        links to

        Web Link FLIP-49: Unified Memory Configuration for TaskExecutors

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            xtsong Xintong Song
            xtsong Xintong Song
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 50m
                3h 50m

                Slack

                  Issue deployment