Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13980

FLIP-49 Unified Memory Configuration for TaskExecutors

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      With FLINK-13980, a new memory model has been introduced for the task executor. New configuration options have been introduced to control the memory consumption of the task executor process. This affects all types of deployments: standalone, YARN, Mesos, and the new active Kubernetes integration. The memory model of the job manager process has not been changed yet but it is planned to be updated as well.

      If you try to reuse your previous Flink configuration without any adjustments, the new memory model can result in differently computed memory parameters for the JVM and, thus, performance changes.

      Please note that the following options have been removed and have no effect anymore: "taskmanager.memory.fraction", "taskmanager.memory.off-heap", "taskmanager.memory.preallocate".

      The following options, if used, are interpreted as other new options in order to maintain backwards compatibility where it makes sense: "taskmanager.heap.size", "taskmanager.memory.size", "taskmanager.network.memory.min", "taskmanager.network.memory.max", "taskmanager.network.memory.fraction".


      The container cut-off configuration options, `containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`, have no effect for task executor processes anymore but they still have the same semantics for the JobManager process.

      More details, please refer to Flink user documents.
      Show
      With FLINK-13980 , a new memory model has been introduced for the task executor. New configuration options have been introduced to control the memory consumption of the task executor process. This affects all types of deployments: standalone, YARN, Mesos, and the new active Kubernetes integration. The memory model of the job manager process has not been changed yet but it is planned to be updated as well. If you try to reuse your previous Flink configuration without any adjustments, the new memory model can result in differently computed memory parameters for the JVM and, thus, performance changes. Please note that the following options have been removed and have no effect anymore: "taskmanager.memory.fraction", "taskmanager.memory.off-heap", "taskmanager.memory.preallocate". The following options, if used, are interpreted as other new options in order to maintain backwards compatibility where it makes sense: "taskmanager.heap.size", "taskmanager.memory.size", "taskmanager.network.memory.min", "taskmanager.network.memory.max", "taskmanager.network.memory.fraction". The container cut-off configuration options, `containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`, have no effect for task executor processes anymore but they still have the same semantics for the JobManager process. More details, please refer to Flink user documents.

      Description

      This is the umbrella issue of 'FLIP-49: Unified Memory Configuration for TaskExecutors'.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                xintongsong Xintong Song
                Reporter:
                xintongsong Xintong Song
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h 50m
                  3h 50m