Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19177

FLIP-141: Intra-Slot Managed Memory Sharing

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      The configuration python.fn-execution.buffer.memory.size and python.fn-execution.framework.memory.size have been removed and so will not take effect any more. Besides, the default value of python.fn-execution.memory.managed has been changed to true and so managed memory will be used by default for Python workers. In cases where Python UDFs are used together with RocksDB state backend in streaming or built-in batch algorithms in batch, the user can control how managed memory should be shared between data processing (RocksDB state backend or batch algorithms) and Python, by overwriting [managed memory consumer weights]({% link ops/mem_setup_tm.md %}#consumer-weights).
      Show
      The configuration python.fn-execution.buffer.memory.size and python.fn-execution.framework.memory.size have been removed and so will not take effect any more. Besides, the default value of python.fn-execution.memory.managed has been changed to true and so managed memory will be used by default for Python workers. In cases where Python UDFs are used together with RocksDB state backend in streaming or built-in batch algorithms in batch, the user can control how managed memory should be shared between data processing (RocksDB state backend or batch algorithms) and Python, by overwriting [managed memory consumer weights]({% link ops/mem_setup_tm.md %}#consumer-weights).

      Description

      This is the umbrella ticket of FLIP-141: Intra-Slot Managed Memory Sharing.

      FLIP-53 introduced a fraction based approach for sharing managed memory within a slot. This approach needs to be extended as python operators, which also use managed memory, are introduced. This FLIP proposes a design for extending intra-slot managed memory sharing for python operators and other potential future managed memory use cases.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                xintongsong Xintong Song
                Reporter:
                xintongsong Xintong Song
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: