Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-33004

Decoupling topology and network memory to support complex job topologies

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 1.18.0, 1.19.0
    • None
    • Runtime / Network
    • None

    Description

      Currently, the default value of taskmanager.memory.network.fraction option in Flink is 0.1, and after the topology of the job is complex enough, it will run with an insufficient network buffer. We currently encountered this issue when running TPC-DS test set q9, and bypassed it by adjusting taskmanager.memory.network.fraction to 0.2. Theoretically, we should have network memory decoupled from the job topology so that arbitrarily complex jobs can be supported.

      Attachments

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-33668 Decoupling Shuffle network memory and job topology

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              lsy dalongliu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: