Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18203

Reduce objects usage in redistributing union states

    XMLWordPrintableJSON

Details

    Description

      #RoundRobinOperatorStateRepartitioner#repartitionUnionState creates a new OperatorStreamStateHandle instance for every StreamStateHandle instance used in every execution, which causes the number of new OperatorStreamStateHandle instances up to m * n (jobvertex parallelism * count of all executions' StreamStateHandle).

      But in fact, all executions can share the same collection of StreamStateHandle and the number of OperatorStreamStateHandle can be reduced down to the count of all executions' StreamStateHandle.

      I met this problem on production when we're testing a job with parallelism=10k and the memory problem is getting more serious when yarn containers go dead and the job starts doing failover.

      Attachments

        Activity

          People

            Unassigned Unassigned
            wind_ljy Jiayi Liao
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: