Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
1.4.2, 1.5.2, 1.6.0
-
None
-
None
-
None
Description
I found that the dataOutputSerializer clear the intermediate buffer when the buffer exceed the 5M (as a fixed configuration)， But when we encountered the rebalance or keyByPartition and the downstream has a large parallel it will also consume a lot memory, we can do two things :
1. make this config configurable
2. like the https://issues.apache.org/jira/projects/FLINK/issues/FLINK-1326 mentioned, we can make the serializer one for the output
What's your idea ? StephanEwen