Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.12.0
Description
Motivation
Hash-based blocking shuffle and sort-merge based blocking shuffle are two main blocking shuffle implementations wildly adopted by existing distributed data processing frameworks. Hash-based implementation writes data sent to different reducer tasks into separate files concurrently while sort-merge based approach writes those data together into a single file and merges those small files into bigger ones. Compared to sort-merge based approach, hash-based approach has several weak points when it comes to running large scale batch jobs:
1. Stability
For high parallelism (tens of thousands) batch job, current hash-based blocking shuffle implementation writes too many files concurrently which gives high pressure to the file system, for example, maintenance of too many file metas, high system cpu consumption and exhaustion of inodes or file descriptors. All of these can be potential stability issues which we encountered in our production environment before we switch to sort-merge based blocking shuffle.
Sort-Merge based blocking shuffle don’t have the problem because for one result partition, only one file is written at the same time.
2. Performance
Large amounts of small shuffle files and random io can influence shuffle performance a lot especially for hdd (for ssd, sequential read is also important because of read ahead and cache).
For batch job processing massive data, small amount of data per subpartition is common, because to reduce the job completion time, we usually increase the job parallelism to reduce the amount of data processed per task and the average data amount per subpartition is relevant to:
(the amount of data per task) / (parallelism) = (total amount of data) / (parallelism^2)
which means increasing parallelism can decrease the amount of data per subpartition rapidly.
Besides, data skew is another cause of small subpartition files. By merging data of all subpartitions together in one file, more sequential read can be achieved.
3. Resource
For current hash-based implementation, each subpartition needs at least one buffer. For large scale batch shuffles, the memory consumption can be huge. For example, we need at least 320M network memory per result partition if parallelism is set to 10000 and because of the huge network consumption, it is hard to config the network memory for large scale batch job and sometimes parallelism can not be increased just because of insufficient network memory which leads to bad user experience.
By introducing the sort-merge based approach to Flink, we can improve Flink’s capability of running large scale batch jobs.
Attachments
Issue Links
- links to