Major Description
Right now sort-based shuffle is slower than hash-based for operations like groupByKey where data is passed straight from the map task to the reduce task, because it keeps building up a buffer in memory and spilling it to disk instead of directly opening more files (thus having more GC pressure), and then it has to read back and merge all the spills (incurring both serialization cost and GC pressure). When the number of partitions is small enough (say less than 100 or 200), we should just open N files, write stuff to them as in hash-based shuffle, and then concatenate them at the end to still end up with a single file (avoiding the many-files problem of the hash-based implementation).
It may also be possible to avoid the concatenation but that introduces complexity in serving the files, so I'd try concatenating them for now. We can benchmark it to see the performance.
Attachments
Issue Links
- links to
