Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Invalid
-
0.9.0, 1.0.0
-
None
-
None
Description
Right now, Spark uses the total usage of "shuffle" memory of each thread to predict if need to spill. I think it is not very reasonable. For example, there are two threads pulling "shuffle" data. The total memory used to buffer data is 21G. The first time to trigger spilling it when one thread has used 7G memory to buffer "shuffle" data, here I assume another one has used the same size. Unfortunately, I still have remaining 7G to use. So, I think current prediction mode is too conservative, and can not maximize the usage of "shuffle" memory. In my solution, I use the growth rate of "shuffle" memory. Again, the growth of each time is limited, maybe 10K * 1024(my assumption), then the first time to trigger spilling is when the remaining "shuffle" memory is less than threads * growth * 2, i.e. 2 * 10M * 2. I think it can maximize the usage of "shuffle" memory. In my solution, there is also a conservative assumption, i.e. all of threads is pulling shuffle data in one executor. However it dose not have much effect, the grow is limited after all. Any suggestion?
