Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.2.2, 2.3.0, 2.4.0
-
None
-
None
Description
Hello,
it seems that when creating a ChunkedByteBufferOutputStream, the chunk size is always configured to be 4MB. I suggest we make it configurable via spark conf. This would allow to solve issues like SPARK-24917 (by increasing the chunk size to a bigger value in the conf).
What do you think ?