[SPARK-45333] Fix one unit mistake related to spark.eventLog.buffer.kb

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• External issue URL: https://github.com/apache/spark/pull/42294
• Assignee: Mengran Lan
• Reporter: Mengran Lan
• Shepherd: Hyukjin Kwon
• Created: 26/Sep/23 05:51
• Updated: 27/Sep/23 00:38
• Resolved: 26/Sep/23 05:57

Description

The configuration "spark.eventLog.buffer.kb" has a definition with KiB unit.

private[spark] val EVENT_LOG_OUTPUT_BUFFER_SIZE = ConfigBuilder("spark.eventLog.buffer.kb")
  .doc("Buffer size to use when writing to output streams, in KiB unless otherwise specified.")
  .version("1.0.0")
  .bytesConf(ByteUnit.KiB)
  .createWithDefaultString("100k")

It is used in a context acting as the size of bytes.

protected val outputBufferSize = sparkConf.get(EVENT_LOG_OUTPUT_BUFFER_SIZE).toInt 

This may be a mistake and needs to be fixed.

The default buffer size for the event-log output buffer will be 100k as configured.

Issue Links

• links to: GitHub Pull Request #42294