Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45333

Fix one unit mistake related to spark.eventLog.buffer.kb

    XMLWordPrintableJSON

Details

    Description

      The configuration "spark.eventLog.buffer.kb" has a definition with KiB unit.

      private[spark] val EVENT_LOG_OUTPUT_BUFFER_SIZE = ConfigBuilder("spark.eventLog.buffer.kb")
        .doc("Buffer size to use when writing to output streams, in KiB unless otherwise specified.")
        .version("1.0.0")
        .bytesConf(ByteUnit.KiB)
        .createWithDefaultString("100k")

      It is used in a context acting as the size of bytes.

      protected val outputBufferSize = sparkConf.get(EVENT_LOG_OUTPUT_BUFFER_SIZE).toInt 

      This may be a mistake and needs to be fixed.

      The default buffer size for the event-log output buffer will be 100k as configured.

      Attachments

        Issue Links

          Activity

            People

              amoylan Mengran Lan
              amoylan Mengran Lan
              Hyukjin Kwon Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: