[SPARK-25645] Add provision to disable EventLoggingListener default flush/hsync/hflush for all events

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.2
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Devaraj Kavali
• Created: 04/Oct/18 22:57
• Updated: 05/Oct/18 06:13
• Resolved: 05/Oct/18 06:13

Description

EventLoggingListener.scala

private def logEvent(event: SparkListenerEvent, flushLogger: Boolean = false) {
    val eventJson = JsonProtocol.sparkEventToJson(event)
    // scalastyle:off println
    writer.foreach(_.println(compact(render(eventJson))))
    // scalastyle:on println
    if (flushLogger) {
      writer.foreach(_.flush())
      hadoopDataStream.foreach(ds => ds.getWrappedStream match {
        case wrapped: DFSOutputStream => wrapped.hsync(EnumSet.of(SyncFlag.UPDATE_LENGTH))
        case _ => ds.hflush()
      })
    }

There are events which come with flushLogger=true and go through the underlying stream flush, Here I tried running apps with disabling the flush/hsync/hflush for all events and see that there is significant improvement in the app completion time and also there are no event drops, posting more details in the comments section.

Issue Links

• is duplicated by: SPARK-24787 Events being dropped at an alarming rate due to hsync being slow for eventLogging