[SPARK-26818] Make MLEvents JSON ser/de safe

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: ML
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 02/Feb/19 15:08
• Updated: 12/Dec/22 18:11
• Resolved: 03/Feb/19 13:20

Description

Looks ML events are not JSON serializable. We can make it serialisable like:

@DeveloperApi
case class SparkListenerSQLExecutionEnd(executionId: Long, time: Long)
  extends SparkListenerEvent {

  // The name of the execution, e.g. `df.collect` will trigger a SQL execution with name "collect".
  @JsonIgnore private[sql] var executionName: Option[String] = None

  // The following 3 fields are only accessed when `executionName` is defined.

  // The duration of the SQL execution, in nanoseconds.
  @JsonIgnore private[sql] var duration: Long = 0L

  // The `QueryExecution` instance that represents the SQL execution
  @JsonIgnore private[sql] var qe: QueryExecution = null

  // The exception object that caused this execution to fail. None if the execution doesn't fail.
  @JsonIgnore private[sql] var executionFailure: Option[Exception] = None
}

.

Issue Links

• links to: GitHub Pull Request #23728