[SPARK-50036] Include SPARK_LOG_SCHEMA in the context of repl shell

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 18/Oct/24 23:07
• Updated: 21/Oct/24 17:42
• Resolved: 21/Oct/24 17:42

Description

Before the Change:

Users needed to import LOG_SCHEMA to read structured logs as a JSON data source:
import org.apache.spark.util.LogUtils.LOG_SCHEMA

val logDf = spark.read.schema(LOG_SCHEMA).json("path/to/logs")
 
After the Change:

• No Import Needed in REPL Shells: You can now use SPARK_LOG_SCHEMA directly in REPL environments like spark-shell and pyspark without importing it.
• Renamed for Clarity: LOG_SCHEMA has been renamed to SPARK_LOG_SCHEMA to make its purpose more clear.

Now, you can read structured logs without the import:
val logDf = spark.read.schema(SPARK_LOG_SCHEMA).json("path/to/logs")
 

Issue Links

• links to: GitHub Pull Request #48552