[SPARK-47574] Introduce Structured Logging Framework

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 26/Mar/24 18:47
• Updated: 05/Apr/24 14:11
• Resolved: 29/Mar/24 05:59

Description

Introduce Structured Logging Framework as per https://docs.google.com/document/d/1rATVGmFLNVLmtxSpWrEceYm7d-ocgu8ofhryVs4g3XU/edit?usp=sharing .

• The default logging output format will be json lines. For example 

  {
     "ts":"2023-03-12T12:02:46.661-0700",
     "level":"ERROR",
     "msg":"Cannot determine whether executor 289 is alive or not",
     "context":{
         "executor_id":"289"
     },
     "exception":{
        "class":"org.apache.spark.SparkException",
        "msg":"Exception thrown in awaitResult",
        "stackTrace":"..."
     },
     "source":"BlockManagerMasterEndpoint"
  } 

• Introduce a new configuration `spark.log.structuredLogging.enabled` to control the default log4j configuration. Users can set it as false to get plain text log outputs
• The change will start with logError method. Example changes on the API: from
  `logError(s"Cannot determine whether executor $executorId is alive or not.", e)` to `logError(log"Cannot determine whether executor ${MDC(EXECUTOR_ID, executorId)} is alive or not.", e)`

 

Issue Links

• links to: GitHub Pull Request #45729