Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47240

SPIP: Structured Logging Framework for Apache Spark

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0
    • None
    • Spark Core

    Description

      This proposal aims to enhance Apache Spark's logging system by implementing structured logging. This transition will change the format of the default log files from plain text to JSON, making them more accessible and analyzable. The new logs will include crucial identifiers such as worker, executor, query, job, stage, and task IDs, thereby making the logs more informative and facilitating easier search and analysis.

      Current Logging Format

      The current format of Spark logs is plain text, which can be challenging to parse and analyze efficiently. An example of the current log format is as follows:

      23/11/29 17:53:44 ERROR BlockManagerMasterEndpoint: Fail to know the executor 289 is alive or not.
      org.apache.spark.SparkException: Exception thrown in awaitResult: 
          <stacktrace…>
      Caused by: org.apache.spark.rpc.RpcEndpointNotFoundException: ..
      

      Proposed Structured Logging Format

      The proposed change involves structuring the logs in JSON format, which organizes the log information into easily identifiable fields. Here is how the new structured log format would look:

      {
         "ts":"23/11/29 17:53:44",
         "level":"ERROR",
         "msg":"Fail to know the executor 289 is alive or not",
         "context":{
             "executor_id":"289"
         },
         "exception":{
            "class":"org.apache.spark.SparkException",
            "msg":"Exception thrown in awaitResult",
            "stackTrace":"..."
         },
         "source":"BlockManagerMasterEndpoint"
      } 

      This format will enable users to upload and directly query driver/executor/master/worker log files using Spark SQL for more effective problem-solving and analysis, such as tracking executor losses or identifying faulty tasks:

      spark.read.json("hdfs://hdfs_host/logs").createOrReplaceTempView("logs")
      /* To get all the executor lost logs */
      SELECT * FROM logs WHERE contains(message, 'Lost executor');
      /* To get all the distributed logs about executor 289 */
      SELECT * FROM logs WHERE executor_id = 289;
      /* To get all the errors on host 100.116.29.4 */
      SELECT * FROM logs WHERE host = "100.116.29.4" and log_level="ERROR";
      

       

      SPIP doc: https://docs.google.com/document/d/1rATVGmFLNVLmtxSpWrEceYm7d-ocgu8ofhryVs4g3XU/edit?usp=sharing

      Attachments

        1.
        Introduce Structured Logging Framework Sub-task Resolved Gengliang Wang
        2.
        Implement logWarning API in structured logging framework Sub-task Resolved Gengliang Wang
        3.
        Implement logInfo API in structured logging framework Sub-task Resolved Gengliang Wang
        4.
        Spark core: Migrate logError with variables to structured logging framework Sub-task Resolved Gengliang Wang
        5.
        Spark core: Migrate logInfo with variables to structured logging framework Sub-task Resolved Anh Tuan Pham
        6.
        SQL catalyst: Migrate logError with variables to structured logging framework Sub-task Resolved Gengliang Wang
        7.
        Spark core: Migrate logWarn with variables to structured logging framework Sub-task Resolved Daniel
        8.
        SQL catalyst: Migrate logWarn with variables to structured logging framework Sub-task Resolved Daniel
        9.
        SQL catalyst: Migrate logInfo with variables to structured logging framework Sub-task Resolved Daniel
        10.
        SQL core: Migrate logError with variables to structured logging framework Sub-task Resolved Daniel
        11.
        SQL core: Migrate logWarn with variables to structured logging framework Sub-task Resolved BingKun Pan
        12.
        SQL core: Migrate logInfo with variables to structured logging framework Sub-task Resolved BingKun Pan
        13.
        Hive module: Migrate logError with variables to structured logging framework Sub-task Resolved Haejoon Lee
        14.
        Hive module: Migrate logWarn with variables to structured logging framework Sub-task Resolved BingKun Pan
        15.
        Hive module: Migrate logInfo with variables to structured logging framework Sub-task Resolved Gengliang Wang
        16.
        Hive-thriftserver: Migrate logError with variables to structured logging framework Sub-task Resolved Gengliang Wang
        17.
        Hive-thriftserver: Migrate logWarn with variables to structured logging framework Sub-task Resolved Haejoon Lee
        18.
        Hive-thriftserver: Migrate logInfo with variables to structured logging framework Sub-task Resolved Haejoon Lee
        19.
        Connector module: Migrate logError with variables to structured logging framework Sub-task Resolved BingKun Pan
        20.
        Connector module: Migrate logWarn with variables to structured logging framework Sub-task Resolved BingKun Pan
        21.
        Connector module: Migrate logInfo with variables to structured logging framework Sub-task Resolved BingKun Pan
        22.
        Streaming: Migrate logError with variables to structured logging framework Sub-task Resolved BingKun Pan
        23.
        Streaming: Migrate logWarn with variables to structured logging framework Sub-task Resolved BingKun Pan
        24.
        Streaming: Migrate logInfo with variables to structured logging framework Sub-task Resolved Daniel
        25.
        MLLib: Migrate logError with variables to structured logging framework Sub-task Resolved BingKun Pan
        26.
        MLLib: Migrate logWarn with variables to structured logging framework Sub-task Resolved BingKun Pan
        27.
        MLLib: Migrate logInfo with variables to structured logging framework Sub-task Resolved Anh Tuan Pham
        28.
        Graphx: Migrate logs with variables to structured logging framework Sub-task Resolved Haejoon Lee
        29.
        Resource managers: Migrate logError with variables to structured logging framework Sub-task Resolved BingKun Pan
        30.
        Resource managers: Migrate logWarn with variables to structured logging framework Sub-task Resolved BingKun Pan
        31.
        Resource managers: Migrate logInfo with variables to structured logging framework Sub-task Resolved BingKun Pan
        32.
        Enable structured logging in all the test log4j2.properties Sub-task Open Unassigned
        33.
        Create log4j templates for both structured logging and plain text logging Sub-task Resolved Unassigned
        34.
        Add documentation for Structured logging framework Sub-task Resolved Gengliang Wang
        35.
        Improve user experience of loading logs as json data source Sub-task Resolved Unassigned
        36.
        Structured logging framework: support log concatenation Sub-task Resolved Gengliang Wang
        37.
        Support `three` methods of the log `concatenation` in the `structured logging framework` Sub-task Open Unassigned
        38.
        Enable structured logging in log4j2.properties.template and update `configuration.md` Sub-task Resolved Gengliang Wang
        39.
        Improve `*LoggingSuite*` Sub-task Resolved BingKun Pan
        40.
        Add scala style check for invalid MDC usage Sub-task Resolved Gengliang Wang
        41.
        Sort LogKey alphabetically Sub-task Resolved Daniel
        42.
        Introduce a tool that can sort alphabetically enumeration field in `LogEntry` automatically Sub-task Resolved BingKun Pan
        43.
        Guidelines for the Structured Logging Framework Sub-task Resolved Gengliang Wang
        44.
        Use milliseconds as the time unit in loggings Sub-task Resolved Gengliang Wang
        45.
        Make the value of MDC can support `null` Sub-task Resolved BingKun Pan
        46.
        Make the external Spark ecosystem can use structured logging mechanisms Sub-task Resolved BingKun Pan
        47.
        Store LogKey name as a value to avoid generating new string instances Sub-task Resolved Gengliang Wang
        48.
        Implement the structured log framework on the java side Sub-task Resolved BingKun Pan
        49.
        Improve the readability of JSON loggings Sub-task Resolved Gengliang Wang
        50.
        RPackageUtils: Migrate logInfo/logWarn/logError with variables to structured logging framework Sub-task Open Unassigned
        51.
        Provide a constant table schema for querying structured logs Sub-task Resolved Gengliang Wang
        52.
        Disable structured logging for Interpreter by default Sub-task Resolved Gengliang Wang
        53.
        Provide options to enable structured logging for Interpreter Sub-task Open Gengliang Wang
        54.
        Make spark.log.structuredLogging.enabled effective Sub-task Resolved Gengliang Wang
        55.
        Provide a constant table schema in PySpark for querying structured logs Sub-task Resolved Gengliang Wang
        56.
        Unify MDC key `mdc.taskName` and `task_name` Sub-task Resolved Gengliang Wang
        57.
        Spark core (java side): Migrate `error/warn/info` with variables to structured logging framework Sub-task Resolved BingKun Pan
        58.
        Remove logDebug and logTrace with MDC in java structured logging framework Sub-task Resolved Gengliang Wang
        59.
        SQL (java side): Migrate `error/warn/info` with variables to structured logging framework Sub-task Resolved BingKun Pan
        60.
        Common (java side): Migrate `error/warn/info` with variables to structured logging framework Sub-task Resolved BingKun Pan
        61.
        Ban import `org.slf4j.Logger` & `org.slf4j.LoggerFactory` Sub-task Resolved BingKun Pan
        62.
        Rename Java Logger as SparkLogger Sub-task Resolved Gengliang Wang
        63.
        Reorganize `LogKey` Sub-task Resolved BingKun Pan
        64.
        Add external third-party ecosystem access guide to the doc Sub-task Resolved BingKun Pan
        65.
        Add scala style check for logging message inline variables Sub-task Resolved Amanda Liu
        66.
        Structured Logging Framework Scala Style Migration Sub-task Resolved Amanda Liu
        67.
        Change the scope of object LogKeys as private in Spark Sub-task Resolved Gengliang Wang
        68.
        Migrate the remaining code to structured logging framework Sub-task Resolved BingKun Pan
        69.
        Remove unused LogKeys Sub-task Resolved Unassigned
        70.
        Structured Logging Framework Scala Style Migration [Part 2] Sub-task Open Unassigned
        71.
        Log entry should be constructed only once Sub-task Resolved Gengliang Wang
        72.
        Add Streaming related fields to log4j ThreadContext Sub-task Resolved Wei Liu
        73.
        Fix to not log message context with explicitly LogEntry constructed when Structured Logging conf is off Sub-task Resolved Xinyi Yu

        Activity

          People

            Unassigned Unassigned
            Gengliang.Wang Gengliang Wang
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: