Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47240

SPIP: Structured Logging Framework for Apache Spark

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0
    • None
    • Spark Core

    Description

      This proposal aims to enhance Apache Spark's logging system by implementing structured logging. This transition will change the format of the default log files from plain text to JSON, making them more accessible and analyzable. The new logs will include crucial identifiers such as worker, executor, query, job, stage, and task IDs, thereby making the logs more informative and facilitating easier search and analysis.

      Current Logging Format

      The current format of Spark logs is plain text, which can be challenging to parse and analyze efficiently. An example of the current log format is as follows:

      23/11/29 17:53:44 ERROR BlockManagerMasterEndpoint: Fail to know the executor 289 is alive or not.
      org.apache.spark.SparkException: Exception thrown in awaitResult: 
          <stacktrace…>
      Caused by: org.apache.spark.rpc.RpcEndpointNotFoundException: ..
      

      Proposed Structured Logging Format

      The proposed change involves structuring the logs in JSON format, which organizes the log information into easily identifiable fields. Here is how the new structured log format would look:

      {
         "ts":"23/11/29 17:53:44",
         "level":"ERROR",
         "msg":"Fail to know the executor 289 is alive or not",
         "context":{
             "executor_id":"289"
         },
         "exception":{
            "class":"org.apache.spark.SparkException",
            "msg":"Exception thrown in awaitResult",
            "stackTrace":"..."
         },
         "source":"BlockManagerMasterEndpoint"
      } 

      This format will enable users to upload and directly query driver/executor/master/worker log files using Spark SQL for more effective problem-solving and analysis, such as tracking executor losses or identifying faulty tasks:

      spark.read.json("hdfs://hdfs_host/logs").createOrReplaceTempView("logs")
      /* To get all the executor lost logs */
      SELECT * FROM logs WHERE contains(message, 'Lost executor');
      /* To get all the distributed logs about executor 289 */
      SELECT * FROM logs WHERE executor_id = 289;
      /* To get all the errors on host 100.116.29.4 */
      SELECT * FROM logs WHERE host = "100.116.29.4" and log_level="ERROR";
      

       

      SPIP doc: https://docs.google.com/document/d/1rATVGmFLNVLmtxSpWrEceYm7d-ocgu8ofhryVs4g3XU/edit?usp=sharing

      Attachments

        1.
        Introduce Structured Logging Framework Sub-task Resolved Gengliang Wang
        2.
        Implement logWarning API in structured logging framework Sub-task Resolved Gengliang Wang
        3.
        Implement logInfo API in structured logging framework Sub-task Resolved Gengliang Wang
        4.
        Spark core: Migrate logError with variables to structured logging framework Sub-task Resolved Gengliang Wang
        5.
        Hive module: Migrate logInfo with variables to structured logging framework Sub-task Resolved Gengliang Wang
        6.
        Hive-thriftserver: Migrate logError with variables to structured logging framework Sub-task Resolved Gengliang Wang
        7.
        SQL catalyst: Migrate logError with variables to structured logging framework Sub-task Resolved Gengliang Wang
        8.
        Add documentation for Structured logging framework Sub-task Resolved Gengliang Wang
        9.
        Structured logging framework: support log concatenation Sub-task Resolved Gengliang Wang
        10.
        Enable structured logging in log4j2.properties.template and update `configuration.md` Sub-task Resolved Gengliang Wang
        11.
        Add scala style check for invalid MDC usage Sub-task Resolved Gengliang Wang
        12.
        Guidelines for the Structured Logging Framework Sub-task Resolved Gengliang Wang
        13.
        Use milliseconds as the time unit in loggings Sub-task Resolved Gengliang Wang
        14.
        Store LogKey name as a value to avoid generating new string instances Sub-task Resolved Gengliang Wang
        15.
        Provide a constant table schema for querying structured logs Sub-task Resolved Gengliang Wang
        16.
        Improve the readability of JSON loggings Sub-task Resolved Gengliang Wang
        17.
        Disable structured logging for Interpreter by default Sub-task Resolved Gengliang Wang
        18.
        Provide options to enable structured logging for Interpreter Sub-task Open Gengliang Wang
        19.
        Make spark.log.structuredLogging.enabled effective Sub-task Resolved Gengliang Wang
        20.
        Provide a constant table schema in PySpark for querying structured logs Sub-task Resolved Gengliang Wang
        21.
        Unify MDC key `mdc.taskName` and `task_name` Sub-task Resolved Gengliang Wang
        22.
        Remove logDebug and logTrace with MDC in java structured logging framework Sub-task Resolved Gengliang Wang
        23.
        Rename Java Logger as SparkLogger Sub-task Resolved Gengliang Wang
        24.
        Change the scope of object LogKeys as private in Spark Sub-task Resolved Gengliang Wang
        25.
        Log entry should be constructed only once Sub-task Resolved Gengliang Wang
        26.
        Include SPARK_LOG_SCHEMA in the context of repl shell Sub-task Resolved Gengliang Wang
        27.
        SQL core: Migrate logWarn with variables to structured logging framework Sub-task Resolved Pan Bingkun
        28.
        SQL core: Migrate logInfo with variables to structured logging framework Sub-task Resolved Pan Bingkun
        29.
        Hive module: Migrate logWarn with variables to structured logging framework Sub-task Resolved Pan Bingkun
        30.
        Connector module: Migrate logError with variables to structured logging framework Sub-task Resolved Pan Bingkun
        31.
        Connector module: Migrate logWarn with variables to structured logging framework Sub-task Resolved Pan Bingkun
        32.
        Connector module: Migrate logInfo with variables to structured logging framework Sub-task Resolved Pan Bingkun
        33.
        Streaming: Migrate logError with variables to structured logging framework Sub-task Resolved Pan Bingkun
        34.
        Streaming: Migrate logWarn with variables to structured logging framework Sub-task Resolved Pan Bingkun
        35.
        MLLib: Migrate logError with variables to structured logging framework Sub-task Resolved Pan Bingkun
        36.
        MLLib: Migrate logWarn with variables to structured logging framework Sub-task Resolved Pan Bingkun
        37.
        Resource managers: Migrate logError with variables to structured logging framework Sub-task Resolved Pan Bingkun
        38.
        Resource managers: Migrate logWarn with variables to structured logging framework Sub-task Resolved Pan Bingkun
        39.
        Resource managers: Migrate logInfo with variables to structured logging framework Sub-task Resolved Pan Bingkun
        40.
        Improve `*LoggingSuite*` Sub-task Resolved Pan Bingkun
        41.
        Introduce a tool that can sort alphabetically enumeration field in `LogEntry` automatically Sub-task Resolved Pan Bingkun
        42.
        Make the value of MDC can support `null` Sub-task Resolved Pan Bingkun
        43.
        Make the external Spark ecosystem can use structured logging mechanisms Sub-task Resolved Pan Bingkun
        44.
        Implement the structured log framework on the java side Sub-task Resolved Pan Bingkun
        45.
        Spark core (java side): Migrate `error/warn/info` with variables to structured logging framework Sub-task Resolved Pan Bingkun
        46.
        SQL (java side): Migrate `error/warn/info` with variables to structured logging framework Sub-task Resolved Pan Bingkun
        47.
        Common (java side): Migrate `error/warn/info` with variables to structured logging framework Sub-task Resolved Pan Bingkun
        48.
        Ban import `org.slf4j.Logger` & `org.slf4j.LoggerFactory` Sub-task Resolved Pan Bingkun
        49.
        Reorganize `LogKey` Sub-task Resolved Pan Bingkun
        50.
        Add external third-party ecosystem access guide to the doc Sub-task Resolved Pan Bingkun
        51.
        Migrate the remaining code to structured logging framework Sub-task Resolved Pan Bingkun
        52.
        Spark core: Migrate logInfo with variables to structured logging framework Sub-task Resolved Anh Tuan Pham
        53.
        Spark core: Migrate logWarn with variables to structured logging framework Sub-task Resolved Daniel
        54.
        SQL catalyst: Migrate logWarn with variables to structured logging framework Sub-task Resolved Daniel
        55.
        SQL catalyst: Migrate logInfo with variables to structured logging framework Sub-task Resolved Daniel
        56.
        SQL core: Migrate logError with variables to structured logging framework Sub-task Resolved Daniel
        57.
        Hive module: Migrate logError with variables to structured logging framework Sub-task Resolved Haejoon Lee
        58.
        Hive-thriftserver: Migrate logWarn with variables to structured logging framework Sub-task Resolved Haejoon Lee
        59.
        Hive-thriftserver: Migrate logInfo with variables to structured logging framework Sub-task Resolved Haejoon Lee
        60.
        Streaming: Migrate logInfo with variables to structured logging framework Sub-task Resolved Daniel
        61.
        MLLib: Migrate logInfo with variables to structured logging framework Sub-task Resolved Anh Tuan Pham
        62.
        Graphx: Migrate logs with variables to structured logging framework Sub-task Resolved Haejoon Lee
        63.
        Enable structured logging in all the test log4j2.properties Sub-task Open Unassigned
        64.
        Create log4j templates for both structured logging and plain text logging Sub-task Resolved Unassigned
        65.
        Improve user experience of loading logs as json data source Sub-task Resolved Unassigned
        66.
        Support `three` methods of the log `concatenation` in the `structured logging framework` Sub-task Open Unassigned
        67.
        Sort LogKey alphabetically Sub-task Resolved Daniel
        68.
        RPackageUtils: Migrate logInfo/logWarn/logError with variables to structured logging framework Sub-task Open Unassigned
        69.
        Add scala style check for logging message inline variables Sub-task Resolved Amanda Liu
        70.
        Structured Logging Framework Scala Style Migration Sub-task Resolved Amanda Liu
        71.
        Remove unused LogKeys Sub-task Resolved Unassigned
        72.
        Structured Logging Framework Scala Style Migration [Part 2] Sub-task Open Unassigned
        73.
        Add Streaming related fields to log4j ThreadContext Sub-task Resolved Wei Liu
        74.
        Fix to not log message context with explicitly LogEntry constructed when Structured Logging conf is off Sub-task Resolved Xinyi Yu
        75.
        Fix incorrect logkeys Sub-task Open Unassigned

        Activity

          People

            Unassigned Unassigned
            Gengliang.Wang Gengliang Wang
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: