Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31769

Add support of MDC in spark driver logs.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      In order to align driver logs with spark executors logs, add support to log MDC state for properties that are shared between the driver the server executors (e.g. application name. task id)
      The use case is applicable in 2 cases:

      1. streaming
      2. running spark under job server (in this case you have a long-running spark context that handles many tasks from different sources).

      See also SPARK-8981 that handles the MDC in executor context.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              igreenfi Izek Greenfield
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: