Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18535

Redact sensitive information from Spark logs and UI



    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.2, 2.2.0
    • Component/s: Spark Core, Web UI, YARN
    • Labels:


      A Spark user may have to provide a sensitive information for a Spark configuration property, or a source out an environment variable in the executor or driver environment that contains sensitive information. A good example of this would be when reading/writing data from/to S3 using Spark. The S3 secret and S3 access key can be placed in a hadoop credential provider. However, one still needs to provide the password for the credential provider to Spark, which is typically supplied as an environment variable to the driver and executor environments. This environment variable shows up in logs, and may also show up in the UI.

      1. For logs, it shows up in a few places:
      1A. Event logs under SparkListenerEnvironmentUpdate event.
      1B. YARN logs, when printing the executor launch context.
      2. For UI, it would show up in the Environment tab, but it is redacted if it contains the words "password" or "secret" in it. And, these magic words are hardcoded and hence not customizable.

      This JIRA is to track the work to make sure sensitive information is redacted from all logs and UIs in Spark, while still being passed on to all relevant places it needs to get passed on to.


        1. redacted.png
          36 kB
          Mark Grover

          Issue Links



              • Assignee:
                mgrover Mark Grover
                mgrover Mark Grover
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created: