Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22479

SaveIntoDataSourceCommand logs jdbc credentials

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.1, 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      JDBC credentials are not redacted in plans including a 'SaveIntoDataSourceCommand'.

      Steps to reproduce:

      spark-shell --packages org.postgresql:postgresql:42.1.1
      
      import org.apache.spark.sql.execution.QueryExecution
      import org.apache.spark.sql.util.QueryExecutionListener
      val listener = new QueryExecutionListener {
        override def onFailure(funcName: String, qe: QueryExecution, exception: Exception): Unit = {}
        override def onSuccess(funcName: String, qe: QueryExecution, duration: Long): Unit = {
          System.out.println(qe.toString())
        }
      }
      spark.listenerManager.register(listener)
      spark.range(100).write.format("jdbc").option("url", "jdbc:postgresql:sparkdb").option("password", "pass").option("driver", "org.postgresql.Driver").option("dbtable", "test").save()
      

      The above will yield the following plan:

      == Parsed Logical Plan ==
      SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists
         +- Range (0, 100, step=1, splits=Some(8))
      
      == Analyzed Logical Plan ==
      SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists
         +- Range (0, 100, step=1, splits=Some(8))
      
      == Optimized Logical Plan ==
      SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists
         +- Range (0, 100, step=1, splits=Some(8))
      
      == Physical Plan ==
      ExecutedCommand
         +- SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists
               +- Range (0, 100, step=1, splits=Some(8))
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                onursatici Onur Satici
                Reporter:
                onursatici Onur Satici
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: