[SPARK-38915] Log executed SQL queries

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.2.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Gérald Quintana
• Created: 15/Apr/22 15:18
• Updated: 16/Apr/22 19:02

Description

When Spark issues an SQL query to the database through JDBC, log (at debug or info level) this query in order to be able to grab it. This is useful to check the performance of this query, running a SQL explain plan with the appropriate tool.
As SQL queries are sometimes generated, it is sometimes hard to predict what query Spark will sent to the database.

Example: I would like to watch this query https://github.com/apache/spark/blob/v3.2.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala#L352