[SPARK-44683] Logging level isn't passed to RocksDB state store provider correctly

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: Structured Streaming
• Assignee: Siying Dong
• Reporter: Siying Dong
• Created: 04/Aug/23 23:57
• Updated: 08/Aug/23 02:16
• Resolved: 08/Aug/23 02:12

Description

We pass log4j's log level to RocksDB so that RocksDB debug log can go to log4j. However, we pass in log level after we create the logger. However, the way it is set isn't effective. This has two impacts: (1) setting DEBUG level don't make RocksDB generate DEBUG level logs; (2) setting WARN or ERROR level does prevent INFO level logging, but RocksDB still makes JNI calls to Scala, which is an unnecessary overhead.