[SPARK-24485] Measure and log elapsed time for filesystem operations in HDFSBackedStateStoreProvider

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 07/Jun/18 13:05
• Updated: 12/Dec/22 18:11
• Resolved: 13/Jun/18 04:36

Description

There're couple of operations which communicate with file system (mostly remote HDFS in production) in HDFSBackedStateStoreProvider, which contribute huge part of latency.

It would be better to measure the latency (elapsed time) and log to help debugging when there's unexpected huge latency on state store.

Issue Links

• links to: [Github] Pull Request #21506 (HeartSaVioR)
• links to: GitHub Pull Request #21506