[SPARK-30804] Measure and log elapsed time for "compact" operation in CompactibleFileStreamLog

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 13/Feb/20 04:39
• Updated: 12/Dec/22 18:10
• Resolved: 24/Apr/20 03:35

Description

"compact" operation in FileStreamSourceLog and FileStreamSinkLog is introduced to solve "small files" problem, but introduced non-trivial latency which is another headache in long run query.

There're bunch of reports from community for the same issue (see SPARK-24295, SPARK-29995, SPARK-30462) - before trying to solve the problem, it would be better to measure the latency (elapsed time) and log to help indicating the issue when the additional latency becomes concerns.

Issue Links

• links to: GitHub Pull Request #27557