[SPARK-33606] Delay in batch processing when creating Spark metadata compact file

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.3
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Stela Stefanova
• Created: 30/Nov/20 12:04
• Updated: 30/Nov/20 12:04

Description

Hi,

We are using Spark structured streaming to consume data from a Kafka cluster, transform it, and write it in an S3 Objectstore Service. Currently our Spark metadata compact files have 4,046,119 lines (1.2GB), and when a compact file gets written there is a delay with the batch processing - the batches are processing 1.8-2 minutes, when it usually takes 10-15 seconds. This delay in the processing generates lag in Kafka, which is a significant problem for us. Is this a known issue and if it is, are there any plans for fixing it in the future?