[SPARK-19977] Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.1.0
• Component/s: Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: Ray Qiu
• Created: 16/Mar/17 13:05
• Updated: 17/May/20 17:47
• Resolved: 26/Mar/17 18:52

Description

Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30+ seconds in a Spark Streaming application, where multiple Kafka direct streams are processed. These kafka streams are processed separately (not combined via union).

It causes the task processing time to increase greatly and eventually stops working.