[SPARK-20692] unknowing delay in event timeline

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.6.2
• Component/s: DStreams

Environment

Spark 1.6.1 + kafka 0.8.2

• Assignee: Unassigned
• Reporter: Zhiwen Sun
• Created: 10/May/17 10:42
• Updated: 10/May/17 10:56
• Resolved: 10/May/17 10:56

Description

Spark streaming job with 1s interval.

Process time of micro batch suddenly became to 4s while is is usually 0.4s .

When we check where the time spent, we find a unknown delay in job.

There is no executor computing or shuffle reading. It is about 4s blank in event timeline,

event timeline snapshot is in attachment.