[SPARK-8080] Custom Receiver.store with Iterator type do not give correct count at Spark UI

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.2, 1.3.1
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: DStreams, Web UI
• Assignee: Dibyendu Bhattacharya
• Reporter: Dibyendu Bhattacharya
• Created: 03/Jun/15 14:51
• Updated: 19/Jun/15 03:09
• Resolved: 19/Jun/15 03:01

Description

In Custom receiver if I call store with Iterator type (store(dataIterator: Iterator[T]): Unit ) , Spark UI does not show the correct count of records in block which leads to wrong value for Input Rate, Scheduling Delay and Input SIze.

Issue Links

• is related to: SPARK-8112 Received block event count through the StreamingListener can be negative
• links to: [Github] Pull Request #6614 (dibbhatt)
• links to: [Github] Pull Request #6707 (dibbhatt)