[SPARK-9059] Update Python Direct Kafka Word count examples to show the use of HasOffsetRanges

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Component/s: DStreams
• Labels: starter
• Assignee: Unassigned
• Reporter: Tathagata Das
• Created: 15/Jul/15 03:57
• Updated: 10/Dec/15 06:49
• Resolved: 09/Dec/15 16:23

Description

Update Python examples of Direct Kafka word count to access the offset ranges using HasOffsetRanges and print it. For example in Scala,

var offsetRanges: Array[OffsetRange] = _
...
directKafkaDStream.foreachRDD { rdd => 
    offsetRanges = rdd.asInstanceOf[HasOffsetRanges]  
}
...
transformedDStream.foreachRDD { rdd => 

    // some operation
    println("Processed ranges: " + offsetRanges)
}

See https://spark.apache.org/docs/latest/streaming-kafka-integration.html for more info, and the master source code for more updated information on python.

Issue Links

• links to: [Github] Pull Request #7467 (kico445)