[SPARK-16950] fromOffsets parameter in Kafka's Direct Streams does not work in python3

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.0.1, 2.1.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: PySpark
• Assignee: Mariusz Strzelecki
• Reporter: Mariusz Strzelecki
• Created: 08/Aug/16 12:44
• Updated: 30/Jan/17 12:34
• Resolved: 09/Aug/16 16:45

Description

KafkaUtils.createDirectStream does not work in python3 when you set parameter fromOffsets (which is starting offsets of the stream on Kafka). This is because the long type is removed from python3 and py4j maps numeric variables to java.lang.Integer or java.lang.Long depending on number size, which causes ClassCastException for small offsets variables.

This behaviour was noticed before and tests for this functionality are disabled in python3: https://github.com/apache/spark/blob/89e67d6667d5f8be9c6fb6c120fbcd350ae2950d/python/pyspark/streaming/tests.py#L1061

Issue Links

• is duplicated by: SPARK-17411 Cannot set fromOffsets in createDirectStream function
• is duplicated by: SPARK-15559 TopicAndPartition should provide __hash__ method
• links to: [Github] Pull Request #14540 (szczeles)