[SPARK-6211] Test Python Kafka API using Python unit tests

• Type: Test
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: DStreams, Tests
• Target Version/s: 1.4.0
• Assignee: Saisai Shao
• Reporter: Tathagata Das
• Created: 06/Mar/15 23:42
• Updated: 10/Apr/15 06:15
• Resolved: 10/Apr/15 06:15

Description

This is tricky in python because the KafkaStreamSuiteBase (which has the functionality of creating embedded kafka clusters) is in the test package, which is not in the python path. To fix that, we have to ways.

1. Add test jar to classpath in python test. Thats kind of trickier.

2. Bring that into the src package (maybe renamed as KafkaTestUtils), and then wrap that in python to use it from python.

If (2) does not add any extra test dependencies to the main Kafka pom, then 2 should be simpler to do.

Issue Links

• blocks: SPARK-5155 Python API for MQTT streaming
• links to: [Github] Pull Request #4961 (jerryshao)