Description
When including maven coordinates to download dependencies in pyspark, pyspark returns a GatewayError, because it cannot read the proper port to communicate with the JVM. This is because pyspark relies on STDIN to read the port number and in the meantime Ivy prints out a whole lot of logs.
Attachments
Issue Links
- is related to
-
SPARK-2313 PySpark should accept port via a command line argument rather than STDIN
- Resolved
I think that this should be fixed now that my patch for
SPARK-2313has been merged. brkyvz, do you think we should add a regression test for this bug? Do you have tests for Maven coordinate inclusion?