[SPARK-26349] Pyspark should not accept insecure p4yj gateways

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: PySpark
• Target Version/s: 3.0.0
• Assignee: Imran Rashid
• Reporter: Imran Rashid
• Created: 12/Dec/18 17:26
• Updated: 08/Jan/19 19:31
• Resolved: 08/Jan/19 19:27

Description

Pyspark normally produces a secure py4j connection between python & the jvm, but it also lets users provide their own connection. Spark should fail fast if that connection is insecure.

Note this is breaking a public api, which is why this is targeted at 3.0.0. SPARK-26019 is about adding a warning, but still allowing the old behavior to work, in 2.x

Issue Links

• is related to: SPARK-26019 pyspark/accumulators.py: "TypeError: object of type 'NoneType' has no len()" in authenticate_and_accum_updates()
• links to: GitHub Pull Request #23441