[DATAFU-167] Fix Scala Python Bridge support in Spark 2 minor version updates

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.1
• Fix Version/s: 1.8.0
• Labels: up-for-grabs

Environment

emphasized text

Language

python scala

• Assignee: Eyal Allweil
• Reporter: Eyal Allweil
• Created: 11/Oct/22 12:52
• Updated: 26/Jan/23 15:20
• Resolved: 26/Jan/23 15:20

Description

The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3, doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up).

This can be reproduced in testing by running the tests with the command

./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3 -PscalaCompatVersion=2.11

The error message is

 AttributeError: 'GatewayParameters' object has no attribute 'auth_token'

	at org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137)

 

Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter; it's possible/probable that using the auth_token parameter will both fix this problem and be better in general.

 

Please note that in order to test Spark 2.4.4, you need to upgrade the scalatest version used.

 

A description of using an auth token with py4j can be found here:

https://www.py4j.org/advanced_topics.html#authentication

 

It's likely that (at least some of the) changes will need to be made here:

https://github.com/apache/datafu/blob/master/datafu-spark/src/main/scala/spark/utils/overwrites/SparkPythonRunner.scala#L63
https://github.com/apache/datafu/blob/master/datafu-spark/src/main/resources/pyspark_utils/bridge_utils.py#L43

Issue Links

• blocks: DATAFU-168 Support Spark 2.4.6 and up - fix collectLimitedList compilation
• is required by: DATAFU-169 Support Spark 3.x