[BEAM-13610] Conflict with Spark 3.1.2 scala version

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Duplicate
• Affects Version/s: 2.34.0, 2.35.0
• Fix Version/s: Not applicable
• Component/s: runner-spark
• Assignee: Unassigned
• Reporter: Ronan SY
• Created: 07/Jan/22 14:53
• Updated: 25/Mar/22 17:30
• Resolved: 25/Mar/22 17:30

Description

When trying to use Beam with spark 3.1.2 we are running into this issue :

InvalidClassException: scala.collection.mutable.WrappedArray 

As explained here : https://www.mail-archive.com/[e-mail]/msg297820.html

It's an incompatibility issue :  spark 3.1.2 is compiled with scala 2.12.10 but this issue is fixed only for scala >= 2.12.14.

 

It seems that this issue can be fixed :

=> either by compiling spark & beam with a scala version >= 2.12.14

=> either by upgrading the spark version used by beam to 3.2.0, which can be compiled with scala 2.13

 

Do you know if the switch to spark 3.2.0 is planned ?

Issue Links

• Dependent: BEAM-7093 Support Spark 3 in Spark runner
• is a clone of: BEAM-12762 java.io.InvalidClassException with Spark 3.1.2