[BEAM-12762] java.io.InvalidClassException with Spark 3.1.2

• Type: Bug
• Status: Triage Needed
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-spark
• Assignee: Unassigned
• Reporter: Kyle Weaver
• Created: 16/Aug/21 20:54
• Updated: 22/Nov/24 07:50

Description

This was reported on the mailing list.
 

---

 
Using spark downloaded from below link,
 
https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
 
I get below error when submitting a pipeline. 
Full error is on https://gist.github.com/yuwtennis/7b0c1dc0dcf98297af1e3179852ca693.
 
------------------------------------------------------------------------------------------------------------------
21/08/16 01:10:26 WARN TransportChannelHandler: Exception in connection from /192.168.11.2:35601
java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef; local class incompatible: stream classdesc serialVersionUID = 3456489343829468865, local class serialVersionUID = 1028182004549731694
at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
...
------------------------------------------------------------------------------------------------------------------
 
SDK Harness and Job service are deployed as below.
 
1. SDK Harness
 
sudo docker run --net=host apache/beam_spark3_job_server:2.31.0 --spark-master-url=spark://localhost:7077 --clean-artifacts-per-job true
 
2. Job service
 
sudo docker run --net=host apache/beam_python3.8_sdk:2.31.0 --worker_pool
 

• apache/beam_spark_job_server:2.31.0 for spark 2.4.8
   
  3. SDK client code
   
  https://gist.github.com/yuwtennis/2e4c13c79f71e8f713e947955115b3e2
   Spark 2.4.8 succeeded without any errors using above components.
   
  https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz

 

Issue Links

• is cloned by: BEAM-13610 Conflict with Spark 3.1.2 scala version