[HOP-3717] Scala version issues while using spark runtime configuration

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Documentation
• Assignee: Hans Van Akelyen
• Reporter: Varun Kapoor
• Created: 31/Jan/22 10:36
• Updated: 31/May/22 11:11
• Resolved: 31/May/22 11:11

Description

Hop v1.1.0 comes with default scala libraries of version 2.11.x

I have spark cluster running over kubernetes (v3.1.2), which comes with scala library of 2.12.0

While running any pipeline with Scala provided with HOP results into the following error

Following the documentation where it says

If you get an error about a missing Scala library, you also need to include scala version 2.12 libraries. You can copy these from a Spark installation ${SPARK_HOME}/jars/scala--2.12.10.jar into plugins/engines/beam/lib.*
On copying scala 2.12.10 jars to the above path, we have a different exception

 

 

Also,I was not able to understand why hop has a different version of scala than spark when the compatibility matrix on documentation says Hop V1.1.0 is compatible with Spark 3.1.2