[HDDS-7501] Spark 3.2.2 and Ozone integration

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.2.1
• Labels: spark

Environment

• spark 3.2.2
• hadoop 3.1.2
• ozone 1.2.1

• Target Version/s: 1.2.1
• Flags: Important
• Assignee: Unassigned
• Reporter: Dmitry Kravchuk
• Created: 17/Nov/22 09:22
• Updated: 25/Apr/23 16:32
• Resolved: 25/Apr/23 16:32

Description

We've integrated Ozone with HDFS and everything works fine from the shell - it's possible to use hdfs dfs -ls command for ozone filies.

When we're trying to read files with spark it fails with strange error of class incomplitance.

org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message

Application log is attached.pyspark_ozone_1.txt

Can you please help?

Issue Links

• duplicates: HDDS-6926 Cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message