[SPARK-32988] ExternalCatalog vs ExternalCatalogWithListener: backward compatibility problem

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 2.3.2, 2.4.0
• Component/s: Java API
• Assignee: Unassigned
• Reporter: dinesh sachdev
• Created: 24/Sep/20 18:36
• Updated: 12/Dec/22 18:11
• Resolved: 27/Sep/20 09:25

Description

With Spark 2.3 

spark.sharedState.externalCatalog used to give an object of ExternalCatalog which was actually an instance of HiveExternalCatalog

With Spark 2.4 

spark.sharedState.externalCatalog gives an object of ExternalCatalogWithListener which has HiveExternalCatalog as unwrapped delegate.

 

We use HiveExternalCatalog to get HiveClient to perform Hive Operations.

 

 

This change in method Signature is giving us backward compatibility problems such that code compiled with Spark 2.3 can not work with 2.4 due to the change. This results in Runtime exceptions including ClassCastException, NoSuchMethodError