[SPARK-35816] Spark read write with multiple Hadoop HA cluster limitation

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.3
• Component/s: Spark Submit
• Labels: hadoop-ha spark-sql
• Assignee: Unassigned
• Reporter: Anupam Jain
• Created: 18/Jun/21 16:45
• Updated: 12/Dec/22 18:11

Description

I have two Hadoop HA cluster: h1 and h2. Want to read from h1-HDFS and write to h2-HDFS using spark. As both HDFS are in HA, so need to set spark hadoop configuration with HDFS details

spark.sparkContext().hadoopConfiguration().set(<HADOOP_RPC_ADDRESS_AND_DETAILS>)

So with a single spark session job one of the Hadoop configuration will overwrite with write details and will try to read from that configuration, resulting in no file/path found.

Similar thing will happen with HDFS to external Hive write(I am writing in external Hive table owned HDFS), but more keen on above problem solution