[HUDI-8559] Spark SQL commands fail on AWS EMR

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Labels: pull-request-available
• Assignee: Lin Liu
• Reporter: Y Ethan Guo

Reviewers

Y Ethan Guo

• Created: 20/Nov/24 18:41
• Updated: 24/Nov/24 01:53
• Resolved: 24/Nov/24 01:53

Description

With EMR Spark the below error is thrown for any SQL commands with master.

java.lang.IllegalStateException: Loop detected; file file:/etc/hudi/conf/hudi-defaults.conf already referenced
  at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:147)
  at org.apache.hudi.common.config.DFSPropertiesConfiguration.loadGlobalProps(DFSPropertiesConfiguration.java:127)
  at org.apache.hudi.common.config.DFSPropertiesConfiguration.<clinit>(DFSPropertiesConfiguration.java:71)
  at org.apache.spark.sql.hudi.ProvidesHoodieConfig$.isSchemaEvolutionEnabled(ProvidesHoodieConfig.scala:594)
  at org.apache.spark.sql.hudi.Spark34ResolveHudiAlterTableCommand.apply(Spark34ResolveHudiAlterTableCommand.scala:35)
  at org.apache.spark.sql.hudi.Spark34ResolveHudiAlterTableCommand.apply(Spark34ResolveHudiAlterTableCommand.scala:32)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:239)
  at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
  at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
  at scala.collection.immutable.List.foldLeft(List.scala:91)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeBatch$1(RuleExecutor.scala:236) 

Issue Links

• links to: GitHub Pull Request #12319