[HUDI-8559] Spark SQL commands fail on AWS EMR

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 1.0.0
• Assignee: Lin Liu
• Reporter: Y Ethan Guo
• Created: 2 days ago 18:41
• Updated: 6 hours ago

Description

With EMR Spark the below error is thrown for any SQL commands with master.

java.lang.IllegalStateException: Loop detected; file file:/etc/hudi/conf/hudi-defaults.conf already referenced
  at org.apache.hudi.common.config.DFSPropertiesConfiguration.addPropsFromFile(DFSPropertiesConfiguration.java:147)
  at org.apache.hudi.common.config.DFSPropertiesConfiguration.loadGlobalProps(DFSPropertiesConfiguration.java:127)
  at org.apache.hudi.common.config.DFSPropertiesConfiguration.<clinit>(DFSPropertiesConfiguration.java:71)
  at org.apache.spark.sql.hudi.ProvidesHoodieConfig$.isSchemaEvolutionEnabled(ProvidesHoodieConfig.scala:594)
  at org.apache.spark.sql.hudi.Spark34ResolveHudiAlterTableCommand.apply(Spark34ResolveHudiAlterTableCommand.scala:35)
  at org.apache.spark.sql.hudi.Spark34ResolveHudiAlterTableCommand.apply(Spark34ResolveHudiAlterTableCommand.scala:32)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:239)
  at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
  at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
  at scala.collection.immutable.List.foldLeft(List.scala:91)
  at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeBatch$1(RuleExecutor.scala:236)