Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-2195

Sync Hive Failed When Execute CTAS In Spark2 And Spark3

    XMLWordPrintableJSON

Details

    Description

      When execute CTAS in spark2, the follow exception will throw out:

      java.lang.NoClassDefFoundError: org/json/JSONException
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:10847)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10047)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10128)
      	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
      	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
      

      While executing CTAS in spark3, the follow exception throw out:

      java.lang.NoClassDefFoundError: org/apache/calcite/rel/type/RelDataTypeSystemjava.lang.NoClassDefFoundError: org/apache/calcite/rel/type/RelDataTypeSystem at org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get(SemanticAnalyzerFactory.java:318) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:484) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLs(HoodieHiveClient.java:458) at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLUsingHiveDriver(HoodieHiveClient.java:448) at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:426) at org.apache.hudi.hive.HoodieHiveClient.createTable(HoodieHiveClient.java:322) at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:234) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:179) at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:130)
      

      Attachments

        Issue Links

          Activity

            People

              pzw2018 pengzhiwei
              pzw2018 pengzhiwei
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: