Details
-
Bug
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
None
-
None
Description
I encountered the following warnings and an error when using spark datasource in spark shell, the write is successful though:
scala> val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2)) warning: there was one deprecation warning (since 2.12.0) warning: there was one deprecation warning (since 2.2.0) warning: there were two deprecation warnings in total; for details, enable `:setting -deprecation' or `:replay -deprecation' df: org.apache.spark.sql.DataFrame = [begin_lat: double, begin_lon: double ... 8 more fields] scala> scala> df.write.format("hudi"). | option("hoodie.insert.shuffle.parallelism", "2"). | option("hoodie.upsert.shuffle.parallelism", "2"). | option("hoodie.bulkinsert.shuffle.parallelism", "2"). | option("hoodie.delete.shuffle.parallelism", "2"). | option(PRECOMBINE_FIELD_OPT_KEY, "ts"). | option(RECORDKEY_FIELD_OPT_KEY, "uuid"). | option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath"). | option(TABLE_NAME, tableName). | option("hoodie.parquet.small.file.limit", "0"). | option("hoodie.clustering.inline", "true"). | option("hoodie.clustering.inline.max.commits", "2"). | option("hoodie.clustering.plan.strategy.target.file.max.bytes", "1073741824"). | option("hoodie.clustering.plan.strategy.small.file.limit", "629145600"). | option("hoodie.clustering.plan.strategy.sort.columns", "rider,driver"). | option("hoodie.layout.optimize.enable", "true"). | mode(Append). | save(basePath) warning: there was one deprecation warning; for details, enable `:setting -deprecation' or `:replay -deprecation' 21/11/26 20:53:41 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf 21/11/26 20:53:41 ERROR DFSPropertiesConfiguration: Error reading in properties from dfs 21/11/26 20:53:41 WARN DFSPropertiesConfiguration: Didn't find config file under default conf file dir: file:/etc/hudi/conf
Attachments
Issue Links
- links to