[CARBONDATA-4273] Cannot create table with partitions in Spark in EMR

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: spark-integration
• Labels: EMR spark

Environment

Release label:emr-5.24.1
Hadoop distribution:Amazon 2.8.5
Applications:
Hive 2.3.4, Pig 0.17.0, Hue 4.4.0, Flink 1.8.0, Spark 2.4.2, Presto 0.219, JupyterHub 0.9.6

Jar complied with:
apache-carbondata:2.2.0
spark:2.4.5
hadoop:2.8.3

• Assignee: Indhumathi Muthumurugesh
• Reporter: Bigicecream
• Created: 22/Aug/21 11:30
• Updated: 01/Sep/21 09:37
• Resolved: 01/Sep/21 06:45

Description

 

When trying to create a table like this:

CREATE TABLE IF NOT EXISTS will_not_work(
timestamp string,
name string
)
PARTITIONED BY (dt string, hr string)
STORED AS carbondata
LOCATION 's3a://my-bucket/CarbonDataTests/will_not_work

The folder 's3a://my-bucket/CarbonDataTests/will_not_work' is a not existing folder
I get the following error:

org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException: Partition is not supported for external table
  at org.apache.spark.sql.parser.CarbonSparkSqlParserUtil$.buildTableInfoFromCatalogTable(CarbonSparkSqlParserUtil.scala:219)
  at org.apache.spark.sql.CarbonSource$.createTableInfo(CarbonSource.scala:235)
  at org.apache.spark.sql.CarbonSource$.createTableMeta(CarbonSource.scala:394)
  at org.apache.spark.sql.execution.command.table.CarbonCreateDataSourceTableCommand.processMetadata(CarbonCreateDataSourceTableCommand.scala:69)
  at org.apache.spark.sql.execution.command.MetadataCommand$$anonfun$run$1.apply(package.scala:137)
  at org.apache.spark.sql.execution.command.MetadataCommand$$anonfun$run$1.apply(package.scala:137)
  at org.apache.spark.sql.execution.command.Auditable$class.runWithAudit(package.scala:118)
  at org.apache.spark.sql.execution.command.MetadataCommand.runWithAudit(package.scala:134)
  at org.apache.spark.sql.execution.command.MetadataCommand.run(package.scala:137)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
  at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
  at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
  at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364)
  at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
  at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363)
  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
  ... 64 elided