[CARBONDATA-2729] Schema Compatibility problem between version 1.3.0 and 1.4.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0, 1.4.1
• Component/s: file-format
• Assignee: Mohammad Shahid Khan
• Reporter: Mohammad Shahid Khan
• Created: 11/Jul/18 12:31
• Updated: 14/Jul/18 06:51
• Resolved: 14/Jul/18 06:51

Description

The table schema created with 1.3 is not compatible with 1.4.0.

Steps:

Do the below operation  on hive meta store.

1.  Create table and load data in 1.3 version.   

spark.sql("create database carbonversion_1_1");
spark.sql("use carbonversion_1_1");
spark.sql("""create table testIntType (c1 string,c2 int) STORED BY 'org.apache.carbondata.format'""")
spark.sql("insert into testIntType select 'a',200")

2. Switch to 1.4 version and create data map on the table created in step 1.

spark.sql("use carbonversion_1_3");
spark.sql("create datamap dm8 on table testinttype3 using 'preaggregate' as select c1, sum(c2) from testinttype3 group by c1")

 Stack trace:

Exception in thread "main" java.lang.NullPointerException
at org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalSchemaEvolution(ThriftWrapperSchemaConverterImpl.java:87)
at org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableSchema(ThriftWrapperSchemaConverterImpl.java:277)
at org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl.fromWrapperToExternalTableInfo(ThriftWrapperSchemaConverterImpl.java:310)
at org.apache.spark.sql.hive.CarbonHiveMetaStore.getThriftTableInfo(CarbonHiveMetaStore.scala:101)

Issue Links

• links to: GitHub Pull Request #2498