[CARBONDATA-3815] Insert into table select from another table throws exception for spatial tables

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: core, spark-integration
• Assignee: Unassigned
• Reporter: Venugopal Reddy K
• Created: 11/May/20 08:52
• Updated: 15/May/20 17:47
• Resolved: 15/May/20 17:47

Description

Issue:

Insert into table select from another table throws exception for spatial tables. NoSuchElementException exception is thrown with 'mygeohash' column.

Exception in thread "main" java.util.NoSuchElementException: key not found: mygeohashException in thread "main" java.util.NoSuchElementException: key not found: mygeohash at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:59) at scala.collection.mutable.HashMap.apply(HashMap.scala:65) at org.apache.spark.sql.execution.command.management.CarbonInsertIntoCommand$$anonfun$getReArrangedIndexAndSelectedSchema$5.apply(CarbonInsertIntoCommand.scala:504) at org.apache.spark.sql.execution.command.management.CarbonInsertIntoCommand$$anonfun$getReArrangedIndexAndSelectedSchema$5.apply(CarbonInsertIntoCommand.scala:497) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.sql.execution.command.management.CarbonInsertIntoCommand.getReArrangedIndexAndSelectedSchema(CarbonInsertIntoCommand.scala:496) at org.apache.spark.sql.execution.command.management.CarbonInsertIntoCommand.processData(CarbonInsertIntoCommand.scala:164)

Step to reproduce:

1. Create source table and target table spatial tables.
2. Load data to source table.
3. Insert into target table select from source table.

TestCase:

spark.sql(s"""
CREATE TABLE source(
timevalue BIGINT,
longitude LONG,
latitude LONG) COMMENT "This is a GeoTable"
STORED AS carbondata
TBLPROPERTIES ('INDEX_HANDLER'='mygeohash',
'INDEX_HANDLER.mygeohash.type'='geohash',
'INDEX_HANDLER.mygeohash.sourcecolumns'='longitude, latitude',
'INDEX_HANDLER.mygeohash.originLatitude'='39.832277',
'INDEX_HANDLER.mygeohash.gridSize'='50',
'INDEX_HANDLER.mygeohash.minLongitude'='115.811865',
'INDEX_HANDLER.mygeohash.maxLongitude'='116.782233',
'INDEX_HANDLER.mygeohash.minLatitude'='39.832277',
'INDEX_HANDLER.mygeohash.maxLatitude'='40.225281',
'INDEX_HANDLER.mygeohash.conversionRatio'='1000000')
""".stripMargin)

 

val path = s"$rootPath/examples/spark/src/main/resources/geodata.csv"
spark.sql(s"""
LOAD DATA LOCAL INPATH '$path'
INTO TABLE source
OPTIONS('COMPLEX_DELIMITER_LEVEL_1'='#')
""".stripMargin)

 

spark.sql(s"""
CREATE TABLE target(
timevalue BIGINT,
longitude LONG,
latitude LONG) COMMENT "This is a GeoTable"
STORED AS carbondata
TBLPROPERTIES ('INDEX_HANDLER'='mygeohash',
'INDEX_HANDLER.mygeohash.type'='geohash',
'INDEX_HANDLER.mygeohash.sourcecolumns'='longitude, latitude',
'INDEX_HANDLER.mygeohash.originLatitude'='39.832277',
'INDEX_HANDLER.mygeohash.gridSize'='50',
'INDEX_HANDLER.mygeohash.minLongitude'='115.811865',
'INDEX_HANDLER.mygeohash.maxLongitude'='116.782233',
'INDEX_HANDLER.mygeohash.minLatitude'='39.832277',
'INDEX_HANDLER.mygeohash.maxLatitude'='40.225281',
'INDEX_HANDLER.mygeohash.conversionRatio'='1000000')
""".stripMargin)
 
 
spark.sql("insert into target select * from source")