Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3815

Insert into table select from another table throws exception for spatial tables

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: core, spark-integration
    • Labels:
      None

      Description

      Issue:

      Insert into table select from another table throws exception for spatial tables. NoSuchElementException exception is thrown with 'mygeohash' column.

      Exception in thread "main" java.util.NoSuchElementException: key not found: mygeohashException in thread "main" java.util.NoSuchElementException: key not found: mygeohash at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:59) at scala.collection.mutable.HashMap.apply(HashMap.scala:65) at org.apache.spark.sql.execution.command.management.CarbonInsertIntoCommand$$anonfun$getReArrangedIndexAndSelectedSchema$5.apply(CarbonInsertIntoCommand.scala:504) at org.apache.spark.sql.execution.command.management.CarbonInsertIntoCommand$$anonfun$getReArrangedIndexAndSelectedSchema$5.apply(CarbonInsertIntoCommand.scala:497) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.sql.execution.command.management.CarbonInsertIntoCommand.getReArrangedIndexAndSelectedSchema(CarbonInsertIntoCommand.scala:496) at org.apache.spark.sql.execution.command.management.CarbonInsertIntoCommand.processData(CarbonInsertIntoCommand.scala:164)

      Step to reproduce:

      1. Create source table and target table spatial tables.
      2. Load data to source table.
      3. Insert into target table select from source table.

      TestCase:

      spark.sql(s"""
      CREATE TABLE source(
      timevalue BIGINT,
      longitude LONG,
      latitude LONG) COMMENT "This is a GeoTable"
      STORED AS carbondata
      TBLPROPERTIES ('INDEX_HANDLER'='mygeohash',
      'INDEX_HANDLER.mygeohash.type'='geohash',
      'INDEX_HANDLER.mygeohash.sourcecolumns'='longitude, latitude',
      'INDEX_HANDLER.mygeohash.originLatitude'='39.832277',
      'INDEX_HANDLER.mygeohash.gridSize'='50',
      'INDEX_HANDLER.mygeohash.minLongitude'='115.811865',
      'INDEX_HANDLER.mygeohash.maxLongitude'='116.782233',
      'INDEX_HANDLER.mygeohash.minLatitude'='39.832277',
      'INDEX_HANDLER.mygeohash.maxLatitude'='40.225281',
      'INDEX_HANDLER.mygeohash.conversionRatio'='1000000')
      """.stripMargin)

       

      val path = s"$rootPath/examples/spark/src/main/resources/geodata.csv"
      spark.sql(s"""
      LOAD DATA LOCAL INPATH '$path'
      INTO TABLE source
      OPTIONS('COMPLEX_DELIMITER_LEVEL_1'='#')
      """.stripMargin)

       

      spark.sql(s"""
      CREATE TABLE target(
      timevalue BIGINT,
      longitude LONG,
      latitude LONG) COMMENT "This is a GeoTable"
      STORED AS carbondata
      TBLPROPERTIES ('INDEX_HANDLER'='mygeohash',
      'INDEX_HANDLER.mygeohash.type'='geohash',
      'INDEX_HANDLER.mygeohash.sourcecolumns'='longitude, latitude',
      'INDEX_HANDLER.mygeohash.originLatitude'='39.832277',
      'INDEX_HANDLER.mygeohash.gridSize'='50',
      'INDEX_HANDLER.mygeohash.minLongitude'='115.811865',
      'INDEX_HANDLER.mygeohash.maxLongitude'='116.782233',
      'INDEX_HANDLER.mygeohash.minLatitude'='39.832277',
      'INDEX_HANDLER.mygeohash.maxLatitude'='40.225281',
      'INDEX_HANDLER.mygeohash.conversionRatio'='1000000')
      """.stripMargin)
       
       
      spark.sql("insert into target select * from source")

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              VenuReddy Venugopal Reddy K
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 40m
                1h 40m