Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14927

DataFrame. saveAsTable creates RDD partitions but not Hive partitions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 1.5.2, 1.6.1
    • None
    • SQL
    • Mac OS X 10.11.4 local

    Description

      This is a followup to http://stackoverflow.com/questions/31341498/save-spark-dataframe-as-dynamic-partitioned-table-in-hive . I tried to use suggestions in the answers but couldn't make it to work in Spark 1.6.1

      I am trying to create partitions programmatically from `DataFrame. Here is the relevant code (adapted from a Spark test):

      hc.setConf("hive.metastore.warehouse.dir", "tmp/tests")
      // hc.setConf("hive.exec.dynamic.partition", "true")
      // hc.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
      hc.sql("create database if not exists tmp")
      hc.sql("drop table if exists tmp.partitiontest1")

      Seq(2012 -> "a").toDF("year", "val")
      .write
      .partitionBy("year")
      .mode(SaveMode.Append)
      .saveAsTable("tmp.partitiontest1")
      hc.sql("show partitions tmp.partitiontest1").show

      Full file is here: https://gist.github.com/SashaOv/7c65f03a51c7e8f9c9e018cd42aa4c4a

      I get the error that the table is not partitioned:

      ======================
      HIVE FAILURE OUTPUT
      ======================
      SET hive.support.sql11.reserved.keywords=false
      SET hive.metastore.warehouse.dir=tmp/tests
      OK
      OK
      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Table tmp.partitiontest1 is not a partitioned table
      ======================

      It looks like the root cause is that `org.apache.spark.sql.hive.HiveMetastoreCatalog.newSparkSQLSpecificMetastoreTable` always creates table with empty partitions.

      Any help to move this forward is appreciated.

      Attachments

        Activity

          People

            Unassigned Unassigned
            SashaO Sasha Ovsankin
            Votes:
            4 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: