Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21687

Spark SQL should set createTime for Hive partition

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0, 2.2.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      In Spark SQL, we often use `insert overwite table t partition(p=xx)` to create partition for partitioned table. `createTime` is an important information to manage data lifecycle, e.g TTL.
      However, we found that Spark SQL doesn't call setCreateTime in `HiveClientImpl#toHivePartition` as follows:

      Unable to find source-code formatter for language: scala. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
      def toHivePartition(
            p: CatalogTablePartition,
            ht: HiveTable): HivePartition = {
          val tpart = new org.apache.hadoop.hive.metastore.api.Partition
          val partValues = ht.getPartCols.asScala.map { hc =>
            p.spec.get(hc.getName).getOrElse {
              throw new IllegalArgumentException(
                s"Partition spec is missing a value for column '${hc.getName}': ${p.spec}")
            }
          }
          val storageDesc = new StorageDescriptor
          val serdeInfo = new SerDeInfo
          p.storage.locationUri.map(CatalogUtils.URIToString(_)).foreach(storageDesc.setLocation)
          p.storage.inputFormat.foreach(storageDesc.setInputFormat)
          p.storage.outputFormat.foreach(storageDesc.setOutputFormat)
          p.storage.serde.foreach(serdeInfo.setSerializationLib)
          serdeInfo.setParameters(p.storage.properties.asJava)
          storageDesc.setSerdeInfo(serdeInfo)
          tpart.setDbName(ht.getDbName)
          tpart.setTableName(ht.getTableName)
          tpart.setValues(partValues.asJava)
          tpart.setSd(storageDesc)
          new HivePartition(ht, tpart)
        }
      

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            debugger87 Chaozhong Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development