Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6405 Support append feature for HCatalog
  3. HIVE-6475

Implement support for appending to mutable tables in HCatalog

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      Introduces append feature for HCatalog writes.

      Previously, if an unpartitioned table had data in it, or if a partition in a partitioned table had data in it, or if the partition even existed, HCat would fail if a user attempted to write to them. Now, that behaviour is extended so that the strict behaviour exists only if the table in question has a parameter "immutable" set to "true" (see HIVE-6406).

      With this patch, we can append to existing partitions or non-partitioned tables that already have data in them, as long as the new data being written is compatible to the old data (i.e. one cannot mix fileformats when attempting an append)

      As a further note, append is currently not compatible with dynamic partitioning, and a dynamic partitioning job is still unable to append to a table, even if it is a mutable table.
      Show
      Introduces append feature for HCatalog writes. Previously, if an unpartitioned table had data in it, or if a partition in a partitioned table had data in it, or if the partition even existed, HCat would fail if a user attempted to write to them. Now, that behaviour is extended so that the strict behaviour exists only if the table in question has a parameter "immutable" set to "true" (see HIVE-6406 ). With this patch, we can append to existing partitions or non-partitioned tables that already have data in them, as long as the new data being written is compatible to the old data (i.e. one cannot mix fileformats when attempting an append) As a further note, append is currently not compatible with dynamic partitioning, and a dynamic partitioning job is still unable to append to a table, even if it is a mutable table.

      Description

      Part of HIVE-6405, this is the implementation of the append feature on the HCatalog side. If a table is mutable, we must support being able to append to existing data instead of erroring out as a duplicate publish.

        Attachments

        1. 6475.log
          347 kB
          Sushanth Sowmyan
        2. 6475.log.hadoop2
          356 kB
          Sushanth Sowmyan
        3. HIVE-6475.2.patch
          50 kB
          Sushanth Sowmyan
        4. HIVE-6475.patch
          46 kB
          Sushanth Sowmyan

          Activity

            People

            • Assignee:
              sushanth Sushanth Sowmyan
              Reporter:
              sushanth Sushanth Sowmyan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: