Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5011

Dynamic partitioning in HCatalog broken on external tables

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: HCatalog
    • Labels:
      None

      Description

      Dynamic partitioning with HCatalog has been broken as a result of HCATALOG-500 trying to support user-set paths for external tables.

      The goal there was to be able to support other custom destinations apart from the normal "hive-style" partitions. However, it is not currently possible for users to set paths for dynamic ptn writes, since we don't support any way for users to specify "patterns"(like, say "${rootdir}/$v1.$v2/") into which writes happen, only "locations", and the values for dyn. partitions are not known ahead of time. Also, specifying a custom path messes with the way dynamic ptn. code tries to determine what was written to where from the output committer, which means that even if we supported patterned-writes instead of location-writes, we still have to do some more deep diving into the output committer code to support it.

      Thus, my current proposal is that we honour writes to user-specified paths for external tables ONLY for static partition writes - i.e., if we can determine that the write is a dyn. ptn. write, we will ignore the user specification. (Note that this does not mean we ignore the table's external location - we honour that - we just don't honour any HCatStorer/etc provided additional location - we stick to what metadata tells us the root location is.

        Attachments

        1. HIVE-5011.patch
          8 kB
          Sushanth Sowmyan

          Issue Links

            Activity

              People

              • Assignee:
                sushanth Sushanth Sowmyan
                Reporter:
                sushanth Sushanth Sowmyan
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: