Uploaded image for project: 'HCatalog'
  1. HCatalog
  2. HCATALOG-232

Dynamic Partitioning broken: keys set to HIVE_DEFAULT_PARTITION

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.3
    • 0.3
    • mapreduce
    • None

    Description

      Looks like dynamic-partitioning is broken with 0.3. This is a regression from 0.2.

      Consider 2 tables (say source_search_Uberwald and target_search_Uberwald) with identical schemas and the following 4 partition keys:

      datestamp string None
      srcid string None
      action string None
      testid string None

      Consider the following Pig script (run on 0.9.2)

      a = load 'source_search_Uberwald' using org.apache.hcatalog.pig.HCatLoader();
      b = filter a by ( datestamp == '20091102' );
      store b into 'target_search_Uberwald' using org.apache.hcatalog.pig.HCatStorer('srcid=191740');

      One would expect that the target table would now have partitions corresponding to each partition in the source table (where the srcid is as specified).

      What one sees, however, is one of two symptoms:
      1. In addition to expected partitions on target, there's at least one more partition, with all dynamic-part-vals set to "HIVE_DEFAULT_PARTITION", or
      2. No new partitions on target table. However, the target table's directory has subdirectories named "action=_HIVE_DEFAULT_PARTITION_", etc.

      Attachments

        Activity

          People

            mithun Mithun Radhakrishnan
            mithun Mithun Radhakrishnan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: