Hive
  1. Hive
  2. HIVE-1767

Merge files does not work with dynamic partition

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. HIVE-1767.1.patch
      68 kB
      He Yongqiang

      Activity

      Hide
      Ning Zhang added a comment -

      Can you elaborate the bug in the description?

      Show
      Ning Zhang added a comment - Can you elaborate the bug in the description?
      Hide
      He Yongqiang added a comment -

      When there is a dynamic partition, the merge task never gets launched. I was trying to figure out what's happening there.

      It turns out a bug in ConditionalResolverMergeFiles that leads to the size information counted is not correct for dp queries.

      Attached a patch for it.

      Show
      He Yongqiang added a comment - When there is a dynamic partition, the merge task never gets launched. I was trying to figure out what's happening there. It turns out a bug in ConditionalResolverMergeFiles that leads to the size information counted is not correct for dp queries. Attached a patch for it.
      Hide
      Namit Jain added a comment -

      load data local inpath '/Users/heyongqiang/Documents/workspace/Hive-Index/data/files/srcbucket20.txt' INTO TABLE srcpart_merge_dp partition(ds='2008-04-08', hr=11);
      load data local inpath '/Users/heyongqiang/Documents/workspace/Hive-Index/data/files/srcbucket21.txt' INTO TABLE srcpart_merge_dp partition(ds='2008-04-08', hr=11);
      load data local inpath '/Users/heyongqiang/Documents/workspace/Hive-Index/data/files/srcbucket22.txt' INTO TABLE srcpart_merge_dp partition(ds='2008-04-08', hr=11);
      load data local inpath '/Users/heyongqiang/Documents/workspace/Hive-Index/data/files/srcbucket23.txt' INTO TABLE srcpart_merge_dp partition(ds='2008-04-08', hr=11);

      The test needs to be updated

      Show
      Namit Jain added a comment - load data local inpath '/Users/heyongqiang/Documents/workspace/Hive-Index/data/files/srcbucket20.txt' INTO TABLE srcpart_merge_dp partition(ds='2008-04-08', hr=11); load data local inpath '/Users/heyongqiang/Documents/workspace/Hive-Index/data/files/srcbucket21.txt' INTO TABLE srcpart_merge_dp partition(ds='2008-04-08', hr=11); load data local inpath '/Users/heyongqiang/Documents/workspace/Hive-Index/data/files/srcbucket22.txt' INTO TABLE srcpart_merge_dp partition(ds='2008-04-08', hr=11); load data local inpath '/Users/heyongqiang/Documents/workspace/Hive-Index/data/files/srcbucket23.txt' INTO TABLE srcpart_merge_dp partition(ds='2008-04-08', hr=11); The test needs to be updated
      Hide
      Namit Jain added a comment -

      +1

      running tests

      Show
      Namit Jain added a comment - +1 running tests
      Hide
      Namit Jain added a comment -

      Committed. Thanks Yongqiang

      Show
      Namit Jain added a comment - Committed. Thanks Yongqiang

        People

        • Assignee:
          He Yongqiang
          Reporter:
          He Yongqiang
        • Votes:
          0 Vote for this issue
          Watchers:
          0 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development