Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1307

More generic and efficient merge method

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently if hive.merge.mapfiles/mapredfiles=true, a new mapreduce job is create to read the input files and output to one reducer for merging. This MR job is created at compile time and one MR job for one partition. In the case of dynamic partition case, multiple partitions could be created at execution time and generating merging MR job at compile time is impossible.

      We should generalize the merge framework to allow multiple partitions and most of the time a map-only job should be sufficient if we use CombineHiveInputFormat.

        Attachments

        1. HIVE-1307_2_branch_0.6.patch
          5.03 MB
          Ning Zhang
        2. HIVE-1307_branch_0.6.patch
          4.84 MB
          Ning Zhang
        3. HIVE-1307_java_only.patch
          33 kB
          Ning Zhang
        4. HIVE-1307.0.patch
          12 kB
          Ning Zhang
        5. HIVE-1307.2.patch
          1.75 MB
          Ning Zhang
        6. HIVE-1307.3_java.patch
          37 kB
          Ning Zhang
        7. HIVE-1307.3.patch
          1.83 MB
          Ning Zhang
        8. HIVE-1307.4.patch
          2.32 MB
          Ning Zhang
        9. HIVE-1307.5.patch
          2.32 MB
          Ning Zhang
        10. HIVE-1307.6.patch
          4.98 MB
          Ning Zhang
        11. HIVE-1307.7.patch
          4.98 MB
          Ning Zhang
        12. HIVE-1307.8.patch
          4.98 MB
          Ning Zhang
        13. HIVE-1307.9.patch
          4.98 MB
          Ning Zhang
        14. HIVE-1307.patch
          343 kB
          Ning Zhang

          Activity

            People

            • Assignee:
              nzhang Ning Zhang
              Reporter:
              nzhang Ning Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: