Hive
  1. Hive
  2. HIVE-2037

Merge result file size should honor hive.merge.size.per.task

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      The merge job set mapred.min.split.size to the value of hive.merge.size.per.task, which roughly equals to the output file size. However the input split size is also determined by mapred.min.split.size.per.node, mapred.min.split.size.per.rack, and mapred.max.split.size. They should be set the same as hive.merge.size.per.task as well.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1m 29s 1 Ning Zhang 09/Mar/11 07:13
        Patch Available Patch Available Resolved Resolved
        15h 57m 1 Joydeep Sen Sarma 09/Mar/11 23:10
        Resolved Resolved Closed Closed
        282d 45m 1 Carl Steinbach 16/Dec/11 23:56
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Joydeep Sen Sarma made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.8.0 [ 12316178 ]
        Resolution Fixed [ 1 ]
        Hide
        Joydeep Sen Sarma added a comment -

        committed. thanks Ning.

        Show
        Joydeep Sen Sarma added a comment - committed. thanks Ning.
        Hide
        Ning Zhang added a comment -

        @joy, the unit tests are clean.

        Show
        Ning Zhang added a comment - @joy, the unit tests are clean.
        Hide
        Joydeep Sen Sarma added a comment -

        looks ok - please run the tests and i will commit.

        Show
        Joydeep Sen Sarma added a comment - looks ok - please run the tests and i will commit.
        Ning Zhang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ning Zhang made changes -
        Field Original Value New Value
        Attachment HIVE-2037.patch [ 12473106 ]
        Ning Zhang created issue -

          People

          • Assignee:
            Ning Zhang
            Reporter:
            Ning Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development