Pig
  1. Pig
  2. PIG-176

pig creates many small files when it spills

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, on spill pig can generate millions of small (under 128K) files. Partially this is due to PIG-170 but even with that patch, you can still try and spill small bags.

      The proposal is to not spill small files. Alan told me that the logic is already there but we just need to bump the size limit.

      1. pig176_v2.patch
        9 kB
        Pi Song
      2. pig_176_smallbags_v1.patch
        6 kB
        Pi Song

        Activity

        Olga Natkovich created issue -
        Pi Song made changes -
        Field Original Value New Value
        Attachment pig_176_smallbags_v1.patch [ 12379732 ]
        Pi Song made changes -
        Attachment pig176_v2.patch [ 12380505 ]
        Alan Gates made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.1.0 [ 12312848 ]
        Alan Gates made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Alan Gates
            Reporter:
            Olga Natkovich
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development