Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3505

Make AvroStorage sync interval take default from io.file.buffer.size

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      The default sync interval is 16K which is very bad for bzip compression which can take bigger chunk of data for compression. Hadoop's Bzip2code uses io.file.buffer.size as the buffer size. Most tuned environments have it set to 128K which gives better compression.

      1. PIG-3505-1.patch
        4 kB
        Rohini Palaniswamy

        Activity

        Hide
        cheolsoo Cheolsoo Park added a comment -

        +1.

        Show
        cheolsoo Cheolsoo Park added a comment - +1.
        Hide
        rohini Rohini Palaniswamy added a comment -

        Committed to trunk. Thanks for the review Cheolsoo.

        Show
        rohini Rohini Palaniswamy added a comment - Committed to trunk. Thanks for the review Cheolsoo.

          People

          • Assignee:
            rohini Rohini Palaniswamy
            Reporter:
            rohini Rohini Palaniswamy
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development