Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3015

Rewrite of AvroStorage

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.12.0
    • None
    • None
    • Patch Available

    Description

      The current AvroStorage implementation has a lot of issues: it requires old versions of Avro, it copies data much more than needed, and it's verbose and complicated. (One pet peeve of mine is that old versions of Avro don't support Snappy compression.)

      I rewrote AvroStorage from scratch to fix these issues. In early tests, the new implementation is significantly faster, and the code is a lot simpler. Rewriting AvroStorage also enabled me to implement support for Trevni (as TrevniStorage).

      I'm opening this ticket to facilitate discussion while I figure out the best way to contribute the changes back to Apache.

      Attachments

        1. with_dates.pig
          0.4 kB
          Joseph Adler
        2. TestInput.java
          1 kB
          Cheolsoo Park
        3. Test.java
          2 kB
          Cheolsoo Park
        4. PIG-3015-doc-2.patch
          13 kB
          Cheolsoo Park
        5. PIG-3015-doc.patch
          13 kB
          Cheolsoo Park
        6. PIG-3015-9.patch
          154 kB
          Joseph Adler
        7. PIG-3015-7.patch
          160 kB
          Cheolsoo Park
        8. PIG-3015-6.patch
          160 kB
          Joseph Adler
        9. PIG-3015-5.patch
          153 kB
          Joseph Adler
        10. PIG-3015-4.patch
          163 kB
          Cheolsoo Park
        11. PIG-3015-3.patch
          154 kB
          Cheolsoo Park
        12. PIG-3015-22June2013.diff
          182 kB
          Cheolsoo Park
        13. PIG-3015-20May2013.diff
          179 kB
          Joseph Adler
        14. PIG-3015-2.patch
          153 kB
          Cheolsoo Park
        15. PIG-3015-12.patch
          16 kB
          Joseph Adler
        16. PIG-3015-11.patch
          159 kB
          Cheolsoo Park
        17. PIG-3015-10.patch
          160 kB
          Cheolsoo Park
        18. good.avro
          2.86 MB
          Cheolsoo Park
        19. bad.avro
          2.86 MB
          Cheolsoo Park

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. PIG-2266 bug with input file joining optimization in Pig

          • Major - Major loss of function.
          • Closed
          is related to

          New Feature - A new feature of the product, which has yet to be developed. PIG-3111 ToAvro to convert any Pig record to an Avro bytearray

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. AVRO-806 add a column-major file format

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. AVRO-1241 improve trevni performance on string deserialization

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-3585 Integrate Trevni as another columnar oriented file format

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Patch Available
          relates to

          Bug - A problem which impairs or prevents the functions of the product. AVRO-1235 Avro does not handle corrupt records

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. PIG-3059 Global configurable minimum 'bad record' thresholds

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. PIG-3111 ToAvro to convert any Pig record to an Avro bytearray

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. AVRO-1241 improve trevni performance on string deserialization

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Review board

          Activity

            People

              jadler Joseph Adler
              jadler Joseph Adler
              Votes:
              6 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: