Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      The current AvroStorage implementation has a lot of issues: it requires old versions of Avro, it copies data much more than needed, and it's verbose and complicated. (One pet peeve of mine is that old versions of Avro don't support Snappy compression.)

      I rewrote AvroStorage from scratch to fix these issues. In early tests, the new implementation is significantly faster, and the code is a lot simpler. Rewriting AvroStorage also enabled me to implement support for Trevni (as TrevniStorage).

      I'm opening this ticket to facilitate discussion while I figure out the best way to contribute the changes back to Apache.

      1. with_dates.pig
        0.4 kB
        Joseph Adler
      2. TestInput.java
        1 kB
        Cheolsoo Park
      3. Test.java
        2 kB
        Cheolsoo Park
      4. PIG-3015-doc-2.patch
        13 kB
        Cheolsoo Park
      5. PIG-3015-doc.patch
        13 kB
        Cheolsoo Park
      6. PIG-3015-9.patch
        154 kB
        Joseph Adler
      7. PIG-3015-7.patch
        160 kB
        Cheolsoo Park
      8. PIG-3015-6.patch
        160 kB
        Joseph Adler
      9. PIG-3015-5.patch
        153 kB
        Joseph Adler
      10. PIG-3015-4.patch
        163 kB
        Cheolsoo Park
      11. PIG-3015-3.patch
        154 kB
        Cheolsoo Park
      12. PIG-3015-22June2013.diff
        182 kB
        Cheolsoo Park
      13. PIG-3015-20May2013.diff
        179 kB
        Joseph Adler
      14. PIG-3015-2.patch
        153 kB
        Cheolsoo Park
      15. PIG-3015-12.patch
        16 kB
        Joseph Adler
      16. PIG-3015-11.patch
        159 kB
        Cheolsoo Park
      17. PIG-3015-10.patch
        160 kB
        Cheolsoo Park
      18. good.avro
        2.86 MB
        Cheolsoo Park
      19. bad.avro
        2.86 MB
        Cheolsoo Park

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Joseph Adler
              Reporter:
              Joseph Adler
            • Votes:
              6 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development