Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      The current AvroStorage implementation has a lot of issues: it requires old versions of Avro, it copies data much more than needed, and it's verbose and complicated. (One pet peeve of mine is that old versions of Avro don't support Snappy compression.)

      I rewrote AvroStorage from scratch to fix these issues. In early tests, the new implementation is significantly faster, and the code is a lot simpler. Rewriting AvroStorage also enabled me to implement support for Trevni (as TrevniStorage).

      I'm opening this ticket to facilitate discussion while I figure out the best way to contribute the changes back to Apache.

        Attachments

        1. PIG-3015-22June2013.diff
          182 kB
          Cheolsoo Park
        2. PIG-3015-20May2013.diff
          179 kB
          Joseph Adler
        3. PIG-3015-12.patch
          16 kB
          Joseph Adler
        4. PIG-3015-doc-2.patch
          13 kB
          Cheolsoo Park
        5. PIG-3015-11.patch
          159 kB
          Cheolsoo Park
        6. with_dates.pig
          0.4 kB
          Joseph Adler
        7. PIG-3015-10.patch
          160 kB
          Cheolsoo Park
        8. PIG-3015-9.patch
          154 kB
          Joseph Adler
        9. PIG-3015-doc.patch
          13 kB
          Cheolsoo Park
        10. PIG-3015-7.patch
          160 kB
          Cheolsoo Park
        11. PIG-3015-6.patch
          160 kB
          Joseph Adler
        12. good.avro
          2.86 MB
          Cheolsoo Park
        13. bad.avro
          2.86 MB
          Cheolsoo Park
        14. Test.java
          2 kB
          Cheolsoo Park
        15. TestInput.java
          1 kB
          Cheolsoo Park
        16. PIG-3015-5.patch
          153 kB
          Joseph Adler
        17. PIG-3015-4.patch
          163 kB
          Cheolsoo Park
        18. PIG-3015-3.patch
          154 kB
          Cheolsoo Park
        19. PIG-3015-2.patch
          153 kB
          Cheolsoo Park

          Issue Links

            Activity

              People

              • Assignee:
                jadler Joseph Adler
                Reporter:
                jadler Joseph Adler
              • Votes:
                6 Vote for this issue
                Watchers:
                31 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: