Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-794

Use Avro serialization in Pig

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.2.0
    • None
    • impl
    • None

    Description

      We would like to use Avro serialization in Pig to pass data between MR jobs instead of the current BinStorage. Attached is an implementation of AvroBinStorage which performs significantly better compared to BinStorage on our benchmarks.

      Attachments

        1. jackson-asl-0.9.4.jar
          148 kB
          Rakesh Setty
        2. AvroStorage.patch
          28 kB
          Rakesh Setty
        3. PIG-794.patch
          29 kB
          Giridharan Kesavan
        4. avro-0.1-dev-java_r765402.jar
          101 kB
          Rakesh Setty
        5. AvroStorage_2.patch
          54 kB
          Jeff Zhang
        6. AvroTest.java
          2 kB
          Jeff Zhang
        7. AvroStorage_3.patch
          54 kB
          Jeff Zhang
        8. AvroStorage_4.patch
          37 kB
          Jeff Zhang

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dvryaboy Dmitriy V. Ryaboy
            serakesh Rakesh Setty
            Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment