Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-788

add Snappy compression

        Details

        • Type: New Feature
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 1.5.1
        • Component/s: java, spec
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          Google has released it's Snappy compression codec.

          https://code.google.com/p/snappy/

          Avro should support this for data file compression.

          1. Text File
            AVRO-788.patch
            10 kB
            Doug Cutting
          2. Text File
            AVRO-788.patch
            6 kB
            Doug Cutting

            Issue Links

            is related to

            Improvement - An improvement or enhancement to an existing feature or task. AVRO-798 add checksums to Snappy codec

            • Major - Major loss of function.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              qwertymaniac Harsh J added a comment -

              There is also an acknowledged issue ticket for adding Java support. Link for reference: https://code.google.com/p/snappy/issues/detail?id=15

              Show
              qwertymaniac Harsh J added a comment - There is also an acknowledged issue ticket for adding Java support. Link for reference: https://code.google.com/p/snappy/issues/detail?id=15
              Hide
              Permalink
              cutting Doug Cutting added a comment -

              Here's a patch that implements this.

              Show
              cutting Doug Cutting added a comment - Here's a patch that implements this.
              Hide
              Permalink
              cutting Doug Cutting added a comment -

              Here's a new version of the patch that adds support for Snappy to Avro's MapReduce API and also updates the spec, adding Snappy as an optional codec for now.

              Show
              cutting Doug Cutting added a comment - Here's a new version of the patch that adds support for Snappy to Avro's MapReduce API and also updates the spec, adding Snappy as an optional codec for now.
              Hide
              Permalink
              cutting Doug Cutting added a comment -

              Any reason not to commit this soon?

              Show
              cutting Doug Cutting added a comment - Any reason not to commit this soon?
              Hide
              Permalink
              scott_carey Scott Carey added a comment -

              +1 Straightforward, looks good.

              I wonder about the maven scope. Should this be 'provided' scope due to its use of JNI? See http://code.google.com/p/snappy-java/ towards the bottom. A maven user can override the current scope as needed or exclude the jar, so this isn't critical and can be changed easily later.

              Show
              scott_carey Scott Carey added a comment - +1 Straightforward, looks good. I wonder about the maven scope. Should this be 'provided' scope due to its use of JNI? See http://code.google.com/p/snappy-java/ towards the bottom. A maven user can override the current scope as needed or exclude the jar, so this isn't critical and can be changed easily later.
              Hide
              Permalink
              cutting Doug Cutting added a comment -

              I've thus far avoided learning too much about Maven, so don't have an opinion.

              Show
              cutting Doug Cutting added a comment - I've thus far avoided learning too much about Maven, so don't have an opinion.
              Hide
              Permalink
              scott_carey Scott Carey added a comment -

              Hold off on my +1. I have one other concern. What JNI library is embedded here? Linux? Windows? Do users have to compile their own?

              http://code.google.com/p/snappy-java/source/browse/INSTALL

              I'm going to check this more thoroughly on my mac.

              Show
              scott_carey Scott Carey added a comment - Hold off on my +1. I have one other concern. What JNI library is embedded here? Linux? Windows? Do users have to compile their own? http://code.google.com/p/snappy-java/source/browse/INSTALL I'm going to check this more thoroughly on my mac.
              Hide
              Permalink
              scott_carey Scott Carey added a comment -

              +1 OK, tests pass. They embed compiled libraries for Mac, Windows, and Linux. So this feature is available only for those platforms at this time.

              Show
              scott_carey Scott Carey added a comment - +1 OK, tests pass. They embed compiled libraries for Mac, Windows, and Linux. So this feature is available only for those platforms at this time.
              Hide
              Permalink
              cutting Doug Cutting added a comment -

              I committed this.

              Show
              cutting Doug Cutting added a comment - I committed this.

                People

                • Assignee:
                  cutting Doug Cutting
                  Reporter:
                  cutting Doug Cutting
                • Votes:
                  1 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development