Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.1
    • Component/s: java, spec
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Google has released it's Snappy compression codec.

      https://code.google.com/p/snappy/

      Avro should support this for data file compression.

      1. AVRO-788.patch
        10 kB
        Doug Cutting
      2. AVRO-788.patch
        6 kB
        Doug Cutting

        Issue Links

          Activity

          Hide
          Doug Cutting added a comment -

          I committed this.

          Show
          Doug Cutting added a comment - I committed this.
          Hide
          Scott Carey added a comment -

          +1 OK, tests pass. They embed compiled libraries for Mac, Windows, and Linux. So this feature is available only for those platforms at this time.

          Show
          Scott Carey added a comment - +1 OK, tests pass. They embed compiled libraries for Mac, Windows, and Linux. So this feature is available only for those platforms at this time.
          Hide
          Scott Carey added a comment -

          Hold off on my +1. I have one other concern. What JNI library is embedded here? Linux? Windows? Do users have to compile their own?

          http://code.google.com/p/snappy-java/source/browse/INSTALL

          I'm going to check this more thoroughly on my mac.

          Show
          Scott Carey added a comment - Hold off on my +1. I have one other concern. What JNI library is embedded here? Linux? Windows? Do users have to compile their own? http://code.google.com/p/snappy-java/source/browse/INSTALL I'm going to check this more thoroughly on my mac.
          Hide
          Doug Cutting added a comment -

          I've thus far avoided learning too much about Maven, so don't have an opinion.

          Show
          Doug Cutting added a comment - I've thus far avoided learning too much about Maven, so don't have an opinion.
          Hide
          Scott Carey added a comment -

          +1 Straightforward, looks good.

          I wonder about the maven scope. Should this be 'provided' scope due to its use of JNI? See http://code.google.com/p/snappy-java/ towards the bottom. A maven user can override the current scope as needed or exclude the jar, so this isn't critical and can be changed easily later.

          Show
          Scott Carey added a comment - +1 Straightforward, looks good. I wonder about the maven scope. Should this be 'provided' scope due to its use of JNI? See http://code.google.com/p/snappy-java/ towards the bottom. A maven user can override the current scope as needed or exclude the jar, so this isn't critical and can be changed easily later.
          Hide
          Doug Cutting added a comment -

          Any reason not to commit this soon?

          Show
          Doug Cutting added a comment - Any reason not to commit this soon?
          Hide
          Doug Cutting added a comment -

          Here's a new version of the patch that adds support for Snappy to Avro's MapReduce API and also updates the spec, adding Snappy as an optional codec for now.

          Show
          Doug Cutting added a comment - Here's a new version of the patch that adds support for Snappy to Avro's MapReduce API and also updates the spec, adding Snappy as an optional codec for now.
          Hide
          Doug Cutting added a comment -

          Here's a patch that implements this.

          Show
          Doug Cutting added a comment - Here's a patch that implements this.
          Hide
          Harsh J added a comment -

          There is also an acknowledged issue ticket for adding Java support. Link for reference: https://code.google.com/p/snappy/issues/detail?id=15

          Show
          Harsh J added a comment - There is also an acknowledged issue ticket for adding Java support. Link for reference: https://code.google.com/p/snappy/issues/detail?id=15

            People

            • Assignee:
              Doug Cutting
              Reporter:
              Doug Cutting
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development