Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.4
    • Fix Version/s: 1.7.5
    • Component/s: c
    • Labels:
    • Environment:

      N/A

    • Tags:
      snappy, c

      Description

      Hi again,

      I'd really love to have Snappy compression available in AVRO-C.

      Cheers,
      Lucas

        Activity

        Lucas Martin-King created issue -
        Hide
        Michael Cooper added a comment -

        This should be relatively simple to implement ontop of the changes in AVRO-957.

        Show
        Michael Cooper added a comment - This should be relatively simple to implement ontop of the changes in AVRO-957 .
        Michael Cooper made changes -
        Field Original Value New Value
        Description Hi again,

        I'd really love to have Snappy compression available in AVRO-C.

        Once this is supported, we can use it for a major project we are working on (instead of using a JSON to AVRO converter script), which will result in gigabytes of AVRO data being created every day.

        Cheers,
        Lucas
        Hi again,

        I'd really love to have Snappy compression available in AVRO-C.

        Cheers,
        Lucas
        Hide
        Grisha Trubetskoy added a comment -

        Here is a (git generated, sorry) diff for support for the Snappy codec in Avro-c. (This patch includes the crc32 according to the Avro spec, so that it is fully compatible with the Java version).

        Show
        Grisha Trubetskoy added a comment - Here is a (git generated, sorry) diff for support for the Snappy codec in Avro-c. (This patch includes the crc32 according to the Avro spec, so that it is fully compatible with the Java version).
        Grisha Trubetskoy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 1.7.4 [ 12323742 ]
        Affects Version/s 1.6.0 [ 12316198 ]
        Hide
        Grisha Trubetskoy added a comment -

        Patch for Snappy (AVRO-896)

        Show
        Grisha Trubetskoy added a comment - Patch for Snappy ( AVRO-896 )
        Grisha Trubetskoy made changes -
        Attachment avro-c-snappy.patch [ 12587487 ]
        Grisha Trubetskoy made changes -
        Attachment avro-c-snappy.patch [ 12587487 ]
        Hide
        Grisha Trubetskoy added a comment -

        Snappy patch for Avro-C.

        Show
        Grisha Trubetskoy added a comment - Snappy patch for Avro-C.
        Grisha Trubetskoy made changes -
        Attachment avro-c-1.7.4-snappy.patch [ 12587492 ]
        Grisha Trubetskoy made changes -
        Attachment avro-c-1.7.4-snappy.patch [ 12587492 ]
        Hide
        Grisha Trubetskoy added a comment -

        Snappy patch for Avro-C 1.7.4 (third time's the charm)

        Show
        Grisha Trubetskoy added a comment - Snappy patch for Avro-C 1.7.4 (third time's the charm)
        Grisha Trubetskoy made changes -
        Attachment avro-c-1.7.4-snappy.patch [ 12587493 ]
        Hide
        ASF subversion and git services added a comment -

        Commit 1507309 from dcreager@apache.org in branch 'avro/trunk'
        [ https://svn.apache.org/r1507309 ]

        AVRO-896. C: Snappy compression codec.

        Like all other codecs, this is optional. It will only be activated if the
        snappy C library is installed at build time. Contributed by Grisha Trubetskoy.

        Show
        ASF subversion and git services added a comment - Commit 1507309 from dcreager@apache.org in branch 'avro/trunk' [ https://svn.apache.org/r1507309 ] AVRO-896 . C: Snappy compression codec. Like all other codecs, this is optional. It will only be activated if the snappy C library is installed at build time. Contributed by Grisha Trubetskoy.
        Hide
        Douglas Creager added a comment -

        Had to fix one compiler warning (comparison of signed and unsigned values). Other than that, looks good. Merged in SVN trunk.

        Show
        Douglas Creager added a comment - Had to fix one compiler warning (comparison of signed and unsigned values). Other than that, looks good. Merged in SVN trunk.
        Douglas Creager made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 1.7.5 [ 12324096 ]
        Resolution Fixed [ 1 ]
        Hide
        ASF subversion and git services added a comment -

        Commit 1507318 from dcreager@apache.org in branch 'avro/trunk'
        [ https://svn.apache.org/r1507318 ]

        AVRO-896. C: Snappy is an optional dependency

        Buildbot doesn't have the snappy libraries installed, and it raised an error
        trying to compile the C bindings with the previous AVRO-896 patch applied.
        Turns out we weren't being careful enough when the Snappy library (and
        presumably the other compression libraries) weren't installed. Now this
        correctly results in a good build, of a library with snappy support turned off,
        as intended.

        Show
        ASF subversion and git services added a comment - Commit 1507318 from dcreager@apache.org in branch 'avro/trunk' [ https://svn.apache.org/r1507318 ] AVRO-896 . C: Snappy is an optional dependency Buildbot doesn't have the snappy libraries installed, and it raised an error trying to compile the C bindings with the previous AVRO-896 patch applied. Turns out we weren't being careful enough when the Snappy library (and presumably the other compression libraries) weren't installed. Now this correctly results in a good build, of a library with snappy support turned off, as intended.
        Hide
        ASF subversion and git services added a comment -

        Commit 1507318 from dcreager@apache.org in branch 'avro/trunk'
        [ https://svn.apache.org/r1507318 ]

        AVRO-896. C: Snappy is an optional dependency

        Buildbot doesn't have the snappy libraries installed, and it raised an error
        trying to compile the C bindings with the previous AVRO-896 patch applied.
        Turns out we weren't being careful enough when the Snappy library (and
        presumably the other compression libraries) weren't installed. Now this
        correctly results in a good build, of a library with snappy support turned off,
        as intended.

        Show
        ASF subversion and git services added a comment - Commit 1507318 from dcreager@apache.org in branch 'avro/trunk' [ https://svn.apache.org/r1507318 ] AVRO-896 . C: Snappy is an optional dependency Buildbot doesn't have the snappy libraries installed, and it raised an error trying to compile the C bindings with the previous AVRO-896 patch applied. Turns out we weren't being careful enough when the Snappy library (and presumably the other compression libraries) weren't installed. Now this correctly results in a good build, of a library with snappy support turned off, as intended.
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        624d 18h 12m 1 Grisha Trubetskoy 12/Jun/13 19:30
        Patch Available Patch Available Resolved Resolved
        43d 18h 11m 1 Douglas Creager 26/Jul/13 13:41
        Resolved Resolved Closed Closed
        25d 4h 4m 1 Doug Cutting 20/Aug/13 17:46

          People

          • Assignee:
            Unassigned
            Reporter:
            Lucas Martin-King
          • Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development