CouchDB
  1. CouchDB
  2. COUCHDB-448

Support Gzip encoding for replicating over slow connections

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: HTTP Interface
    • Labels:
      None
    • Skill Level:
      Regular Contributors Level (Easy to Medium)

      Description

      This shouldn't be too hard to add, we should support it in general for all HTTP requests to the server and also allow it to be enabled in the replicator client for pull/push replication.

        Activity

        Hide
        Adam Kocoloski added a comment -

        ibrowse does not unzip response bodies, but it's simple enough to do this ourselves with a call to zlib:gunzip/1

        Show
        Adam Kocoloski added a comment - ibrowse does not unzip response bodies, but it's simple enough to do this ourselves with a call to zlib:gunzip/1
        Hide
        Adam Kocoloski added a comment -

        couch_rep_httpc now unzips response bodies

        Show
        Adam Kocoloski added a comment - couch_rep_httpc now unzips response bodies
        Hide
        Jason Davies added a comment -
        Show
        Jason Davies added a comment - This may be of use: http://github.com/lgerbarg/couchdb/tree/gzip-support
        Hide
        Jason Davies added a comment -

        Note to self or whoever implements this: the ETag should change depending on what Content-Encoding is used.

        Show
        Jason Davies added a comment - Note to self or whoever implements this: the ETag should change depending on what Content-Encoding is used.
        Hide
        Damien Katz added a comment -

        Ideally we'll store attachments gzipped, and then just stream them unchanged for clients that can handle, decompress for clients that can't.

        We'll probably need a config file to avoid mime types that aren't compressable, like images and movies.

        Show
        Damien Katz added a comment - Ideally we'll store attachments gzipped, and then just stream them unchanged for clients that can handle, decompress for clients that can't. We'll probably need a config file to avoid mime types that aren't compressable, like images and movies.
        Hide
        Robert Newson added a comment -

        A useful strategy is to compress the first few k and and then either compress the whole thing or not based on the ratio. saves guessing based on mime type.

        finally, a naive gzip is vulnerable to zip bombs, so google for that too.

        Show
        Robert Newson added a comment - A useful strategy is to compress the first few k and and then either compress the whole thing or not based on the ratio. saves guessing based on mime type. finally, a naive gzip is vulnerable to zip bombs, so google for that too.
        Hide
        Colin Godsey added a comment -

        This should be a configurable option for all HTTP requests, as deflate/gzip is an HTTP standard, and quite useful when pushing large json documents over the internet. Compressed json is still faster than most serialization methods, and pretty dense too.

        Show
        Colin Godsey added a comment - This should be a configurable option for all HTTP requests, as deflate/gzip is an HTTP standard, and quite useful when pushing large json documents over the internet. Compressed json is still faster than most serialization methods, and pretty dense too.

          People

          • Assignee:
            Adam Kocoloski
            Reporter:
            Jason Davies
          • Votes:
            13 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development