Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-448

Support Gzip encoding for replicating over slow connections

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: HTTP Interface
    • Labels:
      None
    • Skill Level:
      Regular Contributors Level (Easy to Medium)

      Description

      This shouldn't be too hard to add, we should support it in general for all HTTP requests to the server and also allow it to be enabled in the replicator client for pull/push replication.

        Activity

        Hide
        colinrgodsey Colin Godsey added a comment -

        This should be a configurable option for all HTTP requests, as deflate/gzip is an HTTP standard, and quite useful when pushing large json documents over the internet. Compressed json is still faster than most serialization methods, and pretty dense too.

        Show
        colinrgodsey Colin Godsey added a comment - This should be a configurable option for all HTTP requests, as deflate/gzip is an HTTP standard, and quite useful when pushing large json documents over the internet. Compressed json is still faster than most serialization methods, and pretty dense too.
        Hide
        rnewson Robert Newson added a comment -

        A useful strategy is to compress the first few k and and then either compress the whole thing or not based on the ratio. saves guessing based on mime type.

        finally, a naive gzip is vulnerable to zip bombs, so google for that too.

        Show
        rnewson Robert Newson added a comment - A useful strategy is to compress the first few k and and then either compress the whole thing or not based on the ratio. saves guessing based on mime type. finally, a naive gzip is vulnerable to zip bombs, so google for that too.
        Hide
        damienkatz Damien Katz added a comment -

        Ideally we'll store attachments gzipped, and then just stream them unchanged for clients that can handle, decompress for clients that can't.

        We'll probably need a config file to avoid mime types that aren't compressable, like images and movies.

        Show
        damienkatz Damien Katz added a comment - Ideally we'll store attachments gzipped, and then just stream them unchanged for clients that can handle, decompress for clients that can't. We'll probably need a config file to avoid mime types that aren't compressable, like images and movies.
        Hide
        jasondd Jason Davies added a comment -

        Note to self or whoever implements this: the ETag should change depending on what Content-Encoding is used.

        Show
        jasondd Jason Davies added a comment - Note to self or whoever implements this: the ETag should change depending on what Content-Encoding is used.
        Hide
        jasondd Jason Davies added a comment -
        Show
        jasondd Jason Davies added a comment - This may be of use: http://github.com/lgerbarg/couchdb/tree/gzip-support
        Hide
        kocolosk Adam Kocoloski added a comment -

        couch_rep_httpc now unzips response bodies

        Show
        kocolosk Adam Kocoloski added a comment - couch_rep_httpc now unzips response bodies
        Hide
        kocolosk Adam Kocoloski added a comment -

        ibrowse does not unzip response bodies, but it's simple enough to do this ourselves with a call to zlib:gunzip/1

        Show
        kocolosk Adam Kocoloski added a comment - ibrowse does not unzip response bodies, but it's simple enough to do this ourselves with a call to zlib:gunzip/1

          People

          • Assignee:
            kocolosk Adam Kocoloski
            Reporter:
            jasondd Jason Davies
          • Votes:
            12 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development