Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5793

High speed compression algorithm like BMDiff

          Details

          • Type: New Feature
          • Status: Open
          • Priority: Minor
          • Resolution: Unresolved
          • Affects Version/s: None
          • Fix Version/s: None
          • Component/s: None
          • Labels:
            None

            Description

            Add a high speed compression algorithm like BMDiff.
            It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB

            Reference:
            http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
            2005 Jeff Dean talk about google architecture - around 46:00.

            http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/

            http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678

            A reference implementation exists in HyperTable.

              Attachments

                Issue Links

                is depended upon by

                New Feature - A new feature of the product, which has yet to be developed. HBASE-2655 2-pass compression support

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      pirroh Michele Catasta
                      Reporter:
                      elhoim elhoim gibor
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      24 Start watching this issue

                      Dates

                      • Created:
                        Updated: