Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8089 Add type support
  3. HBASE-8201

OrderedBytes: an ordered encoding strategy

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.98.0, 0.95.2
    • Client
    • None
    • Reviewed
    • Hide
      OrderedBytes provides a data encoding format in which the resulting byte[] retains the same sort order as the natural types. Encoded formats can be inspected and decoded without forward knowledge of their content. Implementations are provided for integer and floating point numbers with 32- and 64-bits of precision, numeric values of arbitrary precision, Strings, and byte[]s. Utility methods for counting and skipping encoded entries are also provided.

      Most of the encoding formats are modeled after the SQLite4 key encoding format. Implementations of variable-length encodings are very similar. Fixed-width encodings are modeled after the fixed-width formats provided by the Orderly library. Javadocs on the OrderedBytes class describe the encoding format in detail. See http://sqlite.org/src4/doc/trunk/www/key_encoding.wiki for additional context.

      Notable deviation from the sqlite4 spec include:
       - Different header byte values. This is to give users more room to place their own, custom encoding extensions as they see fit.
       - BlobCopy is modified to include a termination byte of 0x00 in descending order. This is necessary in order to maintain reverse sort order of empty values. It is also renamed from "BlobLast".
       - Support for fixed-length integer and float encodings. Based on implementations from the Orderly library.
      Show
      OrderedBytes provides a data encoding format in which the resulting byte[] retains the same sort order as the natural types. Encoded formats can be inspected and decoded without forward knowledge of their content. Implementations are provided for integer and floating point numbers with 32- and 64-bits of precision, numeric values of arbitrary precision, Strings, and byte[]s. Utility methods for counting and skipping encoded entries are also provided. Most of the encoding formats are modeled after the SQLite4 key encoding format. Implementations of variable-length encodings are very similar. Fixed-width encodings are modeled after the fixed-width formats provided by the Orderly library. Javadocs on the OrderedBytes class describe the encoding format in detail. See http://sqlite.org/src4/doc/trunk/www/key_encoding.wiki for additional context. Notable deviation from the sqlite4 spec include:  - Different header byte values. This is to give users more room to place their own, custom encoding extensions as they see fit.  - BlobCopy is modified to include a termination byte of 0x00 in descending order. This is necessary in order to maintain reverse sort order of empty values. It is also renamed from "BlobLast".  - Support for fixed-length integer and float encodings. Based on implementations from the Orderly library.
    • 0.96notable

    Description

      Once the spec is agreed upon, it must be implemented.

      Attachments

        1. 0001-HBASE-8201-OrderedBytes-order-preserving-encoding.patch
          113 kB
          Nick Dimiduk
        2. 0001-HBASE-8201-OrderedBytes-order-preserving-encoding.patch
          113 kB
          Nick Dimiduk
        3. 0001-HBASE-8201-OrderedBytes-order-preserving-encoding.patch
          113 kB
          Nick Dimiduk
        4. 0001-HBASE-8201-OrderedBytes-order-preserving-encoding.patch
          99 kB
          Nick Dimiduk
        5. 0001-HBASE-8201-OrderedBytes-order-preserving-encoding.patch
          98 kB
          Nick Dimiduk
        6. 0001-HBASE-8201-OrderedBytes-order-preserving-encoding.patch
          90 kB
          Nick Dimiduk
        7. 0001-HBASE-8201-OrderedBytes-order-preserving-encoding.patch
          126 kB
          Nick Dimiduk
        8. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          93 kB
          Nick Dimiduk
        9. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          91 kB
          Nick Dimiduk
        10. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          85 kB
          Nick Dimiduk
        11. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          84 kB
          Nick Dimiduk
        12. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          84 kB
          Nick Dimiduk
        13. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          58 kB
          Nick Dimiduk
        14. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          58 kB
          Nick Dimiduk
        15. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          55 kB
          Nick Dimiduk
        16. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          58 kB
          Nick Dimiduk
        17. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          42 kB
          Nick Dimiduk
        18. 0001-HBASE-8201-OrderedBytes-provides-order-preserving-se.patch
          29 kB
          Nick Dimiduk

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HBASE-9091 Update ByteRange to maintain consumer's position

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Sub-task - The sub-task of the issue HBASE-8693 DataType: provide extensible type API

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-9591 Add support for OrderedByte encodings

          • Major - Major loss of function.
          • Open
          is related to

          New Feature - A new feature of the product, which has yet to be developed. DRILL-3492 Add support for encoding of Drill data types into byte ordered format

          • Major - Major loss of function.
          • Closed

          Activity

            People

              ndimiduk Nick Dimiduk
              ndimiduk Nick Dimiduk
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: