Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is modified I/O and compaction path that allows individual moderately sized values (100KB-10MB) to be stored in a way that write amplification is reduced when compared to the normal I/O path. MOB is defined in the column family and it is almost isolated with other components, the features and performance cannot be effected in normal columns.

      For more details on how to use the feature please consult the HBase Reference Guide
      Show
      The Moderate Object Storage (MOB) feature ( HBASE-11339 [1]) is modified I/O and compaction path that allows individual moderately sized values (100KB-10MB) to be stored in a way that write amplification is reduced when compared to the normal I/O path. MOB is defined in the column family and it is almost isolated with other components, the features and performance cannot be effected in normal columns. For more details on how to use the feature please consult the HBase Reference Guide

      Description

      It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction.
      In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase.

        Attachments

        1. MOB user guide.docx
          27 kB
          Jiajia Li
        2. MOB user guide_v6.docx
          26 kB
          Jingcheng Du
        3. MOB user guide_v5.docx
          29 kB
          Jiajia Li
        4. MOB user guide_v4.docx
          29 kB
          Jiajia Li
        5. MOB user guide_v3.docx
          27 kB
          Jiajia Li
        6. MOB user guide_v2.docx
          27 kB
          Jiajia Li
        7. merge-150212.patch
          670 kB
          Jonathan Hsieh
        8. merge.150710.patch
          809 kB
          Jingcheng Du
        9. merge.150212c.patch
          669 kB
          Jonathan Hsieh
        10. merge.150212b.patch
          669 kB
          Jonathan Hsieh
        11. HBase MOB Design-v5.pdf
          1.30 MB
          Jingcheng Du
        12. HBase MOB Design-v4.pdf
          2.37 MB
          Jingcheng Du
        13. HBase MOB Design-v3.pdf
          2.35 MB
          Jingcheng Du
        14. HBase MOB Design-v2.pdf
          2.34 MB
          Jingcheng Du
        15. HBase MOB Design.pdf
          2.15 MB
          Jingcheng Du
        16. hbase-11339-in-dev.patch
          102 kB
          Jingcheng Du
        17. hbase-11339-150519.patch
          758 kB
          Jonathan Hsieh
        18. hbase-11339.150417.patch
          743 kB
          Jonathan Hsieh
        19. 11339-master-v9.patch
          809 kB
          Jingcheng Du
        20. 11339-master-v8.patch
          809 kB
          Jingcheng Du
        21. 11339-master-v7.txt
          811 kB
          Ted Yu
        22. 11339-master-v6.txt
          811 kB
          Ted Yu
        23. 11339-master-v5.txt
          811 kB
          Ted Yu
        24. 11339-master-v4.txt
          811 kB
          Ted Yu
        25. 11339-master-v3.txt
          811 kB
          Ted Yu
        26. 11339-master-v10.patch
          817 kB
          Ted Yu

          Issue Links

          1.
          Read and write MOB in HBase Sub-task Resolved Jingcheng Du
          2.
          External MOB compaction tools Sub-task Resolved Jingcheng Du
          3.
          Handle the MOB in compaction Sub-task Resolved Jingcheng Du
          4.
          MOB integration testing Sub-task Resolved Jingcheng Du
          5.
          Snapshot for MOB Sub-task Resolved Jingcheng Du
          6.
          Metrics for MOB Sub-task Resolved Jingcheng Du
          7.
          Native MOB Compaction mechanisms. Sub-task Resolved Jingcheng Du
          8.
          Improve the value size of the reference cell in mob column Sub-task Resolved Jingcheng Du
          9.
          Document MOB in Ref Guide Sub-task Resolved Misty Stanley-Jones
          10.
          isMob and mobThreshold do not follow column descriptor property naming conventions Sub-task Resolved Misty Stanley-Jones
          11.
          Avoid major compaction in TestMobSweeper Sub-task Resolved Jonathan Hsieh
          12.
          Shorten the run time of integration test by default when using mvn failsafe:integration-test Sub-task Resolved Jingcheng Du
          13.
          mob status should print human readable numbers. Sub-task Resolved Jingcheng Du
          14.
          Support the mob attributes in hbase shell when create/alter table Sub-task Resolved Jingcheng Du
          15.
          Clean the code after adding IS_MOB and MOB_THRESHOLD to column family Sub-task Resolved Jingcheng Du
          16.
          Shorten the mob snapshot unit tests Sub-task Resolved Jiajia Li
          17.
          [mob] improve how we resolve mobfiles in reads Sub-task Resolved Jiajia Li
          18.
          Correct a typo in the mob metrics Sub-task Resolved Jingcheng Du
          19.
          Incorrect implementation of CompactionRequest.isRetainDeleteMarkers Sub-task Resolved Jingcheng Du
          20.
          Move the mob table name tag to the 2nd one Sub-task Resolved Jingcheng Du
          21.
          Explicitly flush the file name in sweep job Sub-task Resolved Jingcheng Du
          22.
          Incorrect 'mobFileCacheMissCount' calculated in the mob metrics Sub-task Resolved Jiajia Li
          23.
          Incorrect log info in the store compaction of mob Sub-task Resolved Jiajia Li
          24.
          Ignore the count of mob compaction metrics when there is issue Sub-task Resolved Jiajia Li
          25.
          SnapshotInfo tool does not find mob data in snapshots Sub-task Resolved Jonathan Hsieh
          26.
          Have compaction scanner save info about delete markers Sub-task Resolved Jingcheng Du
          27.
          Add unit tests that exercise the added hfilelink link mob paths Sub-task Resolved Jingcheng Du
          28.
          Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir Sub-task Resolved Jiajia Li
          29.
          sweep job needs to exit non-zero if job fails for any reason. Sub-task Resolved Jonathan Hsieh
          30.
          Add mob cell count to the metadata of each mob file Sub-task Resolved Jingcheng Du
          31.
          treat mob region as any other region when generating rs manifest. Sub-task Resolved Jonathan Hsieh
          32.
          Use table lock instead of MobZookeeper Sub-task Resolved Jingcheng Du
          33.
          Add shell commands to trigger the mob file compactor Sub-task Resolved Jingcheng Du
          34.
          Add read lock to ExpiredMobFileCleanerChore Sub-task Resolved Jingcheng Du
          35.
          Refactor MOB Snapshot logic to reduce code duplication. Sub-task Resolved Jingcheng Du
          36.
          improve mob sweeper javadoc Sub-task Resolved Jonathan Hsieh
          37.
          IllegalArgumentException in compaction when table has a namespace Sub-task Resolved Jingcheng Du
          38.
          NPE in ExpiredMobFileCleanerChore Sub-task Resolved Jingcheng Du
          39.
          Add support for mob in TestAcidGuarantees and IntegrationTestAcidGuarantees Sub-task Resolved Jonathan Hsieh
          40.
          Add mob compaction actions and monkeys to Chaos Monkey Sub-task Resolved Jonathan Hsieh
          41.
          add mob_threshold option to load test tool Sub-task Resolved Jonathan Hsieh
          42.
          [mob] reads hang when trying to read rows with large mobs (>10MB) Sub-task Resolved Jonathan Hsieh
          43.
          fix new javadoc warns introduced by mob. Sub-task Resolved Jonathan Hsieh
          44.
          Skip the disabled table in mob compaction chore and MasterRpcServices Sub-task Resolved Jingcheng Du
          45.
          Flakey failures of TestAcidGuarantees#testMobScanAtomicity Sub-task Resolved Jingcheng Du
          46.
          Mob files are not encrypting in mob compaction and Sweeper Sub-task Resolved Jingcheng Du
          47.
          Add delay for the first execution of ExpiredMobFileCleanerChore and MobFileCompactorChore Sub-task Resolved Jingcheng Du
          48.
          Remove KeyValueUtil.ensureKeyValue(cell) from MOB code. Sub-task Resolved Jingcheng Du
          49.
          Use the same HFileContext with store files in mob files Sub-task Resolved Jingcheng Du
          50.
          Handle the rename, annotation and typo stuff in MOB Sub-task Resolved Jingcheng Du
          51.
          Remove the DeleteTableHandler Sub-task Resolved Jingcheng Du
          52.
          Remove the DeleteTableHandler Sub-task Resolved Jingcheng Du
          53.
          Disable the MobCompactionChore when the interval is not larger than 0 Sub-task Resolved Jingcheng Du
          54.
          Revert the changes in pom.xml Sub-task Resolved Jingcheng Du
          55.
          Use LimitInputStream in hbase-common instead of ProtobufUtil.LimitedInputStream Sub-task Resolved Jingcheng Du
          56.
          Check the mob files when there are mob-enabled columns in HFileCorruptionChecker Sub-task Resolved Jingcheng Du
          57.
          Do not reset the mvcc for bulk loaded mob reference cells in reading Sub-task Resolved Jingcheng Du
          58.
          Race in multi threaded PartitionedMobCompactor causes NPE Sub-task Resolved Jingcheng Du
          59.
          Wrong mob metrics names in TestRegionServerMetrics Sub-task Resolved Jingcheng Du
          60.
          Return empty value when the mob file is corrupt instead of throwing exceptions Sub-task Resolved Jingcheng Du
          61.
          Do not reset mvcc in compactions for mob-enabled column Sub-task Resolved Jingcheng Du
          62.
          Add mob integrity check in HFilePrettyPrinter Sub-task Resolved Jingcheng Du
          63.
          Not cleaning Mob data when Mob CF is removed from table Sub-task Resolved Pankaj Kumar

            Activity

              People

              • Assignee:
                jingcheng.du@intel.com Jingcheng Du
                Reporter:
                jingcheng.du@intel.com Jingcheng Du
              • Votes:
                1 Vote for this issue
                Watchers:
                47 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: