VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is modified I/O and compaction path that allows individual moderately sized values (100KB-10MB) to be stored in a way that write amplification is reduced when compared to the normal I/O path. MOB is defined in the column family and it is almost isolated with other components, the features and performance cannot be effected in normal columns.

      For more details on how to use the feature please consult the HBase Reference Guide
      Show
      The Moderate Object Storage (MOB) feature ( HBASE-11339 [1]) is modified I/O and compaction path that allows individual moderately sized values (100KB-10MB) to be stored in a way that write amplification is reduced when compared to the normal I/O path. MOB is defined in the column family and it is almost isolated with other components, the features and performance cannot be effected in normal columns. For more details on how to use the feature please consult the HBase Reference Guide

    Description

      It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction.
      In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase.

      Attachments

        1. 11339-master-v10.patch
          817 kB
          Ted Yu
        2. 11339-master-v3.txt
          811 kB
          Ted Yu
        3. 11339-master-v4.txt
          811 kB
          Ted Yu
        4. 11339-master-v5.txt
          811 kB
          Ted Yu
        5. 11339-master-v6.txt
          811 kB
          Ted Yu
        6. 11339-master-v7.txt
          811 kB
          Ted Yu
        7. 11339-master-v8.patch
          809 kB
          Jingcheng Du
        8. 11339-master-v9.patch
          809 kB
          Jingcheng Du
        9. hbase-11339.150417.patch
          743 kB
          Jonathan Hsieh
        10. hbase-11339-150519.patch
          758 kB
          Jonathan Hsieh
        11. hbase-11339-in-dev.patch
          102 kB
          Jingcheng Du
        12. HBase MOB Design.pdf
          2.15 MB
          Jingcheng Du
        13. HBase MOB Design-v2.pdf
          2.34 MB
          Jingcheng Du
        14. HBase MOB Design-v3.pdf
          2.35 MB
          Jingcheng Du
        15. HBase MOB Design-v4.pdf
          2.37 MB
          Jingcheng Du
        16. HBase MOB Design-v5.pdf
          1.30 MB
          Jingcheng Du
        17. merge.150212b.patch
          669 kB
          Jonathan Hsieh
        18. merge.150212c.patch
          669 kB
          Jonathan Hsieh
        19. merge.150710.patch
          809 kB
          Jingcheng Du
        20. merge-150212.patch
          670 kB
          Jonathan Hsieh
        21. MOB user guide_v2.docx
          27 kB
          Jiajia Li
        22. MOB user guide_v3.docx
          27 kB
          Jiajia Li
        23. MOB user guide_v4.docx
          29 kB
          Jiajia Li
        24. MOB user guide_v5.docx
          29 kB
          Jiajia Li
        25. MOB user guide_v6.docx
          26 kB
          Jingcheng Du
        26. MOB user guide.docx
          27 kB
          Jiajia Li

        Issue Links

        1.
        Read and write MOB in HBase Sub-task Closed Jingcheng Du Actions
        2.
        External MOB compaction tools Sub-task Closed Jingcheng Du Actions
        3.
        Handle the MOB in compaction Sub-task Closed Jingcheng Du Actions
        4.
        MOB integration testing Sub-task Closed Jingcheng Du Actions
        5.
        Snapshot for MOB Sub-task Closed Jingcheng Du Actions
        6.
        Metrics for MOB Sub-task Closed Jingcheng Du Actions
        7.
        Native MOB Compaction mechanisms. Sub-task Closed Jingcheng Du Actions
        8.
        Improve the value size of the reference cell in mob column Sub-task Closed Jingcheng Du Actions
        9.
        Document MOB in Ref Guide Sub-task Closed M Linville Actions
        10.
        isMob and mobThreshold do not follow column descriptor property naming conventions Sub-task Closed M Linville Actions
        11.
        Avoid major compaction in TestMobSweeper Sub-task Closed Jonathan Hsieh Actions
        12.
        Shorten the run time of integration test by default when using mvn failsafe:integration-test Sub-task Closed Jingcheng Du Actions
        13.
        mob status should print human readable numbers. Sub-task Closed Jingcheng Du Actions
        14.
        Support the mob attributes in hbase shell when create/alter table Sub-task Closed Jingcheng Du Actions
        15.
        Clean the code after adding IS_MOB and MOB_THRESHOLD to column family Sub-task Closed Jingcheng Du Actions
        16.
        Shorten the mob snapshot unit tests Sub-task Closed Jiajia Li Actions
        17.
        [mob] improve how we resolve mobfiles in reads Sub-task Closed Jiajia Li Actions
        18.
        Correct a typo in the mob metrics Sub-task Closed Jingcheng Du Actions
        19.
        Incorrect implementation of CompactionRequest.isRetainDeleteMarkers Sub-task Closed Jingcheng Du Actions
        20.
        Move the mob table name tag to the 2nd one Sub-task Closed Jingcheng Du Actions
        21.
        Explicitly flush the file name in sweep job Sub-task Closed Jingcheng Du Actions
        22.
        Incorrect 'mobFileCacheMissCount' calculated in the mob metrics Sub-task Closed Jiajia Li Actions
        23.
        Incorrect log info in the store compaction of mob Sub-task Closed Jiajia Li Actions
        24.
        Ignore the count of mob compaction metrics when there is issue Sub-task Closed Jiajia Li Actions
        25.
        SnapshotInfo tool does not find mob data in snapshots Sub-task Closed Jonathan Hsieh Actions
        26.
        Have compaction scanner save info about delete markers Sub-task Closed Jingcheng Du Actions
        27.
        Add unit tests that exercise the added hfilelink link mob paths Sub-task Closed Jingcheng Du Actions
        28.
        Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir Sub-task Closed Jiajia Li Actions
        29.
        sweep job needs to exit non-zero if job fails for any reason. Sub-task Closed Jonathan Hsieh Actions
        30.
        Add mob cell count to the metadata of each mob file Sub-task Closed Jingcheng Du Actions
        31.
        treat mob region as any other region when generating rs manifest. Sub-task Closed Jonathan Hsieh Actions
        32.
        Use table lock instead of MobZookeeper Sub-task Closed Jingcheng Du Actions
        33.
        Add shell commands to trigger the mob file compactor Sub-task Closed Jingcheng Du Actions
        34.
        Add read lock to ExpiredMobFileCleanerChore Sub-task Closed Jingcheng Du Actions
        35.
        Refactor MOB Snapshot logic to reduce code duplication. Sub-task Closed Jingcheng Du Actions
        36.
        improve mob sweeper javadoc Sub-task Closed Jonathan Hsieh Actions
        37.
        IllegalArgumentException in compaction when table has a namespace Sub-task Closed Jingcheng Du Actions
        38.
        NPE in ExpiredMobFileCleanerChore Sub-task Closed Jingcheng Du Actions
        39.
        Add support for mob in TestAcidGuarantees and IntegrationTestAcidGuarantees Sub-task Closed Jonathan Hsieh Actions
        40.
        Add mob compaction actions and monkeys to Chaos Monkey Sub-task Closed Jonathan Hsieh Actions
        41.
        add mob_threshold option to load test tool Sub-task Closed Jonathan Hsieh Actions
        42.
        [mob] reads hang when trying to read rows with large mobs (>10MB) Sub-task Closed Jonathan Hsieh Actions
        43.
        fix new javadoc warns introduced by mob. Sub-task Closed Jonathan Hsieh Actions
        44.
        Skip the disabled table in mob compaction chore and MasterRpcServices Sub-task Closed Jingcheng Du Actions
        45.
        Flakey failures of TestAcidGuarantees#testMobScanAtomicity Sub-task Closed Jingcheng Du Actions
        46.
        Mob files are not encrypting in mob compaction and Sweeper Sub-task Closed Jingcheng Du Actions
        47.
        Add delay for the first execution of ExpiredMobFileCleanerChore and MobFileCompactorChore Sub-task Closed Jingcheng Du Actions
        48.
        Remove KeyValueUtil.ensureKeyValue(cell) from MOB code. Sub-task Closed Jingcheng Du Actions
        49.
        Use the same HFileContext with store files in mob files Sub-task Closed Jingcheng Du Actions
        50.
        Handle the rename, annotation and typo stuff in MOB Sub-task Closed Jingcheng Du Actions
        51.
        Remove the DeleteTableHandler Sub-task Closed Jingcheng Du Actions
        52.
        Remove the DeleteTableHandler Sub-task Closed Jingcheng Du Actions
        53.
        Disable the MobCompactionChore when the interval is not larger than 0 Sub-task Closed Jingcheng Du Actions
        54.
        Revert the changes in pom.xml Sub-task Closed Jingcheng Du Actions
        55.
        Use LimitInputStream in hbase-common instead of ProtobufUtil.LimitedInputStream Sub-task Closed Jingcheng Du Actions
        56.
        Check the mob files when there are mob-enabled columns in HFileCorruptionChecker Sub-task Closed Jingcheng Du Actions
        57.
        Do not reset the mvcc for bulk loaded mob reference cells in reading Sub-task Closed Jingcheng Du Actions
        58.
        Race in multi threaded PartitionedMobCompactor causes NPE Sub-task Closed Jingcheng Du Actions
        59.
        Wrong mob metrics names in TestRegionServerMetrics Sub-task Closed Jingcheng Du Actions
        60.
        Return empty value when the mob file is corrupt instead of throwing exceptions Sub-task Closed Jingcheng Du Actions
        61.
        Do not reset mvcc in compactions for mob-enabled column Sub-task Closed Jingcheng Du Actions
        62.
        Add mob integrity check in HFilePrettyPrinter Sub-task Closed Jingcheng Du Actions
        63.
        Not cleaning Mob data when Mob CF is removed from table Sub-task Closed Pankaj Kumar Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jingcheng.du@intel.com Jingcheng Du
            jingcheng.du@intel.com Jingcheng Du
            Votes:
            1 Vote for this issue
            Watchers:
            50 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment