VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is modified I/O and compaction path that allows individual moderately sized values (100KB-10MB) to be stored in a way that write amplification is reduced when compared to the normal I/O path. MOB is defined in the column family and it is almost isolated with other components, the features and performance cannot be effected in normal columns.

      For more details on how to use the feature please consult the HBase Reference Guide
      Show
      The Moderate Object Storage (MOB) feature ( HBASE-11339 [1]) is modified I/O and compaction path that allows individual moderately sized values (100KB-10MB) to be stored in a way that write amplification is reduced when compared to the normal I/O path. MOB is defined in the column family and it is almost isolated with other components, the features and performance cannot be effected in normal columns. For more details on how to use the feature please consult the HBase Reference Guide

      Description

      It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction.
      In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase.

        Attachments

        1. MOB user guide_v6.docx
          26 kB
          Jingcheng Du
        2. 11339-master-v10.patch
          817 kB
          Ted Yu
        3. 11339-master-v9.patch
          809 kB
          Jingcheng Du
        4. 11339-master-v8.patch
          809 kB
          Jingcheng Du
        5. 11339-master-v7.txt
          811 kB
          Ted Yu
        6. 11339-master-v6.txt
          811 kB
          Ted Yu
        7. 11339-master-v5.txt
          811 kB
          Ted Yu
        8. 11339-master-v4.txt
          811 kB
          Ted Yu
        9. 11339-master-v3.txt
          811 kB
          Ted Yu
        10. merge.150710.patch
          809 kB
          Jingcheng Du
        11. hbase-11339-150519.patch
          758 kB
          Jonathan Hsieh
        12. hbase-11339.150417.patch
          743 kB
          Jonathan Hsieh
        13. HBase MOB Design-v5.pdf
          1.30 MB
          Jingcheng Du
        14. merge.150212c.patch
          669 kB
          Jonathan Hsieh
        15. merge.150212b.patch
          669 kB
          Jonathan Hsieh
        16. merge-150212.patch
          670 kB
          Jonathan Hsieh
        17. MOB user guide_v5.docx
          29 kB
          Jiajia Li
        18. MOB user guide_v4.docx
          29 kB
          Jiajia Li
        19. MOB user guide_v3.docx
          27 kB
          Jiajia Li
        20. MOB user guide_v2.docx
          27 kB
          Jiajia Li
        21. MOB user guide.docx
          27 kB
          Jiajia Li
        22. HBase MOB Design-v4.pdf
          2.37 MB
          Jingcheng Du
        23. HBase MOB Design-v3.pdf
          2.35 MB
          Jingcheng Du
        24. HBase MOB Design-v2.pdf
          2.34 MB
          Jingcheng Du
        25. hbase-11339-in-dev.patch
          102 kB
          Jingcheng Du
        26. HBase MOB Design.pdf
          2.15 MB
          Jingcheng Du

        Issue Links

        1.
        Read and write MOB in HBase Sub-task Resolved Jingcheng Du Actions
        2.
        External MOB compaction tools Sub-task Resolved Jingcheng Du Actions
        3.
        Handle the MOB in compaction Sub-task Resolved Jingcheng Du Actions
        4.
        MOB integration testing Sub-task Resolved Jingcheng Du Actions
        5.
        Snapshot for MOB Sub-task Resolved Jingcheng Du Actions
        6.
        Metrics for MOB Sub-task Resolved Jingcheng Du Actions
        7.
        Native MOB Compaction mechanisms. Sub-task Resolved Jingcheng Du Actions
        8.
        Improve the value size of the reference cell in mob column Sub-task Resolved Jingcheng Du Actions
        9.
        Document MOB in Ref Guide Sub-task Resolved M Linville Actions
        10.
        isMob and mobThreshold do not follow column descriptor property naming conventions Sub-task Resolved M Linville Actions
        11.
        Avoid major compaction in TestMobSweeper Sub-task Resolved Jonathan Hsieh Actions
        12.
        Shorten the run time of integration test by default when using mvn failsafe:integration-test Sub-task Resolved Jingcheng Du Actions
        13.
        mob status should print human readable numbers. Sub-task Resolved Jingcheng Du Actions
        14.
        Support the mob attributes in hbase shell when create/alter table Sub-task Resolved Jingcheng Du Actions
        15.
        Clean the code after adding IS_MOB and MOB_THRESHOLD to column family Sub-task Resolved Jingcheng Du Actions
        16.
        Shorten the mob snapshot unit tests Sub-task Resolved Jiajia Li Actions
        17.
        [mob] improve how we resolve mobfiles in reads Sub-task Resolved Jiajia Li Actions
        18.
        Correct a typo in the mob metrics Sub-task Resolved Jingcheng Du Actions
        19.
        Incorrect implementation of CompactionRequest.isRetainDeleteMarkers Sub-task Resolved Jingcheng Du Actions
        20.
        Move the mob table name tag to the 2nd one Sub-task Resolved Jingcheng Du Actions
        21.
        Explicitly flush the file name in sweep job Sub-task Resolved Jingcheng Du Actions
        22.
        Incorrect 'mobFileCacheMissCount' calculated in the mob metrics Sub-task Resolved Jiajia Li Actions
        23.
        Incorrect log info in the store compaction of mob Sub-task Resolved Jiajia Li Actions
        24.
        Ignore the count of mob compaction metrics when there is issue Sub-task Resolved Jiajia Li Actions
        25.
        SnapshotInfo tool does not find mob data in snapshots Sub-task Resolved Jonathan Hsieh Actions
        26.
        Have compaction scanner save info about delete markers Sub-task Resolved Jingcheng Du Actions
        27.
        Add unit tests that exercise the added hfilelink link mob paths Sub-task Resolved Jingcheng Du Actions
        28.
        Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir Sub-task Resolved Jiajia Li Actions
        29.
        sweep job needs to exit non-zero if job fails for any reason. Sub-task Resolved Jonathan Hsieh Actions
        30.
        Add mob cell count to the metadata of each mob file Sub-task Resolved Jingcheng Du Actions
        31.
        treat mob region as any other region when generating rs manifest. Sub-task Resolved Jonathan Hsieh Actions
        32.
        Use table lock instead of MobZookeeper Sub-task Resolved Jingcheng Du Actions
        33.
        Add shell commands to trigger the mob file compactor Sub-task Resolved Jingcheng Du Actions
        34.
        Add read lock to ExpiredMobFileCleanerChore Sub-task Resolved Jingcheng Du Actions
        35.
        Refactor MOB Snapshot logic to reduce code duplication. Sub-task Resolved Jingcheng Du Actions
        36.
        improve mob sweeper javadoc Sub-task Resolved Jonathan Hsieh Actions
        37.
        IllegalArgumentException in compaction when table has a namespace Sub-task Resolved Jingcheng Du Actions
        38.
        NPE in ExpiredMobFileCleanerChore Sub-task Resolved Jingcheng Du Actions
        39.
        Add support for mob in TestAcidGuarantees and IntegrationTestAcidGuarantees Sub-task Resolved Jonathan Hsieh Actions
        40.
        Add mob compaction actions and monkeys to Chaos Monkey Sub-task Resolved Jonathan Hsieh Actions
        41.
        add mob_threshold option to load test tool Sub-task Resolved Jonathan Hsieh Actions
        42.
        [mob] reads hang when trying to read rows with large mobs (>10MB) Sub-task Resolved Jonathan Hsieh Actions
        43.
        fix new javadoc warns introduced by mob. Sub-task Resolved Jonathan Hsieh Actions
        44.
        Skip the disabled table in mob compaction chore and MasterRpcServices Sub-task Resolved Jingcheng Du Actions
        45.
        Flakey failures of TestAcidGuarantees#testMobScanAtomicity Sub-task Resolved Jingcheng Du Actions
        46.
        Mob files are not encrypting in mob compaction and Sweeper Sub-task Resolved Jingcheng Du Actions
        47.
        Add delay for the first execution of ExpiredMobFileCleanerChore and MobFileCompactorChore Sub-task Resolved Jingcheng Du Actions
        48.
        Remove KeyValueUtil.ensureKeyValue(cell) from MOB code. Sub-task Resolved Jingcheng Du Actions
        49.
        Use the same HFileContext with store files in mob files Sub-task Resolved Jingcheng Du Actions
        50.
        Handle the rename, annotation and typo stuff in MOB Sub-task Resolved Jingcheng Du Actions
        51.
        Remove the DeleteTableHandler Sub-task Resolved Jingcheng Du Actions
        52.
        Remove the DeleteTableHandler Sub-task Resolved Jingcheng Du Actions
        53.
        Disable the MobCompactionChore when the interval is not larger than 0 Sub-task Resolved Jingcheng Du Actions
        54.
        Revert the changes in pom.xml Sub-task Resolved Jingcheng Du Actions
        55.
        Use LimitInputStream in hbase-common instead of ProtobufUtil.LimitedInputStream Sub-task Resolved Jingcheng Du Actions
        56.
        Check the mob files when there are mob-enabled columns in HFileCorruptionChecker Sub-task Resolved Jingcheng Du Actions
        57.
        Do not reset the mvcc for bulk loaded mob reference cells in reading Sub-task Resolved Jingcheng Du Actions
        58.
        Race in multi threaded PartitionedMobCompactor causes NPE Sub-task Resolved Jingcheng Du Actions
        59.
        Wrong mob metrics names in TestRegionServerMetrics Sub-task Resolved Jingcheng Du Actions
        60.
        Return empty value when the mob file is corrupt instead of throwing exceptions Sub-task Resolved Jingcheng Du Actions
        61.
        Do not reset mvcc in compactions for mob-enabled column Sub-task Resolved Jingcheng Du Actions
        62.
        Add mob integrity check in HFilePrettyPrinter Sub-task Resolved Jingcheng Du Actions
        63.
        Not cleaning Mob data when Mob CF is removed from table Sub-task Resolved Pankaj Kumar Actions

          Activity

            People

            • Assignee:
              jingcheng.du@intel.com Jingcheng Du
              Reporter:
              jingcheng.du@intel.com Jingcheng Du

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment