Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21256

Improve IntegrationTestBigLinkedList for testing huge data

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Recently, I use ITBLL to test some features in our company. I have encountered the following problems:
       
      1. Generator is too slow at the generating stage, the root cause is SecureRandom. There is a global lock in SecureRandom( See the following picture). I use Random instead of SecureRandom, and it could speed up this stage(500% up with 20 mapper).  SecureRandom was brought by HBASE-13382, but speaking of generating random bytes, in my opnion,
      it is the same with Random.

      2. VerifyReducer have a cpu cost of 14% on format method. This is cause by create keyString variable. However, keyString may never be used if test result is correct.(and that's in most cases). Just delay creating keyString can yield huge performance boost in verifing stage.

      3.Arguments check is needed, because there's constraint between arguments. If we broken this constraint, we can not get a correct circular list. 
       
      4.Let big family value size could be configured.

       

      5.Avoid starting RS at backup master

        Attachments

        1. ITBLL-2.png
          520 kB
          Zephyr Guo
        2. ITBLL-1.png
          351 kB
          Zephyr Guo
        3. HBASE-21256-v4.patch
          9 kB
          Zephyr Guo
        4. HBASE-21256-v3.patch
          10 kB
          Zephyr Guo
        5. HBASE-21256-v2.patch
          8 kB
          Zephyr Guo
        6. HBASE-21256-v12.patch
          14 kB
          Zephyr Guo
        7. HBASE-21256-v11.patch
          14 kB
          Zephyr Guo
        8. HBASE-21256-v10.patch
          14 kB
          Zephyr Guo
        9. HBASE-21256-v1.patch
          7 kB
          Zephyr Guo
        10. HBASE-21256-branch-1.patch
          16 kB
          Andrew Kyle Purtell

          Issue Links

            Activity

              People

              • Assignee:
                gzh1992n Zephyr Guo
                Reporter:
                gzh1992n Zephyr Guo
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: