HBase
  1. HBase
  2. HBASE-3499

Users upgrading to 0.90.0 need to have their .META. table updated with the right MEMSTORE_SIZE

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.90.0
    • Fix Version/s: 0.90.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      With Jack Levin, we were able to figure that users that are upgrading from a 0.20.x era cluster have their .META. schema set with a 16KB MEMSTORE_SIZE. This was done in order to minimize lost meta rows when append wasn't available but even if we changed it in HTD, we also have to make sure all users upgrading to 0.90 have it changed too.

      In Jack's case, he ended up with 2143 storefiles in .META. during a cold start, slowing everything down. He reported a few times in the past that his .META. was always extremely busy.

      We should be able to do it as a one-off thing in HMaster when opening .META. (an update in place).

      1. set_meta_memstore_size.rb
        3 kB
        Jean-Daniel Cryans
      2. Chapter 2. Upgrading.html
        6 kB
        stack

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #1739 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1739/)

        Show
        Hudson added a comment - Integrated in HBase-TRUNK #1739 (See https://hudson.apache.org/hudson/job/HBase-TRUNK/1739/ )
        Hide
        stack added a comment -

        Committed script and a chapter describing it in a new 'Upgrading' chapter. Added to branch and trunk.

        Show
        stack added a comment - Committed script and a chapter describing it in a new 'Upgrading' chapter. Added to branch and trunk.
        Hide
        stack added a comment -

        Here is new Chapter on Upgrading that I added that makes mention of this issue and the fixup script.

        Show
        stack added a comment - Here is new Chapter on Upgrading that I added that makes mention of this issue and the fixup script.
        Hide
        ryan rawson added a comment -

        If we reset the size of that every time we start, wouldn't that prevent
        people from customizing that size then?

        Show
        ryan rawson added a comment - If we reset the size of that every time we start, wouldn't that prevent people from customizing that size then?
        Hide
        Jean-Daniel Cryans added a comment -

        Here is the script that I wrote for Jack that fixed his issue. It's totally based on set_meta_block_caching.rb

        In a way I guess we could just add this script, point to it in the documentation and call it a day. It's just less friendly because you have to run it and restart.

        Show
        Jean-Daniel Cryans added a comment - Here is the script that I wrote for Jack that fixed his issue. It's totally based on set_meta_block_caching.rb In a way I guess we could just add this script, point to it in the documentation and call it a day. It's just less friendly because you have to run it and restart.
        Hide
        Jean-Daniel Cryans added a comment -

        Yeah bootstrap is an overloaded word, wrong choice.

        What I meant is that when we open ROOT, check .META.'s HRI and set MEMSTORE_SIZE (not the blocksize like you wrote) and set it back to 64MB.

        I think we could downgrade this if we add in the release notes that the user upgrading from 0.20 needs to fix it himself... oh I thought I attached the script I made for Jack but it seems I forgot. Doing it right now.

        Show
        Jean-Daniel Cryans added a comment - Yeah bootstrap is an overloaded word, wrong choice. What I meant is that when we open ROOT , check .META.'s HRI and set MEMSTORE_SIZE (not the blocksize like you wrote) and set it back to 64MB. I think we could downgrade this if we add in the release notes that the user upgrading from 0.20 needs to fix it himself... oh I thought I attached the script I made for Jack but it seems I forgot. Doing it right now.
        Hide
        stack added a comment -

        @JD You thinking that we'd open the region if NOT bootstrap and check whats in ROOT. If 16M for BLOCKSIZE, edit and write it back out again. An extra open of the ROOT by the Master. Or you thinking something dumber where as soon as ROOT is deployed, we go a get on .META. schema and if 16M, edit it.

        You think this a blocker JD?

        Show
        stack added a comment - @JD You thinking that we'd open the region if NOT bootstrap and check whats in ROOT . If 16M for BLOCKSIZE, edit and write it back out again. An extra open of the ROOT by the Master. Or you thinking something dumber where as soon as ROOT is deployed, we go a get on .META. schema and if 16M, edit it. You think this a blocker JD?
        Hide
        Jean-Daniel Cryans added a comment -

        I thought the master could do it before that in the bootstrap...

        Show
        Jean-Daniel Cryans added a comment - I thought the master could do it before that in the bootstrap...
        Hide
        Todd Lipcon added a comment -

        Should we automatically alter .META. on open as well?

        Show
        Todd Lipcon added a comment - Should we automatically alter .META. on open as well?
        Hide
        stack added a comment -

        I added migration notes for 0.90.x over here: http://wiki.apache.org/hadoop/Hbase/HowToMigrate#90

        Show
        stack added a comment - I added migration notes for 0.90.x over here: http://wiki.apache.org/hadoop/Hbase/HowToMigrate#90

          People

          • Assignee:
            Unassigned
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development