HBase
  1. HBase
  2. HBASE-9472

If the memstore size is under .1 or greater than .9 the memstore size defaults to the default memstore size

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.94.5, 0.99.0
    • Fix Version/s: None
    • Component/s: BlockCache
    • Labels:
      None

      Description

      In HbaseConfiguration.checkForClusterFreeMemoryLimit it does a check to see if the blockCache + memstore > .8 this threshold ensures we do not run out of memory.

      But MemStoreFlusher.getMemStoreLimit does this check:

      if (limit >= 0.9f || limit < 0.1f) {
            LOG.warn("Setting global memstore limit to default of " + defaultLimit +
              " because supplied value outside allowed range of 0.1 -> 0.9");
            effectiveLimit = defaultLimit;
          }
      

      In our cluster we had the block cache set to an upper limit of 0.76 and the memstore upper limit was set to 0.04. We noticed the memstore size was exceeding the limit we had set and after looking at the getMemStoreLimit code it seems that the memstore upper limit is sized to the default value if the configuration value is less than .1 or greater than .9. This now makes the block cache and memstore greater than our available heap.

      We can remove the check for the greater than 90% of the heap as this can never happen due to the check in HbaseConfiguration.checkForClusterFreeMemoryLimit()

      This check doesn't seem necessary anymore as we have the HbaseConfiguration class checking for the cluster free limit. Am I correct in this assumption?

        Activity

        churro morales created issue -
        Hide
        Vladimir Rodionov added a comment -

        Just a note. Memstore size of 0.04 is not a good idea. You will end up creating a lot of tiny flush files and compaction will be running around the clock.

        Show
        Vladimir Rodionov added a comment - Just a note. Memstore size of 0.04 is not a good idea. You will end up creating a lot of tiny flush files and compaction will be running around the clock.
        Hide
        Anoop Sam John added a comment -

        Are you doing bulk load of data and that is why you almost dont need the memstore?

        Show
        Anoop Sam John added a comment - Are you doing bulk load of data and that is why you almost dont need the memstore?
        Hide
        churro morales added a comment -

        We have a 20GB heap for these regionservers which are used for a cluster which is primarily doing random reads. We are trying to hold everything in the block cache. We don't do a lot of writes in comparison to reads in the steady state. When we ran a job to write quite a few entries to this cluster we noticed the memstore was greater than the expected size: of .8GB.

        With respect to the configuration, our expected block cache upper limit is ~ 15GB and our memstore is ~ .8GB

        If the memstore is not allowed to be less than .1, we use the default value which is .4, combined our blockcache + memstore upper limits are: 23GB which is larger than our total heap of 20GB.

        Show
        churro morales added a comment - We have a 20GB heap for these regionservers which are used for a cluster which is primarily doing random reads. We are trying to hold everything in the block cache. We don't do a lot of writes in comparison to reads in the steady state. When we ran a job to write quite a few entries to this cluster we noticed the memstore was greater than the expected size: of .8GB. With respect to the configuration, our expected block cache upper limit is ~ 15GB and our memstore is ~ .8GB If the memstore is not allowed to be less than .1, we use the default value which is .4, combined our blockcache + memstore upper limits are: 23GB which is larger than our total heap of 20GB.
        Hide
        churro morales added a comment -

        For regionservers with large heaps ~ 20GB, it might not make sense to have a restriction where the memstore size has to be a certain percentage of the heap size. Instead the logic used is the memstore has to be either 100MB or 10% of the heap.

        Show
        churro morales added a comment - For regionservers with large heaps ~ 20GB, it might not make sense to have a restriction where the memstore size has to be a certain percentage of the heap size. Instead the logic used is the memstore has to be either 100MB or 10% of the heap.
        churro morales made changes -
        Field Original Value New Value
        Attachment HBASE-9742-0.94.patch [ 12603916 ]
        Hide
        churro morales added a comment -

        If you guys feel this logic is appropriate I can provide a patch for trunk.

        Show
        churro morales added a comment - If you guys feel this logic is appropriate I can provide a patch for trunk.
        Hide
        stack added a comment -

        churro morales Thanks for having a go at fixing this bug of ours. Doing config check in HBaseConfiguration is good enough I'd say. No need of the dup check – especially if it is in disagreement w/ the HBC check. I think the memory math probably better belongs in our memory util classes ratther than in HBC. Trunk patch would be great.

        Show
        stack added a comment - churro morales Thanks for having a go at fixing this bug of ours. Doing config check in HBaseConfiguration is good enough I'd say. No need of the dup check – especially if it is in disagreement w/ the HBC check. I think the memory math probably better belongs in our memory util classes ratther than in HBC. Trunk patch would be great.
        stack made changes -
        Affects Version/s 0.99.0 [ 12325675 ]
        Component/s BlockCache [ 12323302 ]
        Hide
        churro morales added a comment -

        stack are you comfortable with the logic for memstore sizing? If yes, I'll get some patches ready for you folks

        Show
        churro morales added a comment - stack are you comfortable with the logic for memstore sizing? If yes, I'll get some patches ready for you folks

          People

          • Assignee:
            Unassigned
            Reporter:
            churro morales
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development