HBase
  1. HBase
  2. HBASE-2706

Use heuristics to adjust various parameters based on load characteristics

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.20.4
    • Fix Version/s: None
    • Component/s: master
    • Labels:
      None

      Description

      Refer to 'ideas to improve throughput of the base writing' discussion on hbase user mailing list.

      We should use heuristics to adjust various parameters based on load characteristics. Factors include, but not limited to, number of CPU (cores) on each node, amount of physical memory on each node, Heap space allocated to master/region servers.
      HBase parameters shown below would be dynamic whose current values can be seen on master.jsp / regionserver.jsp

      For JVM parameters such as -Xmx, master.jsp should display recommended value to user.

      -------------
      Thanks. I will make this change:

      <property>
      <name>hbase.hregion.memstore.block.multiplier</name>
      <value>8</value>
      </property>

      <property>
      <name>hbase.regionserver.msginterval</name>
      <value>10000</value>
      </property>

      <property>
      <name>hbase.hstore.compactionThreshold</name>
      <value>6</value>
      </property>

      <property>
      <name>hbase.hstore.blockingStoreFiles</name>
      <value>18</value>
      </property>

      and see how it goes.

      Jimmy.

      With the above change, the write goes faster if it works, but now we have problem that the
      insertion doesn't work at all, here is the result:

      2010-06-09 22:29:04,330 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io
      .IOException: Could not find target position 0

        Issue Links

          Activity

          Ted Yu created issue -
          Hide
          stack added a comment -

          Moving to 0.90 out of 0.20.6 (not crittical bug fix)

          Show
          stack added a comment - Moving to 0.90 out of 0.20.6 (not crittical bug fix)
          stack made changes -
          Field Original Value New Value
          Fix Version/s 0.90.0 [ 12313607 ]
          Fix Version/s 0.20.6 [ 12315060 ]
          Hide
          stack added a comment -

          Another to adjust is sizes of caches.

          Show
          stack added a comment - Another to adjust is sizes of caches.
          Hide
          Jonathan Gray added a comment -

          Punting to 0.92

          Show
          Jonathan Gray added a comment - Punting to 0.92
          Jonathan Gray made changes -
          Fix Version/s 0.92.0 [ 12314223 ]
          Fix Version/s 0.90.0 [ 12313607 ]
          Hide
          stack added a comment -

          Moving out of 0.92.0. Pull it back in if you think different.

          Show
          stack added a comment - Moving out of 0.92.0. Pull it back in if you think different.
          stack made changes -
          Fix Version/s 0.92.0 [ 12314223 ]
          Todd Lipcon made changes -
          Link This issue relates to HBASE-5349 [ HBASE-5349 ]
          Hide
          stack added a comment -

          Resolving as no longer valid. Anoop work in this area has been committed.

          Show
          stack added a comment - Resolving as no longer valid. Anoop work in this area has been committed.
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Invalid [ 6 ]
          Ted Yu made changes -
          Resolution Invalid [ 6 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Ted Yu made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development