ZooKeeper config: # The number of milliseconds of each tick tickTime=30000 # The number of ticks that the initial # synchronization phase can take initLimit=5 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=2 # the maximum number of client connections. # increase this if you need to handle more clients maxClientCnxns=60 # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature autopurge.purgeInterval=1 HBase config: hbase.snapshot.enabled true hbase.offheapcache.percentage 0.7 hbase.hregion.memstore.mslab.enabled true hfile.block.cache.size 0.2 hbase.hregion.memstore.flush.size 134217728 hbase.zookeeper.property.tickTime 6000 hbase.regionserver.global.memstore.size 0.4 zookeeper.session.timeout 20000 hbase.zookeeper.property.tickTime 6000 hbase.hregion.majorcompaction 0 hbase.hregion.max.filesize 107374182400 hbase.regionserver.logroll.period 600000 hbase.regionserver.logroll.errors.tolerated 10 hbase.regionserver.wal.roll.on.sync.ms 600000 Avoid rolling wal on slow sync (bumped up from default of 10s) because it is unlikely to improve things on a fixed set of nodes hbase.regionserver.wal.slowsync.roll.threshold 1000 This value disables log rolls due to count of slow syncs because it is unlikely to improve things on a fixed set of nodes. hbase.regionserver.msginterval 5000 hbase.balancer.period 150000 hbase.lease.recovery.timeout 23000 hbase.lease.recovery.dfs.timeout 11000 needs to be set to dfs.datanode.socket.write.timeout + 1 second see source code for more details hbase.hlog.split.skip.errors true hbase.master.procedurewalcleaner.ttl 3600000 hbase.regionserver.hlog.tolerable.lowreplication 2 needed to handle hdfs pipeline errors which happen when a node is down hbase.master.hfilecleaner.ttl 120000 The period (in milliseconds) to keep store files in the archive folder before deleting them from the file system. hbase.ipc.server.max.callqueue.length 1024 hbase.regionserver.handler.count 24 hbase.regionserver.metahandler.count 10 hbase.hregion.compacting.memstore.type NONE Don't use compacting memstore as in our case we have very little of deletes/overrides so we mostly experience overhead rather than benefits. hbase.regionserver.hlog.slowsync.ms 500 dfs.client.read.shortcircuit.skip.checksum true HBase does its own checksumming, don't doublecheck. hbase.split.wal.zk.coordinated false Done use ZK to split WALs HBase env config: # The maximum amount of heap to use, in MB. Default is 1000. export HBASE_HEAPSIZE=5100 # let logrotate do rotation of all log files export HBASE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:MaxDirectMemorySize=512m -XX:+ParallelRefProcEnabled -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -Dlog4j.appender.RFA.MaxBackupIndex=0 -Dlog4j.appender.RFAS.MaxBackupIndex=0" # Tell HBase whether it should manage it's own instance of Zookeeper or not. export HBASE_MANAGES_ZK=false # The default log rolling policy is RFA, where the log file is rolled as per the size defined for the # RFA appender. Please refer to the log4j.properties file to see more details on this appender. # In case one needs to do log rolling on a date change, one should set the environment property # HBASE_ROOT_LOGGER to ",DRFA". # For example: # HBASE_ROOT_LOGGER=INFO,DRFA # The reason for changing default to RFA is to avoid the boundary case of filling out disk space as # DRFA doesn't put any cap on the log size. Please refer to HBase-5655 for more context. #shellcheck disable=SC1083 export HBASE_LIBRARY_PATH=/hadoop/lib/native # Time, in seconds, after which we kill -9 the server if it has not stopped. # Default 1200 seconds. export HBASE_STOP_TIMEOUT=250