ZooKeeper config:
# The number of milliseconds of each tick
tickTime=30000
# The number of ticks that the initial
# synchronization phase can take
initLimit=5
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=2
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1
HBase config:
hbase.snapshot.enabled
true
hbase.offheapcache.percentage
0.7
hbase.hregion.memstore.mslab.enabled
true
hfile.block.cache.size
0.2
hbase.hregion.memstore.flush.size
134217728
hbase.zookeeper.property.tickTime
6000
hbase.regionserver.global.memstore.size
0.4
zookeeper.session.timeout
20000
hbase.zookeeper.property.tickTime
6000
hbase.hregion.majorcompaction
0
hbase.hregion.max.filesize
107374182400
hbase.regionserver.logroll.period
600000
hbase.regionserver.logroll.errors.tolerated
10
hbase.regionserver.wal.roll.on.sync.ms
600000
Avoid rolling wal on slow sync (bumped up from
default of 10s) because it is unlikely to improve things on a
fixed set of nodes
hbase.regionserver.wal.slowsync.roll.threshold
1000
This value disables log rolls due to count of slow
syncs because it is unlikely to improve things on a fixed set of
nodes.
hbase.regionserver.msginterval
5000
hbase.balancer.period
150000
hbase.lease.recovery.timeout
23000
hbase.lease.recovery.dfs.timeout
11000
needs to be set to dfs.datanode.socket.write.timeout + 1 second
see source code for more details
hbase.hlog.split.skip.errors
true
hbase.master.procedurewalcleaner.ttl
3600000
hbase.regionserver.hlog.tolerable.lowreplication
2
needed to handle hdfs pipeline errors which happen when a node is down
hbase.master.hfilecleaner.ttl
120000
The period (in milliseconds) to keep store files in the archive folder
before deleting them from the file system.
hbase.ipc.server.max.callqueue.length
1024
hbase.regionserver.handler.count
24
hbase.regionserver.metahandler.count
10
hbase.hregion.compacting.memstore.type
NONE
Don't use compacting memstore as in our case we have very little of
deletes/overrides so we mostly experience overhead rather than benefits.
hbase.regionserver.hlog.slowsync.ms
500
dfs.client.read.shortcircuit.skip.checksum
true
HBase does its own checksumming, don't doublecheck.
hbase.split.wal.zk.coordinated
false
Done use ZK to split WALs
HBase env config:
# The maximum amount of heap to use, in MB. Default is 1000.
export HBASE_HEAPSIZE=5100
# let logrotate do rotation of all log files
export HBASE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:MaxDirectMemorySize=512m -XX:+ParallelRefProcEnabled -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -Dlog4j.appender.RFA.MaxBackupIndex=0 -Dlog4j.appender.RFAS.MaxBackupIndex=0"
# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=false
# The default log rolling policy is RFA, where the log file is rolled as per the size defined for the
# RFA appender. Please refer to the log4j.properties file to see more details on this appender.
# In case one needs to do log rolling on a date change, one should set the environment property
# HBASE_ROOT_LOGGER to ",DRFA".
# For example:
# HBASE_ROOT_LOGGER=INFO,DRFA
# The reason for changing default to RFA is to avoid the boundary case of filling out disk space as
# DRFA doesn't put any cap on the log size. Please refer to HBase-5655 for more context.
#shellcheck disable=SC1083
export HBASE_LIBRARY_PATH=/hadoop/lib/native
# Time, in seconds, after which we kill -9 the server if it has not stopped.
# Default 1200 seconds.
export HBASE_STOP_TIMEOUT=250