Running hbck --repair or LoadIncrementalHFiles when BucketCache is enabled in offheap mode can cause OOM. This is apparently because bin/hbase does not include $HBASE_REGIONSERVER_OPTS for these tools. This results in HRegion or HFileReaders initialized with a CacheConfig that doesn't have the necessary Direct Memory.
Possible solutions include:
- disable blockcache in the config used by hbck when running its repairs
- include HBASE_REGIONSERVER_OPTS in the HBaseFSCK startup arguments
I'm leaning toward the former because it's possible that hbck is run on a host with different hardware profile as the RS.