Noticed that the test cluster has lots of "hivemetastore-report.json.tmp (Permission denied)" errors (every 5 seconds).
It is happening because user hive is running as user custom-hive while only user hive has permission for /var/log/hive/ . This location is determined by the config hive.service.metrics.file.location, which is set to /var/log/hive/hivemetastore-report.json . The log4j logs are going into /grid/0/log/hive/.
A fix would be to change value of this config to use whatever logging directory is being used instad of hardcoding to /var/log/hive/. However, we are already publishing the metrics via graphana, so this is not needed.
A quick fix would be to change the config to hive.service.metrics.reporter=HADOOP2 in hive-server2-site and hive-metastore .
The old value is -
hive.service.metrics.reporter=JSON_FILE, JMX, HADOOP2. The suggested change above will remove JMX as well, as we anyway aren't setting any of the required JMX params. Setting JMX reporter is an unnecessary overhead.