[HBASE-9498] NPE in HBaseAdmin if master not running

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.98.0, 0.96.0
• Component/s: Admin
• Hadoop Flags: Reviewed
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 10/Sep/13 23:52
• Updated: 20/Nov/15 11:52
• Resolved: 11/Sep/13 15:27

Description

It is caused because master is not up. The prepare fails:

2:47:46.573 PM 	ERROR 	com.cloudera.cmon.firehose.AbstractHBasePoller 	

Error polling HBASE-1, error: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStub(HConnectionManager.java:1641)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterMonitorServiceStubMaker.makeStub(HConnectionManager.java:1667)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveMasterMonitorService(HConnectionManager.java:2152)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.listTables(HConnectionManager.java:2629)
	at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:290)
	at com.cloudera.cmf.cdh5client.hbase.HBaseAdminImpl.listTables(HBaseAdminImpl.java:60)
	at com.cloudera.cmon.firehose.HBaseRegionHealthCanaryPoller$CanaryPollerRunnable.run(HBaseRegionHealthCanaryPoller.java:213)
	at com.cloudera.cmon.firehose.HBaseRegionHealthCanaryPoller$CanaryPollerRunnable.run(HBaseRegionHealthCanaryPoller.java:177)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
	at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:29)
	at com.cloudera.cmon.firehose.HBaseRegionHealthCanaryPoller.performPoll(HBaseRegionHealthCanaryPoller.java:116)
	at com.cloudera.cmon.firehose.AbstractHBasePoller.handleHbaseService(AbstractHBasePoller.java:339)
	at com.cloudera.cmon.firehose.AbstractHBasePoller.runWithTracking(AbstractHBasePoller.java:215)
	at com.cloudera.cmon.firehose.AbstractHBasePoller.run(AbstractHBasePoller.java:140)
	at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.doWork(PeriodicEnterpriseService.java:116)
	at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.run(PeriodicEnterpriseService.java:65)
	at com.cloudera.enterprise.AbstractCDHVersionAwarePeriodicService$4.run(AbstractCDHVersionAwarePeriodicService.java:116)
	at com.cloudera.cmf.cdh5client.CDH5TaskRunner.run(CDH5TaskRunner.java:45)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
	at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:108)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(HConnectionManager.java:1567)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStub(HConnectionManager.java:1612)
	... 20 more

Then when done we do the call:

2:47:51.587 PM 	ERROR 	com.cloudera.cmon.firehose.HBasePoller 	

Encountered exception null
java.lang.NullPointerException
	at org.apache.hadoop.hbase.client.HBaseAdmin$MasterMonitorCallable.close(HBaseAdmin.java:3053)
	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3089)
	at org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2081)
	at com.cloudera.cmf.cdh5client.hbase.HConnectionImpl.getClusterStatus(HConnectionImpl.java:69)
	at com.cloudera.cmon.firehose.HbaseServicePolledStatus.update(HbaseServicePolledStatus.java:137)
	at com.cloudera.cmon.firehose.HBasePoller$1.run(HBasePoller.java:95)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
	at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:29)
	at com.cloudera.cmon.firehose.HBasePoller.performPoll(HBasePoller.java:84)
	at com.cloudera.cmon.firehose.AbstractHBasePoller.handleHbaseService(AbstractHBasePoller.java:339)
	at com.cloudera.cmon.firehose.AbstractHBasePoller.runWithTracking(AbstractHBasePoller.java:215)
	at com.cloudera.cmon.firehose.AbstractHBasePoller.run(AbstractHBasePoller.java:140)
	at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.doWork(PeriodicEnterpriseService.java:116)
	at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.run(PeriodicEnterpriseService.java:65)
	at com.cloudera.enterprise.AbstractCDHVersionAwarePeriodicService$4.run(AbstractCDHVersionAwarePeriodicService.java:116)
	at com.cloudera.cmf.cdh5client.CDH5TaskRunner.run(CDH5TaskRunner.java:45)
	at java.lang.Thread.run(Thread.java:724)

Let me fix. Let me make sure there aren't other prepare/close's that have this issue while at it.

Issue Links

• is duplicated by: HBASE-9545 NPE when trying to get cluster status on an hbase cluster that isn't there