[AMBARI-16121] HBase RegionServers go down after Ambari upgrade due to ClassCastException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.2
• Fix Version/s: 2.2.2
• Component/s: ambari-server
• Assignee: Dmitry Lysnichenko
• Reporter: Dmitry Lysnichenko
• Created: 26/Apr/16 13:51
• Updated: 27/Apr/16 17:19
• Resolved: 27/Apr/16 17:19

Description

ambari-server --hash
2b112376b631384852a6c8aaa2e102d8dd39a9f1
ambari-server-2.2.2.0-456.x86_64

Steps

1. Deploy HDP 2.3.4.0 cluster with Ambari 2.2.0.0 (HA, unsecure cluster and Ranger enabled)
2. Upgrade Ambari to 2.2.2.0

Observed that all HBase RS reported as down

Logs show below error:

016-04-26 09:47:23,575 ERROR [regionserver/host1/ip:16020] coprocessor.CoprocessorHost: The coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$RegionServerEnvironment cannot be cast to org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$RegionServerEnvironment cannot be cast to org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
at org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:131)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:411)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:253)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:156)
at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.<init>(RegionServerCoprocessorHost.java:69)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:888)
at java.lang.Thread.run(Thread.java:745)
2016-04-26 09:47:23,584 FATAL [regionserver/host1/ip:16020] regionserver.HRegionServer: ABORTING region server host1,16020,1461664036377: The coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$RegionServerEnvironment cannot be cast to org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$RegionServerEnvironment cannot be cast to org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
at org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:131)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:411)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:253)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:156)
at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.<init>(RegionServerCoprocessorHost.java:69)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:888)
at java.lang.Thread.run(Thread.java:745)
2016-04-26 09:47:23,586 FATAL [regionserver/host1/ip:16020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.token.TokenProvider]
2016-04-26 09:47:27,474 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2651)
2016-04-26 09:47:27,491 INFO  [Thread-129] provider.AuditProviderFactory: ==> JVMShutdownHook.run()
2016-04-26 09:47:27,492 INFO  [Thread-129] queue.AuditAsyncQueue: Stop called. name=hbaseRegional.async
2016-04-26 09:47:27,492 INFO  [Thread-129] queue.AuditAsyncQueue: Interrupting consumerThread. name=hbaseRegional.async, consumer=hbaseRegional.async.summary
2016-04-26 09:47:27,496 INFO  [Thread-129] provider.AuditProviderFactory: <== JVMShutdownHook.run()
2016-04-26 09:47:27,507 INFO  [Thread-7] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@16423501
2016-04-26 09:47:27,506 INFO  [org.apache.ranger.audit.queue.AuditAsyncQueue0] queue.AuditAsyncQueue: Caught exception in consumer thread. Shutdown might be in progress

Issue Links

• links to: review_request_