[AMBARI-25936] Ambari metrics collector fails with IncompatibleClassChangeError, Found HdfsFileStatus but expected FanOutOneBlockAsyncDFSOutputHelper

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: ambari-metrics
• Assignee: Mohammad Arshad
• Reporter: Mohammad Arshad
• Created: 15/May/23 06:18
• Updated: 15/May/23 15:11
• Resolved: 15/May/23 15:11

Description

Built ambari trunk, ambari metrics master. Using bitop 3.2.0
installed hadoop cluster using ambari,
ambari metrics is failing with following errors

2023-05-15 06:00:35,374 INFO  [master/Host1:61300:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL Host1%2C61300%2C1684130431341:(num 1684130435340)
2023-05-15 06:00:35,376 ERROR [master/Host1:61300:becomeActiveMaster] master.HMaster: Failed to become active master
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:535)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:112)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:615)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:610)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:623)
        at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:53)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:190)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160)
        at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:723)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:129)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:833)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:547)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:488)
        at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:160)
        at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
        at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:295)
        at org.apache.hadoop.hbase.master.region.MasterRegion.createWAL(MasterRegion.java:200)
        at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:255)
        at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:336)
        at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:846)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2175)
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:520)
        at java.lang.Thread.run(Thread.java:750)
2023-05-15 06:00:35,378 ERROR [master/Host1:61300:becomeActiveMaster] master.HMaster: ***** ABORTING master Host1,61300,1684130431341: Unhandled exception. Starting shutdown. *****
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:535)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:112)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:615)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$8.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:610)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)