[HBASE-28721] AsyncFSWAL is broken when running against hadoop 3.4.0

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: hadoop3, wal
• Labels: pull-request-available
• Assignee: Istvan Toth
• Reporter: Duo Zhang
• Created: 10/Jul/24 02:20
• Updated: 20/Sep/24 10:29

Description

2024-07-10T10:09:33,161 ERROR [master/localhost:0:becomeActiveMaster {}] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(258): Couldn't properly initialize access to HDFS internals. Please update your WAL Provider to not make use of the 'asyncfs' provider. See HBASE-16110 for more information.
java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.beginFileLease(long,org.apache.hadoop.hdfs.DFSOutputStream)
        at java.lang.Class.getDeclaredMethod(Class.java:2675) ~[?:?]
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createLeaseManager(FanOutOneBlockAsyncDFSOutputHelper.java:175) ~[hbase-asyncfs-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.<clinit>(FanOutOneBlockAsyncDFSOutputHelper.java:252) ~[hbase-asyncfs-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:375) ~[?:?]
        at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:149) ~[classes/:?]
        at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:174) ~[classes/:?]
        at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:262) ~[classes/:?]
        at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:231) ~[classes/:?]
        at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:383) ~[classes/:?]
        at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:135) ~[classes/:?]
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1003) ~[classes/:?]
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2524) ~[classes/:?]
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:613) ~[classes/:?]
        at org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155) ~[hbase-common-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]

Issue Links

• is depended upon by: HBASE-28846 Change the default Hadoop3 version to 3.4.0, and add tests to make sure HBase works with earlier supported Hadoop versions
• is related to: HBASE-28714 Hadoop check for hadoop 3.4.0 is failing
• links to: GitHub Pull Request #6269
• links to: GitHub Pull Request #6270