[HBASE-22103] HDFS-13209 in Hadoop 3.3.0 breaks asyncwal

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.2.5
• Component/s: hadoop3, wal
• Hadoop Flags: Reviewed
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 25/Mar/19 11:58
• Updated: 21/May/20 09:15
• Resolved: 17/Mar/20 06:46

Description

HDFS-13209 added an additional parameter to DistributedFileSystem.create and broke asyncwal.

2019-03-25 12:19:21,061 ERROR [Listener at localhost/54758] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(562): Couldn't properly initialize access to HDFS internals. Please update your WAL Provider to not make use of the 'asyncfs' provider. See HBASE-16110 for more information.
java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.protocol.ClientProtocol.create(java.lang.String, org.apache.hadoop.fs.permission.FsPermission, java.lang.String, org.apache.hadoop.io.EnumSetWritable, boolean, short, long, [Lorg.apache.hadoop.crypto.CryptoProtocolVersion;)
        at java.lang.Class.getMethod(Class.java:1786)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createFileCreator2(FanOutOneBlockAsyncDFSOutputHelper.java:513)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createFileCreator(FanOutOneBlockAsyncDFSOutputHelper.java:530)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.<clinit>(FanOutOneBlockAsyncDFSOutputHelper.java:557)
        at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
        at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:663)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:126)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:813)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:519)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:460)
        at org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL.newWAL(TestAsyncFSWAL.java:72)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractTestFSWAL.testWALComparator(AbstractTestFSWAL.java:194)

Credit: this bug was found by gabor.bota

Issue Links

• is broken by: HDFS-13209 DistributedFileSystem.create should allow an option to provide StoragePolicy
• is related to: HADOOP-15960 Update guava to 27.0-jre in hadoop-project
• is related to: HBASE-23833 The relocated hadoop-thirdparty protobuf breaks HBase asyncwal
• is related to: HBASE-22953 Supporting Hadoop 3.3.0
• links to: GitHub Pull Request #1284

Sub-Tasks

• Backport HBASE-22103 to branch-2.2: Resolved