[HUDI-3605] hbase dependency need relocation or use other version for hive3

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Invalid
• Fix Version/s: 0.11.0
• Component/s: dependencies
• Story Points: 0
• Assignee: Ethan Guo
• Reporter: rex xiong

Reviewers

Ethan Guo, Raymond Xu

• Created: 11/Mar/22 03:11
• Updated: 22/Mar/22 03:40
• Resolved: 22/Mar/22 03:40

Description

Hive 3 now uses hbase version 2.0.0-alpha4，which has conflict with the hbase version of hudi. See the problem as below.

hive> select * from hudi_mor_part_table_spark3_0110_0309_ro;
OK
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.io.hfile.HFile
.createReader(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hado
op/fs/Path;Lorg/apache/hadoop/hbase/io/FSDataInputStreamWrapper;JLorg/apache/hadoop/hbase/io/hfile/CacheConfig;Lorg/apache/hadoop/conf/Configuration;)Lorg/ap
ache/hadoop/hbase/io/hfile/HFile$Reader;
        at org.apache.hudi.io.storage.HoodieHFileReader.<init>(HoodieHFileReader.java:98)
        at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.deserializeRecords(HoodieHFileDataBlock.java:158)
        at org.apache.hudi.common.table.log.block.HoodieDataBlock.readRecordsFromBlockPayload(HoodieDataBlock.java:167)
        at org.apache.hudi.common.table.log.block.HoodieDataBlock.getRecordItr(HoodieDataBlock.java:125)
        at org.apache.hudi.common.table.log.block.HoodieDataBlock.getRecordItr(HoodieDataBlock.java:151)
        at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processDataBlock(AbstractHoodieLogRecordReader.java:363)
        at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processQueuedBlocksForInstant(AbstractHoodieLogRecordReader.java:427)
        at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:242)
        at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:181)
        at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:101)
        at org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader.<init>(HoodieMetadataMergedLogRecordReader.java:71)
        at org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader.<init>(HoodieMetadataMergedLogRecordReader.java:51)
        at org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader$Builder.build(HoodieMetadataMergedLogRecordReader.java:246)
        at org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:379)
        at org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$openReadersIfNeeded$4(HoodieBackedTableMetadata.java:295)
        at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
        at org.apache.hudi.metadata.HoodieBackedTableMetadata.openReadersIfNeeded(H
oodieBackedTableMetadata.java:283)
        at org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getRecordsByKeys$0(HoodieBackedTableMetadata.java:139)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:138)
        at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:128)
        at org.apache.hudi.metadata.BaseTableMetadata.fetchAllPartitionPaths(BaseTableMetadata.java:274)

Issue Links

• relates to: HUDI-1180 Upgrade HBase to 2.x