[HBASE-28459] HFileOutputFormat2 ClassCastException with s3 magic committer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Sravi Kommineni
• Reporter: Bryan Beaudreault
• Created: 27/Mar/24 12:36
• Updated: 07/May/24 13:14
• Resolved: 07/May/24 13:14

Description

In hadoop3 there's the s3 magic committer which can speed up s3 writes dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in committer as a FileOutputCommitter. This causes a class cast exception when the s3 magic committer is enabled:

 

Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 

 

We can cast to PathOutputCommitter instead, but its only available in hadoop3+. So we will need to use reflection to work around this in branch-2.

Issue Links

• links to: GitHub Pull Request #5851
• links to: GitHub Pull Request #5858