[HBASE-25900] HBoss tests compile/failure against Hadoop 3.3.1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: hbase-filesystem-1.0.0-alpha1
• Fix Version/s: hbase-filesystem-1.0.0-alpha2
• Component/s: Filesystem Integration
• Hadoop Flags: Reviewed
• Assignee: Josh Elser
• Reporter: Steve Loughran
• Created: 19/May/21 13:50
• Updated: 05/Oct/21 16:52
• Resolved: 05/Oct/21 16:52

Description

Changes in Hadoop 3.3.x stop the tests compiling/working.

• changes in signature of nominally private classes (HADOOP-17497): fix, update
• HADOOP-16721 -s3a rename throwing more exceptions, but no longer failing if the dest parent doesn't exist. Fix: change s3a.xml
• HADOOP-17531/HADOOP-17620 distcp moving to listIterator; test failures.
• HADOOP-13327: tests on syncable which expect files being written to to be visible. Fix: skip that test

The fix for HADOOP-17497 stops this compiling against Hadoop < 3.3.1. This is unfortunate but I can't see an easy fix. The new signature takes a parameters class, so we can (and already are) adding new config options without breaking this signature again. And I've tagged it as LimitedPrivate so that future developers will know it's used here

Issue Links

• is caused by: HADOOP-16721 Improve S3A rename resilience
• is caused by: HADOOP-13327 Add OutputStream + Syncable to the Filesystem Specification
• is caused by: HADOOP-17497 refactoring the signature of S3ClientFactory breaks hboss compile
• is caused by: HADOOP-17531 DistCp: Reduce memory usage on copying huge directories
• is caused by: HADOOP-17620 DistCp: Use Iterator for listing target directory as well
• relates to: HBASE-26164 Update dependencies in hbase-filesystem
• links to: GitHub Pull Request #23
• links to: GitHub Pull Request #25