[FLUME-2662] Upgrade to Commons-IO 2.4

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.5.1
• Component/s: Build
• Labels: Dependencies
• Assignee: Unassigned
• Reporter: Roshan Naik
• Created: 08/Apr/15 21:50
• Updated: 21/Mar/22 05:01
• Resolved: 21/Mar/22 05:01

Description

Hadoop 2.7 is now switching to apache-commons-io v2.4. Hbase 1.0 is also using commons-io v2.4.
Flume is currently at 2.1.
Flume runs into issues like this when tests run against them:

testSequenceFile(org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem) Time elapsed: 77 sec <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:854)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:582)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:98)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:78)
at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:69)
at org.apache.flume.sink.hdfs.TestUseRawLocalFileSystem.testSequenceFile(TestUseRawLocalFileSystem.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

I am planning to submit a patch to upgrade commons-io to 2.4. Just wanted to be cautious and check if we have witnessed has been any issue in the past when upgrading apache commons libraries.

Based on what I see here:
http://commons.apache.org/proper/commons-io/upgradeto2_4.html and

http://commons.apache.org/proper/commons-io/upgradeto2_2.html

Commons-IO 2.4 is binary compat with 2.2 which is in turn binary compat
with 2.1.

There is what they call a "rare" case of source incompat as described in
https://issues.apache.org/jira/browse/IO-318

Doesnt look like we are affected.

Issue Links

• duplicates: FLUME-3400 Upgrade commons-io to 2.11.0
• is duplicated by: FLUME-2717 Add commons-io dependency into hadoop-2 profile to enable Flume 1.5 to support Hadoop 2.7
• is related to: BIGTOP-2534 Flume tests fail with java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets