[NIFI-10273] MergeContent tar compression fails on FlowFiles > 8.5GB

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.16.3
• Fix Version/s: 1.19.0
• Component/s: Extensions
• Assignee: Unassigned
• Reporter: Michael Hogue
• Created: 25/Jul/22 09:54
• Updated: 07/Sep/23 19:00
• Resolved: 07/Oct/22 15:41

Description

When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the following error:

 java.lang.RuntimeException: entry size '<FlowFileSize>' is too big (> 8589934591)

This is because apache commons-compress library has a hard maximum :

https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94

077777777777 octal bytes is 8589934591 decimal bytes (~8.5GB).

This can be addressed by configuring a BigNumberMode on the TarArchiveOutputStream:

https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)

Here's an example of another project doing this:

https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files

Issue Links

• fixes: NIFI-1638 Unable to TAR a file that is larger than 8GB
• links to: GitHub Pull Request #6369