[YARN-11369] Commons.compress throws an IllegalArgumentException with large uids after 1.21

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: client
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Benjamin Teke
• Reporter: Benjamin Teke
• Created: 07/Nov/22 16:42
• Updated: 28/Jan/24 02:07
• Resolved: 16/Nov/22 12:07

Description

Encountering COMPRESS-587 with large uids/gids in hadoop-mapreduce-client-uploader/src/main/java/org/apache/hadoop/mapred/uploader/FrameworkUploader.java:

22/09/13 06:39:05 INFO uploader.FrameworkUploader: Adding /cs/cloudera/opt/cloudera/cm/lib/plugins/event-publish-7.7.1-shaded.jar
Exception in thread "main" java.lang.IllegalArgumentException: group id '5049047' is too big ( > 2097151 ). Use STAR or POSIX extensions to overcome this limit

A workaround is to specifically set bignumber mode to BIGNUMBER_POSIX or BIGNUMBER_STAR on the instance.

Issue Links

• links to: GitHub Pull Request #5116