[HADOOP-17905] Modify Text.ensureCapacity() to efficiently max out the backing array size

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: io
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 09/Sep/21 11:06
• Updated: 11/Feb/24 09:39
• Resolved: 30/Sep/21 00:26

Description

This is a continuation of HADOOP-17901.

Right now we use a factor of 1.5x to increase the byte array if it's full. However, if the size reaches a certain point, the increment is only (current size + length). This can cause performance issues if the textual data which we intend to store is beyond this point.

Instead, let's max out the array to the maximum. Based on different sources, a safe choice seems to be Integer.MAX_VALUE - 8 (see ArrayList, AbstractCollection, HashTable, etc).

Issue Links

• causes: HIVE-25686 UDFSpace result length calculation is incorrect after HADOOP-17901 and HADOOP-17905
• Dependent: HADOOP-17901 Performance degradation in Text.append() after HADOOP-16951
• links to: GitHub Pull Request #3423