[PARQUET-585] Slowly ramp up sizes of int[]s in IntList to keep sizes small when data sets are small

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.9.0, 1.8.2
• Assignee: Piyush Narang
• Reporter: Piyush Narang
• Created: 18/Apr/16 23:18
• Updated: 23/Jun/24 03:28
• Resolved: 21/Apr/16 03:49

Description

One of the items that I listed in jira: https://issues.apache.org/jira/browse/PARQUET-580 was to ensure that we slowly ramp up the size of the int[] we create in IntList instead of 64K arrays directly. This ensures that we don't allocate large 64K arrays at the start(which is potentially wasteful if there are only a few hundred bytes).

So rather than create arrays of size 64K at a time, we could create say a 4K int[], then when it fills up an 8K[] and so on till we reach 64K (at which point the behavior is the same as the current implementation).

Issue Links

• links to: PR #341