[FLINK-13161] numBuckets calculate wrong in BinaryHashBucketArea

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.9.0
• Component/s: Table SQL / Runtime
• Labels: pull-request-available
• Assignee: Louis Xu
• Reporter: Louis Xu
• Created: 09/Jul/19 04:42
• Updated: 01/Aug/19 10:57
• Resolved: 18/Jul/19 04:24

Description

The original code is:

 

int minNumBuckets = (int) Math.ceil((estimatedRowCount / loadFactor / NUM_ENTRIES_PER_BUCKET));
int bucketNumSegs = Math.max(1, Math.min(maxSegs, (minNumBuckets >>> table.bucketsPerSegmentBits) +
      ((minNumBuckets & table.bucketsPerSegmentMask) == 0 ? 0 : 1)));
int numBuckets = MathUtils.roundDownToPowerOf2(bucketNumSegs << table.bucketsPerSegmentBits);

default value: loadFactor=0.75, NUM_ENTRIES_PER_BUCKET=15，maxSegs = 33(suppose, only need big than the number which calculated by minBunBuckets)

We suppose table.bucketsPerSegmentBits = 3, table.bucketsPerSegmentMask = 0b111. It means buckets in a segment is 8.

When set estimatedRowCount loop from 1 to 1000, we will see the result in attach file.

I will take an example:

estimatedRowCount: 200, minNumBuckets: 18, bucketNumSegs: 3, numBuckets: 16

We can see it request 3 segment, but only 2 segment needed(16 / 8), left one segment wasted. 

And consider the segment is preallocated, it means some segments will never used.

 

 

Issue Links

• links to: GitHub Pull Request #9027