[FLINK-22863] ArrayIndexOutOfBoundsException may happen when building rescale edges

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.13.0, 1.13.1
• Fix Version/s: 1.14.0, 1.13.2
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Zhilong Hong
• Reporter: Zhilong Hong
• Created: 03/Jun/21 07:29
• Updated: 23/Sep/21 17:22
• Resolved: 04/Jun/21 09:30

Description

For EdgeManagerBuildUtil introduced in FLINK-21326, we find that during the construction of rescale edges, it may throw ArrayIndexOutOfBoundsException like this:

It is mainly caused by the precision of double in Java.

In EdgeManagerBuildUtil#connectPointwise, when upstream parallelism < downstream parallelism, we calculate the indices of downstream vertices that connect to each upstream partition like this:

int start = (int) (Math.ceil(partitionNum * factor)); 
int end = (int) (Math.ceil((partitionNum + 1) * factor));

The index range is [start, end). 

In some cases the value of end may exceed the downstream parallelism and throw the ArrayIndexOutOfBoundsException.

Let's take an example. The upstream parallelism is 7. The downstream parallelism is 29. For the last upstream partition (which partitionNum is 6), (partitionNum + 1) * factor is 29.00002, which is slightly larger than 29. This is caused by the precision of double. Then end = Math.ceil(29.00002), which is 30. ArrayIndexOutOfBoundsException is thrown here.

To solve this issue, we need to add an extra check for the boundary condition like this:

int end = Math.min(targetCount, (int) (Math.ceil((partitionNum + 1) * factor)));

This affects release-1.13.0 and release-1.13.1.

Issue Links

• links to: GitHub Pull Request #16071