[FLINK-14974] Managed memory fractions should be rounded properly to not summed up to be more than 1.0

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.10.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Zhu Zhu
• Reporter: Zhu Zhu
• Created: 27/Nov/19 12:31
• Updated: 29/Nov/19 16:52
• Resolved: 29/Nov/19 16:52

Description

Managed memory fractions should be rounded to floor at a certain precision when divided by the number of operators, otherwise the fractions can be summed up to be more than 1.0 due to the double precision issue, and the last operator may fail to allocate managed memory it is supposed to be able to acquire.

To achieve that, I think we should change StreamingJobGraphGenerator#setManagedMemoryFractionForOperator to use BigDecimal#divide(otherValue, scale, RoundMode.ROUND_DOWN) to calculate the fractions. In this way, the sum of the fractions will not exceed 1.0.
The scale can be a bit larger (maybe 16) so that we only lose little managed memory.

cc chesnay

Issue Links

• links to: GitHub Pull Request #10341