[FLINK-17917] ResourceInformationReflector#getExternalResources should ignore the external resource with a value of 0

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Yangze Guo
• Reporter: Yangze Guo
• Created: 25/May/20 06:10
• Updated: 16/Oct/20 10:55
• Resolved: 26/May/20 10:45

Description

Background: In FLINK-17390, we leverage WorkerSpecContainerResourceAdapter.InternalContainerResource to handle container matching logic. In FLINK-17407, we introduce external resources in WorkerSpecContainerResourceAdapter.InternalContainerResource.
On containers returned by Yarn, we try to get the corresponding worker specs by:

• Convert the container to InternalContainerResource
• Get the WorkerResourceSpec from containerResourceToWorkerSpecs map.

Problem: Container mismatch could happen in the below scenario:

• Flink does not allocate any external resources, the externalResources of InternalContainerResource is an empty map.
• The returned container contains all the resources (with a value of 0) defined in Yarn's resource-types.xml. The externalResources of InternalContainerResource has one or more entries with a value of 0.
• These two InternalContainerResource do not match.

To solve this problem, we could ignore all the external resources with a value of 0 in "ResourceInformationReflector#getExternalResources".

cc trohrmann Could you assign this to me?

Issue Links

• links to: GitHub Pull Request #12315