[BEAM-4041] Performance tests fail due to kubernetes load balancer problems

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: testing
• Assignee: Unassigned
• Reporter: Lukasz Gajowy
• Created: 10/Apr/18 10:58
• Updated: 16/May/20 14:11
• Resolved: 19/Dec/19 15:32

Description

Recently, as we added more IOITs to be run on jenkins using kubernetes, some of them started to fail randomly, because they couldn't retrieve LoadBalancer address. Normally obtaining the address took about one minute. Perfkit waits for the address (actively checking for it) for 3 minutes. This should be enough for getting the address, yet it recently started to exceed the 3 minutes limit. I also noticed that this error didn't happen when there were fewer tests.

Example logs:

https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_Compressed_TextIOIT_HDFS/31/console

Issue Links

• links to: GitHub Pull Request #5425