[BEAM-9550] beam_PostCommit_Python_Chicago_Taxi_Flink OOM

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: runner-flink, test-failures
• Labels: currently-failing
• Assignee: Kamil Wasilewski
• Reporter: Kyle Weaver
• Created: 18/Mar/20 22:17
• Updated: 16/May/20 14:15
• Resolved: 10/Apr/20 09:09

Description

https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Flink/

The following error has been occurring consistently for several days:

07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace
07:57:27 Traceback (most recent call last):
07:57:27 File "tfdv_analyze_and_validate.py", line 227, in <module>
07:57:27 main()
07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main
07:57:27 project=known_args.metric_reporting_project)
07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats
07:57:27 result.wait_until_finish()
07:57:27 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 545, in wait_until_finish
07:57:27 (self._job_id, self._state, self._last_error_message()))
07:57:27 RuntimeError: Pipeline chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb failed in state FAILED: java.lang.OutOfMemoryError: Metaspace

Issue Links

• is related to: BEAM-9673 Migrate the memory configuration of Flink cluster from Flink <= 1.9 to 1.10
• links to: GitHub Pull Request #11193
• links to: GitHub Pull Request #11291