[BEAM-9319] ResourceExhausted: topics-per-project

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.21.0
• Component/s: test-failures, testing
• Assignee: Brian Hulette
• Reporter: Ahmet Altay
• Created: 14/Feb/20 17:45
• Updated: 16/May/20 13:38
• Resolved: 09/Mar/20 17:56

Description

Tests are failing due to quota issues. Do we need to clean up topics after tests or set a shorter TTL?

Log: https://builds.apache.org/job/beam_PreCommit_Python_Commit/11178/

Error:
08:24:40 ======================================================================
08:24:40 ERROR: test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT)
08:24:40 ----------------------------------------------------------------------
08:24:40 Traceback (most recent call last):
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", line 58, in setUp
08:24:40 self.pub_client.topic_path(self.project, INPUT_TOPIC + self.uuid))
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/_gapic.py", line 40, in <lambda>
08:24:40 fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw) # noqa
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 332, in create_topic
08:24:40 request, retry=retry, timeout=timeout, metadata=metadata
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in _call_
08:24:40 return wrapped_func(*args, **kwargs)
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
08:24:40 on_error=on_error,
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/retry.py", line 184, in retry_target
08:24:40 return target()
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
08:24:40 return func(*args, **kwargs)
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
08:24:40 six.raise_from(exceptions.from_grpc_error(exc), exc)
08:24:40 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/build/gradleenv/-194514014/local/lib/python2.7/site-packages/six.py", line 738, in raise_from
08:24:40 raise value
08:24:40 ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=10000, maximum=10000).

Issue Links

• links to: GitHub Pull Request #11072