Reading from a Cloud Pub/Sub topic inside Python's DirectRunner results in a "too many open files" error within 2 minutes for a busy topic.
I am not exactly sure, but it appears that "pubsub.SubscriberClient()" creates a grpc channel which must be closed explicitly after we are done pulling messages from that channel. This may have been due to change that was introduced in grpc v1.12.0 ("a new grpc.Channel.close method is introduced and correct use of gRPC Python now requires that channels be closed after use"). If the underling grcp channel is not closed and many "pubsub.SubscriberClient" instances are created, the system may run out of available file handles, resulting in "too many open files" errors. A similar issue is reported here: https://github.com/googleapis/google-cloud-python/issues/5523.
The issue can be reproduced by running the following file:
Currently, the output from running this script looks something like: