[BEAM-5455] Don't info log for every bundle in the python sdk

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.8.0
• Component/s: sdk-py-harness
• Labels: portability-flink
• Assignee: Micah Wylde
• Reporter: Micah Wylde
• Created: 21/Sep/18 01:06
• Updated: 16/May/20 13:19
• Resolved: 25/Sep/18 02:35

Description

Currently the python sdk logs a bunch of messages at INFO level for every bundle. For the portable flink runner, which uses a bundle size of 1 in streaming mode, this produces huge log spew:

[grpc-default-executor-1] INFO sdk_worker.run - Got work 4
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DataOutputOperation >
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DoOperation window output_tags=['out']>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DoOperation pair_with_one output_tags=['out']>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DoOperation timestamp output_tags=['out']>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DoOperation decode output_tags=['out']>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start <DataInputOperation receivers=[ConsumerSet[.out0, coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DataInputOperation receivers=[ConsumerSet[.out0, coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DoOperation decode output_tags=['out'], receivers=[ConsumerSet[decode.out0, coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DoOperation timestamp output_tags=['out'], receivers=[ConsumerSet[timestamp.out0, coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DoOperation pair_with_one output_tags=['out'], receivers=[ConsumerSet[pair_with_one.out0, coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, FastPrimitivesCoder]], len(consumers)=1]]>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DoOperation window output_tags=['out'], receivers=[ConsumerSet[window.out0, coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, FastPrimitivesCoder]], len(consumers)=1]]>
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish <DataOutputOperation >

 

These log messages should probably be at debug instead.

Issue Links

• relates to: BEAM-5468 Allow runner to set worker log level in Python SDK harness.
• links to: GitHub Pull Request #6459