[BEAM-2326] Verbose INFO logging with stateful DoFns and Dataflow

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 2.0.0
• Component/s: runner-dataflow
• Labels: log
• Assignee: Kenneth Knowles
• Reporter: peay
• Created: 19/May/17 15:01
• Updated: 16/May/20 13:56
• Resolved: 09/Jun/17 01:38

Description

I am seeing a lot of INFO level logging:

 jsonPayload: {
  logger: "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" 
  message: "Processing timers for key {} for stateful DoFn"    
 }
 jsonPayload: {
  message: "Processing key KV{one of my keys} for stateful DoFn"    
  logger: "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn"    
}

out of one of my stateful DoFn. There is one such group of logs for each key I process, which leads to a very large amount of logs and possibly to a significant slowdown.

Also, not sure if the Processing timers log message is missing some string interpolation or if the empty key is on purpose.

At any rate, this seems more like something for DEBUG than INFO given the large volume.