For reducers in large jobs our users cannot easily spot portions of the log associated with problems with their code. An example reducer with INFO-level logging generates ~3500 lines / ~700KiB lines per second. 95% of the log is the client-side of the shuffle org.apache.hadoop.mapreduce.task.reduce.*
Byte percentage breakdown:
While this is information is actually often useful for devops debugging shuffle performance issues, the job users are often lost.
We propose to have a dedicated syslog.shuffle file.