[PIG-4868] Low values for bytes.per.reducer configured by user not honored in Tez for inputs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 12/Apr/16 17:47
• Updated: 08/Jun/16 20:48
• Resolved: 12/Apr/16 23:38

Description

We don't use that in parallelism estimation and only use that in ShuffleVertexManager. Usually it is fine, but in cases when user has set a really low bytes.per.reducer it does not work as our estimation is low. ShuffleVertexManager only decreases parallelism and not increases parallelism.