[HIVE-3700] Hive ignores 'mapreduce.fileoutputcommitter.marksuccessfuljobs'

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: Shims
• Assignee: Unassigned
• Reporter: Joe Crobak
• Created: 11/Nov/12 20:44
• Updated: 16/Apr/19 16:51

Description

For downstream computation, it can be useful to have a _SUCCESS flag dropped in the directory to signify that the data is completely written. This is typically enabled with the property mapreduce.fileoutputcommitter.marksuccessfuljobs, but hive ignores this. I believe it's because the shims layer src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java and shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java both supply a NullOutputCommitter that don't respect this property.