[HADOOP-16546] make sure staging committers collect DTs for the staging FS

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.2.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 04/Sep/19 12:48
• Updated: 25/Feb/21 11:26
• Resolved: 25/Feb/21 11:26

Description

This is not a problem I've seen in the wild, but I've now encountered a problem with hive doing something like this

we need to (somehow) make sure that the staging committers collect DTs for the staging dir FS. If this is the default FS or the same as a source or dest FS, this is handled elsewhere, but otherwise we need to add the staging fs.

I don;t see an easy way to do this, but we could add a new method to PathOutputCommitter to collect DTs; FileOutputFormat can invoke this alongside its ongoing collection of tokens for the output FS. Base impl would be a no-op, obviously.