Partitioned S3A magic committers can leaving pending files, maybe upload data
This surfaced in an assertion failure on a parallel test run.
I thought it was actually a test failure, but with
HADOOP-16207 all the docs are preserved in the local FS and I can understand what happened.
Full details to follow in the comment as they are, well, detailed.
Key point: AM-side job and task cleanup can happen before the worker task finishes its writes. This will result in files under __magic. It may result in pending uploads too -but only if the write began after the AM job cleanup did a list + abort of all pending uploads under the destination directory