[HADOOP-16058] S3A tests to include Terasort

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0, 3.2.1
• Component/s: fs/s3, test
• Target Version/s: 3.3.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 18/Jan/19 22:03
• Updated: 19/Jun/19 21:36
• Resolved: 29/Mar/19 15:28

Description

Add S3A tests to run terasort for the magic and directory committers.

MAPREDUCE-7091 is a requirement for this

Bonus feature: print the results to see which committers are faster in the specific test setup. As that's a function of latency to the store, bandwidth and size of jobs, it's not at all meaningful, just interesting.

Issue Links

• causes: HADOOP-16357 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists
• contains: MAPREDUCE-7090 BigMapOutput example doesn't work with paths off cluster fs
• contains: MAPREDUCE-7091 Terasort on S3A to switch to new committers
• contains: MAPREDUCE-7092 MR examples to work better against cloud stores
• depends upon: MAPREDUCE-7099 Daily test result fails in MapReduce JobClient though there isn't any error
• incorporates: HADOOP-15890 Some S3A committer tests don't match ITest* pattern; don't run in maven
• incorporates: MAPREDUCE-7090 BigMapOutput example doesn't work with paths off cluster fs
• incorporates: MAPREDUCE-7091 Terasort on S3A to switch to new committers
• incorporates: MAPREDUCE-7092 MR examples to work better against cloud stores
• is duplicated by: HADOOP-16153 Allow TeraGen to use schema-specific output committer
• links to: GitHub Pull Request #530
• links to: GitHub Pull Request #565
• links to: GitHub Pull Request #576
• links to: GitHub Pull Request #577
• links to: GitHub Pull Request #631