[TEZ-3334] Tez Custom Shuffle Handler

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Assignee: Unassigned
• Reporter: Jonathan Turner Eagles
• Created: 11/Jul/16 20:34
• Updated: 22/Aug/17 03:02
• Resolved: 01/Jun/17 18:41

Description

For conditions where auto-parallelism is reduced (e.g. TEZ-3222), a custom shuffle handler could help reduce the number of fetches and could more efficiently fetch data. In particular if a reducer is fetching 100 pieces serially from the same mapper it could do this in one fetch call.

Sub-Tasks

• Tez Custom Shuffle Handler POC: Closed
• Tez Custom Shuffle Handler Documentation: Closed
• Fetch Multiple Partitions from the Shuffle Handler: Closed
• Delete intermediate data at DAG level for Shuffle Handler: Closed
• Delete intermediate data at the vertex level for Shuffle Handler: Resolved
• Query fetch stats without fetching for Shuffle Handler: Open
• Add support for Multiple Files Fetch from the Shuffle Handler: Open
• Allow tez to enable mapreduce or tez shuffle handler: Resolved
• Tez Shuffle Bench: Open
• Remove ShuffleHandler dependency on mapred.FadvisedChunkedFile and mapred.FadvisedFileRegion: Closed
• Move Shuffle Handler configuration into the Tez namespace: Closed
• Shuffle Handler: Replace primitive wrapper's valueOf method with parse* method to avoid unnecessary boxing/unboxing: Closed
• Remove ShuffleHandler dependency on mapred.JobShuffleInfoProto and mapreduce.JobID: In Progress
• Provide error information in shuffle response header: Open
• Package Shuffle Handler as a shaded uber-jar: Closed
• Remove extra jetty dependency from Shuffle Handler: Closed
• Shuffle service name should be configureable and should not be hardcoded to ‘mapreduce_shuffle’: Closed
• Allow Task Output Files to reside in DAG specific directories for Custom Shuffle Handler: Closed
• ShuffleHandler should use Path.SEPARATOR instead of '/': Closed
• TestShuffleHandler#testSendMapCount should not used hard coded ShuffleHandler port: Closed
• Modify ShuffleHandler to use Constants.DAG_PREFIX and fix AttemptPathIdentifier#toString(): Closed
• Ability to configure shuffle server listen queue length: Resolved
• Port MAPREDUCE-6763 to Tez ShuffleHandler: Closed
• Make DAG Deletion Tracker configurable for ContainerLaunchers: Closed
• Backport MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log: Closed
• Make Deletion Service Pluggable: Open
• TEZ-3362 causes TestContainerLauncherWrapper#testDelegation to fail: Closed
• Tez Shuffle Handler logging fails to initialize: Closed
• TezConfiguration#TEZ_DELETION_TRACKER_CLASS has the wrong config key-name: Closed
• Remove fusesource.leveldbjni from the tez-auxservices shaded jar: Closed
• Fetcher fetchInputs() can NPE on srcAttempt due to missing entry in pathToAttemptMap: Closed
• Remove google.protobuf from the tez-auxservices shaded jar: Closed
• Composite Fetch account error for disk direct: Closed
• Number of Empty DME logged for Composite fetch is too high: Closed
• Composite Fetch hangs on certain DME empty events.: Closed
• Unordered Fetcher can hang if empty partitions are present: Closed
• Remove the compositeInputAttemptIdentifier from remaining list upon fetch completion in the Ordered case: Closed
• Fix debug log for empty partitions to the expanded partitionId in the Composite case: Closed
• ShuffleScheduler#allEventsReceived check is too tight: Open
• Fetcher can hang if copyMapOutput/fetchInputs returns early: Closed
• Tez Shuffle Handler Content length does not match actual: Closed
• Shuffle Handler Loading cache equality tests always results is false: Closed
• UnorderedPartitionedKVOutput is missing the shuffle service config in the confKeys set: Closed
• Optimize the Shuffle Handler content length calculation for keep alive: Closed
• Give Tez shuffle handler threads custom names: Closed
• Implement keep-alive timeout in tez shuffle handler: Closed
• Pass parameters instead of configuration for changes to support tez shuffle handler: Closed
• LocalContainerLauncher#shouldDelete member variable is not used: Closed
• Incorporate first pass non-essential TEZ-3334 pre-merge feedback: Closed
• ShuffleHandler completedInputSet off-by-one error: Closed
• Tez shuffle jar includes service loader entry for ClientProtocolProvider but not the corresponding class: Closed
• Modify DeletionTracker and deletion threads to be initialized only if enabled for tez_shuffle: Closed
• Use Local FileContext for deleting dag level directories: Closed
• Allow dag level deletion in cases where containers are reused: Closed
• Cleanup http connections and other unnecessary fields in DAG Deletion tracker classes.: Closed
• Clean up DeletionTracker's reflection instantiation and provide ContainerLauncher with dagComplete() functionality: Closed
• Use a new abstract class for ContainerLauncher(s) that provide dagComplete() functionality: Resolved
• Test failures in TestTaskAttempt and TestAMContainerMap: Closed
• Clean up TEZ-3334-CHANGES.txt: Closed
• Delete intermediate attempt data for failed attempts for Shuffle Handler: Resolved
• Remove javax.security from the tez-auxservices shaded jar: Resolved