[TEZ-398] [Umbrella] Tez Engine API changes

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0
• Assignee: Unassigned
• Reporter: Siddharth Seth
• Created: 26/Aug/13 22:31
• Updated: 01/Dec/13 20:20
• Resolved: 26/Sep/13 04:15

Sub-Tasks

• Remove MRRuntimeTask from TezContext: Closed
• Update I/O/P APIs: Closed
• Tez Engine API should not expose or depend on Key Value pairs: Closed
• Change RuntimeTask to handle Input/Output lifecycle: Closed
• Define a TezTaskContext for user code to interact with the AM: Closed
• Introduce a K-V Input/Output: Closed
• Support multiple inputs and connection patterns in Tez: Closed
• Initial work on Tez*Context impls to aid integration: Closed
• Simplify Event class heirarchy: Closed
• Change Shuffle Input/Output to work with the new APIs: Closed
• Change SimpleInput to work with the new engine APIs: Closed
• Change SimpleOutput to work with the new engine APIs: Closed
• Change Map/Reduce Processor to work with the new engine APIs: Closed
• Move new re-factor classes to appropriate module: Closed
• Add input information and failed events: Closed
• Create DataMovementEvent and InputInformationEvent user payloads for shuffle: Closed
• Changes in YarnTezDagChild for logical task using TaskSpec: Closed
• Rename InputDataErrorEvent to InputReadErrorEvent and add Javadocs: Closed
• Add index info to event meta data: Closed
• Basic event router for Logical Runtime task: Closed
• Remove use of event lock: Closed
• Make JobToken available to inputs / outputs: Closed
• Change Combiner to work with new APIs: Closed
• Rename EventMetaData Generator to enforce clarity on both producer/consumer semantics: Closed
• Handle out-of-band fatal errors from inputs/outputs: Closed
• Send regular progress updates from tez task to AM: Closed
• Make newmapreduce.TaskAttemptContextImpl use TezTaskContext: Closed
• Allow users to configure a Partitioner in Tez-Engine compoenents, which can be instantiated via reflection: Closed
• Fix task attempt commit flow: Closed
• Remove shuffle port from AM code: Closed
• Move tez event handling thread into the LogicalTask from YarnTezDagChild: Closed
• Handle events generated by I/O initialize and close: Closed
• Rename *.new* packages back to what they should be, remove dead code from the old packages - mapreduce module: Closed
• Allow only a single instance of SimpleInput.RecordReader to be created: Closed
• Update YARNRunner, examples etc to use the new Inputs, Outputs and Processors: Closed
• Make child jvm error out in case of heartbeat errors: Closed
• Fix serialization error in TezEvent: Closed
• Several bug fixes in new api implementation: Closed
• TASK_STATUS_UPDATE_EVENT not handled in state RUNNING: Closed
• Handle the case where heartbeatResponse.eventList is empty: Closed
• Fix umbilical heartbeat handshake with AM: Closed
• Fix Partitioner instantiation: Closed
• Fix local dir setup and committer setup for MRTask: Closed
• Fix heartbeat response counting to depend on Container and not TaskAttempt: Closed
• YARNTezDAGChild instances hang around after a job completes or when a job is killed: Closed
• Stage 2 tasks are not getting any events over the umbilical: Closed
• Investigate: Output.close events do not go to the AM immediately: Closed
• Fix TaskAttempt status update via heartbeat events: Closed
• Fix MapProcessor to handle map only jobs: Closed
• VertexName not set on the Status_Update_Event: Closed
• Fixes for events in Child JVM: Closed
• Dont send 2 events to fail an attempt. Send die signal to non-registered container: Closed
• Invalid progress reported when running word count job: Closed
• Make OrderedWordCount job work after Engine API changes: Closed
• Fix WordCount job to work after recent changes in Tez Engine.: Closed
• Add log messages to new engine impl to aid in debugging.: Closed
• Implement fault tolerance, retries and event flow for dealing with failed inputs: Closed
• Fix TestMRRJobsDAGApi to work with modified engine layer: Closed
• Various fixes including NPE in shuffle when run MR jobs using mapred apis: Closed
• Fix TestMapProcessor and TestReduceProcessor unit tests: Closed
• Fix TestAMContainer unit test: Closed
• Fixes after the Combiner patch: Closed
• Rename *.new* packages back to what they should be, remove dead code from the old packages - mapreduce module - tez-engine module: Closed
• Merge tez-dag-api and tez-engine-api into a single module - tez-api: Closed
• Rename SimpleInput / SimpleOutput to be MR specific: Closed
• Rename engine.api to runtime.api and engine to runtime.library: Closed