[TEZ-690] Tez API Ease of Use

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Assignee: Bikas Saha
• Reporter: Bikas Saha
• Created: 29/Dec/13 12:41
• Updated: 18/May/16 04:57
• Resolved: 12/May/15 21:43

Description

Recently we wrote the wordcount example from scratch using Tez API's in TEZ-689. The code shows some room for improvement in making the Tez API's more concise and less error prone. This jira tracks some of those changes. The improvements in this jira will be reflected in the cleanliness and conciseness of the word count example job.

Issue Links

• relates to: TEZ-1136 Refactor TestOrderedWordCount to use best practices for Tez API

Sub-Tasks

• TezSession creation should not need an appId in the constructor: Resolved
• Unify job submission in either TezClient or TezSession: Closed
• Remove task commit burden from user code: Open
• Create Abstract class for Input/Processor/Output: Closed
• Remove implicit copying of processor payload to input and output: Closed
• Payload for MROutput committer needs to be specified on the output and not the vertex: Resolved
• Make it easy to create and configure MRInput/MROutput and other inputs/outputs: Closed
• Have sensible defaults for java opts: Closed
• Helper API's to monitor a DAG to completion: Closed
• Helper API to add local resources: Closed
• Allow the DAG configuration to be available as a payload for user code: Open
• Simplify Local Resource specification while building a DAG: Open
• Add parameter checking for context related user API's: Closed
• Add @Nullable annotation at API level: Closed
• Replace interfaces with abstract classes for Processor/Input/Output classes: Closed
• Replace user land interfaces with abstract classes: Closed
• Create SimpleProcessor for processors that only need to implement the run method: Closed
• Configuration for non MR based Inputs/Outputs: Closed
• Add TEZ_TASK_JAVA_OPTS and TEZ_ENV configs to specify values from config: Closed
• Replace confusing names on Vertex API: Closed
• Simplify EdgeManager APIs: Closed
• Consistent naming of Input and Outputs: Closed
• Remove unnecessary MRHelpers methods or make them private: Closed
• InputInitializer and OutputCommitter implicitly use payloads of the input and output: Closed
• Move TezJobConfig to runtime-library and rename to TezRuntimeConfiguration: Closed
• Separate user facing and system facing methods from Inputs/Outputs: Closed
• Split Input/Output interfaces into Framework / User components: Closed
• Make TezUserPayload user facing for payload specification: Closed
• Consistent naming for components in tez-api: Resolved
• Add and review Javadocs for API classes: Closed
• Add examples for uses of the API: Resolved
• Replace Interfaces with Abstract classes for VertexManagerPlugin and EdgeManager: Closed
• Implement common functionality in base abstract class for VertexManagerPlugin: Open
• Implement common functionality in base abstract class for EdgeManager: Open
• Consolidate naming of API classes: Closed
• VertexManagerEvent documentation and use of payload needs fixing: Open
• Replace constructors with create() methods for DAG, Vertex, Edge etc in the API: Closed
• Create *EdgeConfigurer.createDefaultCustomEdge() and force setting partitioners: Closed
• Exception signatures for user pluggable code: Open
• Add additional fields to TezMergedInputContext: Open
• Change I/P/O/etc construction to make use of contexts: Resolved
• get sharedobjectregistry from the context instead of a static: Closed
• rename vertex.addInput/Output() to vertex.addDataSource/Sink(): Closed
• Document public interfaces/classes which are not meant to be implemented/extended: Resolved
• Simplify MRinput/MROutput configuration: Closed
• Simplify Vertex constructor: Closed
• Remove methods annotated as @Private from TezClient: Closed
• Remove getApplicationId from DAGClient: Closed
• Remove @Private methods from DAGClient: Closed
• Use VertexLocationHint consistently everywhere in the API: Closed
• Investigate : interrupts being swallowed by TezClient/DAGClient methods: Resolved
• Annotate all non public classes in tez-runtime-library with @private: Closed
• Change Processor construction to make use of contexts: Closed
• Consolidate MRHelpers: Closed
• Move SimpleMRProcessor, MRInput and MROutput into runtime-library: Closed
• Review and clean packages in the API: Resolved
• Fix preWarm to work after recent API changes: Closed
• EdgeConfigurers should accept a Partitioner configuration, accept parameters for compression and secure shuffle: Closed
• Rename *Configurer: Closed
• Make only wait apis in TezClient to throw InterruptedException: Closed
• DAG should be created via a create method: Closed