Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-690

Tez API Ease of Use

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Recently we wrote the wordcount example from scratch using Tez API's in TEZ-689. The code shows some room for improvement in making the Tez API's more concise and less error prone. This jira tracks some of those changes. The improvements in this jira will be reflected in the cleanliness and conciseness of the word count example job.

        Attachments

          Issue Links

          1.
          TezSession creation should not need an appId in the constructor Sub-task Resolved Unassigned
          2.
          Unify job submission in either TezClient or TezSession Sub-task Closed Bikas Saha
          3.
          Remove task commit burden from user code Sub-task Open Mohammad Kamrul Islam
          4.
          Create Abstract class for Input/Processor/Output Sub-task Closed Mohammad Kamrul Islam
          5.
          Remove implicit copying of processor payload to input and output Sub-task Closed Bikas Saha
          6.
          Payload for MROutput committer needs to be specified on the output and not the vertex Sub-task Resolved Unassigned
          7.
          Make it easy to create and configure MRInput/MROutput and other inputs/outputs Sub-task Closed Bikas Saha
          8.
          Have sensible defaults for java opts Sub-task Closed Hitesh Shah
          9.
          Helper API's to monitor a DAG to completion Sub-task Closed Mohammad Kamrul Islam
          10.
          Helper API to add local resources Sub-task Closed Unassigned
          11.
          Allow the DAG configuration to be available as a payload for user code Sub-task Open Unassigned
          12.
          Simplify Local Resource specification while building a DAG Sub-task Open Unassigned
          13.
          Add parameter checking for context related user API's Sub-task Closed Tsuyoshi Ozawa
          14.
          Add @Nullable annotation at API level Sub-task Closed Tsuyoshi Ozawa
          15.
          Replace interfaces with abstract classes for Processor/Input/Output classes Sub-task Closed Bikas Saha
          16.
          Replace user land interfaces with abstract classes Sub-task Closed Bikas Saha
          17.
          Create SimpleProcessor for processors that only need to implement the run method Sub-task Closed Mohammad Kamrul Islam
          18.
          Configuration for non MR based Inputs/Outputs Sub-task Closed Siddharth Seth
          19.
          Add TEZ_TASK_JAVA_OPTS and TEZ_ENV configs to specify values from config Sub-task Closed Bikas Saha
          20.
          Replace confusing names on Vertex API Sub-task Closed Bikas Saha
          21.
          Simplify EdgeManager APIs Sub-task Closed Bikas Saha
          22.
          Consistent naming of Input and Outputs Sub-task Closed Bikas Saha
          23.
          Remove unnecessary MRHelpers methods or make them private Sub-task Closed Chen He
          24.
          InputInitializer and OutputCommitter implicitly use payloads of the input and output Sub-task Closed Bikas Saha
          25.
          Move TezJobConfig to runtime-library and rename to TezRuntimeConfiguration Sub-task Closed Bikas Saha
          26.
          Separate user facing and system facing methods from Inputs/Outputs Sub-task Closed Unassigned
          27.
          Split Input/Output interfaces into Framework / User components Sub-task Closed Siddharth Seth
          28.
          Make TezUserPayload user facing for payload specification Sub-task Closed Tsuyoshi Ozawa
          29.
          Consistent naming for components in tez-api Sub-task Resolved Jeff Zhang
          30.
          Add and review Javadocs for API classes Sub-task Closed Siddharth Seth
          31.
          Add examples for uses of the API Sub-task Resolved Bikas Saha
          32.
          Replace Interfaces with Abstract classes for VertexManagerPlugin and EdgeManager Sub-task Closed Hitesh Shah
          33.
          Implement common functionality in base abstract class for VertexManagerPlugin Sub-task Open Unassigned
          34.
          Implement common functionality in base abstract class for EdgeManager Sub-task Open Chen He
          35.
          Consolidate naming of API classes Sub-task Closed Bikas Saha
          36.
          VertexManagerEvent documentation and use of payload needs fixing Sub-task Open Unassigned
          37.
          Replace constructors with create() methods for DAG, Vertex, Edge etc in the API Sub-task Closed Siddharth Seth
          38.
          Create *EdgeConfigurer.createDefaultCustomEdge() and force setting partitioners Sub-task Closed Siddharth Seth
          39.
          Exception signatures for user pluggable code Sub-task Open Unassigned
          40.
          Add additional fields to TezMergedInputContext Sub-task Open Unassigned
          41.
          Change I/P/O/etc construction to make use of contexts Sub-task Resolved Siddharth Seth
          42.
          get sharedobjectregistry from the context instead of a static Sub-task Closed Bikas Saha
          43.
          rename vertex.addInput/Output() to vertex.addDataSource/Sink() Sub-task Closed Chen He
          44.
          Document public interfaces/classes which are not meant to be implemented/extended Sub-task Resolved Siddharth Seth
          45.
          Simplify MRinput/MROutput configuration Sub-task Closed Bikas Saha
          46.
          Simplify Vertex constructor Sub-task Closed Bikas Saha
          47.
          Remove methods annotated as @Private from TezClient Sub-task Closed Siddharth Seth
          48.
          Remove getApplicationId from DAGClient Sub-task Closed Jonathan Eagles
          49.
          Remove @Private methods from DAGClient Sub-task Closed Siddharth Seth
          50.
          Use VertexLocationHint consistently everywhere in the API Sub-task Closed Bikas Saha
          51.
          Investigate : interrupts being swallowed by TezClient/DAGClient methods Sub-task Resolved Unassigned
          52.
          Annotate all non public classes in tez-runtime-library with @private Sub-task Closed Hitesh Shah
          53.
          Change Processor construction to make use of contexts Sub-task Closed Siddharth Seth
          54.
          Consolidate MRHelpers Sub-task Closed Siddharth Seth
          55.
          Move SimpleMRProcessor, MRInput and MROutput into runtime-library Sub-task Closed Bikas Saha
          56.
          Review and clean packages in the API Sub-task Resolved Bikas Saha
          57.
          Fix preWarm to work after recent API changes Sub-task Closed Bikas Saha
          58.
          EdgeConfigurers should accept a Partitioner configuration, accept parameters for compression and secure shuffle Sub-task Closed Siddharth Seth
          59.
          Rename *Configurer Sub-task Closed Siddharth Seth
          60.
          Make only wait apis in TezClient to throw InterruptedException Sub-task Closed Hitesh Shah
          61.
          DAG should be created via a create method Sub-task Closed Siddharth Seth

            Activity

              People

              • Assignee:
                bikassaha Bikas Saha
                Reporter:
                bikassaha Bikas Saha
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: