Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-398

[Umbrella] Tez Engine API changes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2.0
    • Component/s: None
    • Labels:
      None

      Attachments

      1. TEZ-398-Engine-Design.pdf
        177 kB
        Hitesh Shah
      1.
      Remove MRRuntimeTask from TezContext Sub-task Closed Bikas Saha
      2.
      Update I/O/P APIs Sub-task Closed Siddharth Seth
      3.
      Tez Engine API should not expose or depend on Key Value pairs Sub-task Closed Unassigned
      4.
      Change RuntimeTask to handle Input/Output lifecycle Sub-task Closed Siddharth Seth
      5.
      Define a TezTaskContext for user code to interact with the AM Sub-task Closed Hitesh Shah
      6.
      Introduce a K-V Input/Output Sub-task Closed Siddharth Seth
      7.
      Support multiple inputs and connection patterns in Tez Sub-task Closed Bikas Saha
      8.
      Initial work on Tez*Context impls to aid integration Sub-task Closed Hitesh Shah
      9.
      Simplify Event class heirarchy Sub-task Closed Hitesh Shah
      10.
      Change Shuffle Input/Output to work with the new APIs Sub-task Closed Siddharth Seth
      11.
      Change SimpleInput to work with the new engine APIs Sub-task Closed Siddharth Seth
      12.
      Change SimpleOutput to work with the new engine APIs Sub-task Closed Hitesh Shah
      13.
      Change Map/Reduce Processor to work with the new engine APIs Sub-task Closed Bikas Saha
      14.
      Move new re-factor classes to appropriate module Sub-task Closed Hitesh Shah
      15.
      Add input information and failed events Sub-task Closed Bikas Saha
      16.
      Create DataMovementEvent and InputInformationEvent user payloads for shuffle Sub-task Closed Siddharth Seth
      17.
      Changes in YarnTezDagChild for logical task using TaskSpec Sub-task Closed Hitesh Shah
      18.
      Rename InputDataErrorEvent to InputReadErrorEvent and add Javadocs Sub-task Closed Siddharth Seth
      19.
      Add index info to event meta data Sub-task Closed Hitesh Shah
      20.
      Basic event router for Logical Runtime task Sub-task Closed Hitesh Shah
      21.
      Remove use of event lock Sub-task Closed Hitesh Shah
      22.
      Make JobToken available to inputs / outputs Sub-task Closed Siddharth Seth
      23.
      Change Combiner to work with new APIs Sub-task Closed Siddharth Seth
      24.
      Rename EventMetaData Generator to enforce clarity on both producer/consumer semantics Sub-task Closed Hitesh Shah
      25.
      Handle out-of-band fatal errors from inputs/outputs Sub-task Closed Hitesh Shah
      26.
      Send regular progress updates from tez task to AM Sub-task Closed Hitesh Shah
      27.
      Make newmapreduce.TaskAttemptContextImpl use TezTaskContext Sub-task Closed Bikas Saha
      28.
      Allow users to configure a Partitioner in Tez-Engine compoenents, which can be instantiated via reflection Sub-task Closed Siddharth Seth
      29.
      Fix task attempt commit flow Sub-task Closed Bikas Saha
      30.
      Remove shuffle port from AM code Sub-task Closed Bikas Saha
      31.
      Move tez event handling thread into the LogicalTask from YarnTezDagChild Sub-task Closed Hitesh Shah
      32.
      Handle events generated by I/O initialize and close Sub-task Closed Siddharth Seth
      33.
      Rename *.new* packages back to what they should be, remove dead code from the old packages - mapreduce module Sub-task Closed Siddharth Seth
      34.
      Allow only a single instance of SimpleInput.RecordReader to be created Sub-task Closed Siddharth Seth
      35.
      Update YARNRunner, examples etc to use the new Inputs, Outputs and Processors Sub-task Closed Siddharth Seth
      36.
      Make child jvm error out in case of heartbeat errors Sub-task Closed Hitesh Shah
      37.
      Fix serialization error in TezEvent Sub-task Closed Hitesh Shah
      38.
      Several bug fixes in new api implementation Sub-task Closed Siddharth Seth
      39.
      TASK_STATUS_UPDATE_EVENT not handled in state RUNNING Sub-task Closed Unassigned
      40.
      Handle the case where heartbeatResponse.eventList is empty Sub-task Closed Siddharth Seth
      41.
      Fix umbilical heartbeat handshake with AM Sub-task Closed Hitesh Shah
      42.
      Fix Partitioner instantiation Sub-task Closed Siddharth Seth
      43.
      Fix local dir setup and committer setup for MRTask Sub-task Closed Siddharth Seth
      44.
      Fix heartbeat response counting to depend on Container and not TaskAttempt Sub-task Closed Bikas Saha
      45.
      YARNTezDAGChild instances hang around after a job completes or when a job is killed Sub-task Closed Unassigned
      46.
      Stage 2 tasks are not getting any events over the umbilical Sub-task Closed Unassigned
      47.
      Investigate: Output.close events do not go to the AM immediately Sub-task Closed Unassigned
      48.
      Fix TaskAttempt status update via heartbeat events Sub-task Closed Bikas Saha
      49.
      Fix MapProcessor to handle map only jobs Sub-task Closed Hitesh Shah
      50.
      VertexName not set on the Status_Update_Event Sub-task Closed Hitesh Shah
      51.
      Fixes for events in Child JVM Sub-task Closed Hitesh Shah
      52.
      Dont send 2 events to fail an attempt. Send die signal to non-registered container Sub-task Closed Bikas Saha
      53.
      Invalid progress reported when running word count job Sub-task Closed Hitesh Shah
      54.
      Make OrderedWordCount job work after Engine API changes Sub-task Closed Bikas Saha
      55.
      Fix WordCount job to work after recent changes in Tez Engine. Sub-task Closed Hitesh Shah
      56.
      Add log messages to new engine impl to aid in debugging. Sub-task Closed Hitesh Shah
      57.
      Implement fault tolerance, retries and event flow for dealing with failed inputs Sub-task Closed Bikas Saha
      58.
      Fix TestMRRJobsDAGApi to work with modified engine layer Sub-task Closed Hitesh Shah
      59.
      Various fixes including NPE in shuffle when run MR jobs using mapred apis Sub-task Closed Hitesh Shah
      60.
      Fix TestMapProcessor and TestReduceProcessor unit tests Sub-task Closed Siddharth Seth
      61.
      Fix TestAMContainer unit test Sub-task Closed Unassigned
      62.
      Fixes after the Combiner patch Sub-task Closed Siddharth Seth
      63.
      Rename *.new* packages back to what they should be, remove dead code from the old packages - mapreduce module - tez-engine module Sub-task Closed Siddharth Seth
      64.
      Merge tez-dag-api and tez-engine-api into a single module - tez-api Sub-task Closed Siddharth Seth
      65.
      Rename SimpleInput / SimpleOutput to be MR specific Sub-task Closed Siddharth Seth
      66.
      Rename engine.api to runtime.api and engine to runtime.library Sub-task Closed Hitesh Shah

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            sseth Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: