Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This is a proposal for a system specialized in running Hadoop/Pig jobs in a control dependency DAG (Direct Acyclic Graph), a Hadoop workflow application.

      Attached there is a complete specification and a high level overview presentation.


      Highlights

      A Workflow application is DAG that coordinates the following types of actions: Hadoop, Pig, Ssh, Http, Email and sub-workflows.

      Flow control operations within the workflow applications can be done using decision, fork and join nodes. Cycles in workflows are not supported.

      Actions and decisions can be parameterized with job properties, actions output (i.e. Hadoop counters, Ssh key/value pairs output) and file information (file exists, file size, etc). Formal parameters are expressed in the workflow definition as {{$

      {VAR}

      }} variables.

      A Workflow application is a ZIP file that contains the workflow definition (an XML file), all the necessary files to run all the actions: JAR files for Map/Reduce jobs, shells for streaming Map/Reduce jobs, native libraries, Pig scripts, and other resource files.

      Before running a workflow job, the corresponding workflow application must be deployed in HWS.

      Deploying workflow application and running workflow jobs can be done via command line tools, a WS API and a Java API.

      Monitoring the system and workflow jobs can be done via a web console, command line tools, a WS API and a Java API.

      When submitting a workflow job, a set of properties resolving all the formal parameters in the workflow definitions must be provided. This set of properties is a Hadoop configuration.

      Possible states for a workflow jobs are: CREATED, RUNNING, SUSPENDED, SUCCEEDED, KILLED and FAILED.

      In the case of a action failure in a workflow job, depending on the type of failure, HWS will attempt automatic retries, it will request a manual retry or it will fail the workflow job.

      HWS can make HTTP callback notifications on action start/end/failure events and workflow end/failure events.

      In the case of workflow job failure, the workflow job can be resubmitted skipping previously completed actions. Before doing a resubmission the workflow application could be updated with a patch to fix a problem in the workflow application code.


      1. Hadoop_Summit_Oozie.pdf
        1.91 MB
        Alejandro Abdelnur
      2. hws-preso-v1_0_2009FEB22.pdf
        1.88 MB
        Alejandro Abdelnur
      3. hws-spec2009MAR09.pdf
        241 kB
        Alejandro Abdelnur
      4. hws-v1_0_2009FEB22.pdf
        229 kB
        Alejandro Abdelnur
      5. oozie-0.18.3.o0.1-SNAPSHOT-distro.tar.gz
        7.65 MB
        Alejandro Abdelnur
      6. oozie-patch.txt
        4.82 MB
        Alejandro Abdelnur
      7. oozie-spec-20090521.pdf
        204 kB
        Alejandro Abdelnur
      8. oozie-src-20090605.tar.gz
        7.51 MB
        Alejandro Abdelnur
      9. packages.tar.gz
        9.66 MB
        Alejandro Abdelnur

        Activity

        Alejandro Abdelnur made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Chris Douglas made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Alejandro Abdelnur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Alejandro Abdelnur made changes -
        Attachment oozie-patch.txt [ 12410748 ]
        Alejandro Abdelnur made changes -
        Attachment packages.tar.gz [ 12410745 ]
        Alejandro Abdelnur made changes -
        Attachment Hadoop_Summit_Oozie.pdf [ 12410356 ]
        Alejandro Abdelnur made changes -
        Summary Hadoop Workflow System (HWS) Oozie, Hadoop Workflow System
        Alejandro Abdelnur made changes -
        Alejandro Abdelnur made changes -
        Attachment oozie-src-20090605.tar.gz [ 12409972 ]
        Alejandro Abdelnur made changes -
        Attachment oozie-spec-20090521.pdf [ 12408872 ]
        Alejandro Abdelnur made changes -
        Attachment hws-spec2009MAR09.pdf [ 12401735 ]
        Alejandro Abdelnur made changes -
        Attachment hws-v1_0_2009FEB22.pdf [ 12400686 ]
        Alejandro Abdelnur made changes -
        Field Original Value New Value
        Attachment hws-preso-v1_0_2009FEB22.pdf [ 12400685 ]
        Alejandro Abdelnur created issue -

          People

          • Assignee:
            Alejandro Abdelnur
            Reporter:
            Alejandro Abdelnur
          • Votes:
            6 Vote for this issue
            Watchers:
            52 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development