Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2339

[fluent-job] Minimum Viable Fluent Job API

    Details

      Description

      Users often complain about the XML they have to write for Oozie jobs. It would be nice if they could write them in something like Java, but we don't want to have to maintain a separate Java API for this. I was looking around and saw that JAXB might be the right thing here. From what I can tell, it lets you create Java classes from XSD schemas. So, we should be able to auto-generate a Java API for writing Oozie jobs, without having to really maintain it.

      We should investigate if this is feasible and, if so, implement it.

      Some useful looking links:

      Key features:

      • must have:
        • inside a fluent-job-api artifact
        • able to create workflow / coordinator / bundle definitions programmatically
        • synchronizing each and every XSD change on rebuild
        • can write workflow.xml, coordinator.xml, bundle.xml, and jobs.properties artifacts of every XSD version
        • cloneability of workflow etc. Object s
        • perform cross checks, e.g. that the workflow graph is a DAG
        • only latest XSD versions should be supported as must have
      • nice to have:
        • XSD version(s) can be provided. When not provided, latest ones are considered as valid
        • implement a fluent API
        • have a Python / Jython / Py4J REPL to make it easy to experiment with also for data engineers / data scientists
        • create documentation about usage
        • can read workflow.xml, coordinator.xml, bundle.xml, and jobs.properties artifacts of every XSD version
        • can convert between XSD versions
        • support XSD change on the fly (within REPL)
        • support HDFS reads / writes
        • support dry run on an Oozie server to perform checks

        Attachments

        1. OOZIE-2339.001.patch
          622 kB
          Andras Piros
        2. OOZIE-2339.002.patch
          635 kB
          Andras Piros
        3. OOZIE-2339.003.patch
          645 kB
          Andras Piros
        4. OOZIE-2339.004.patch
          671 kB
          Andras Piros
        5. OOZIE-2339.005.patch
          714 kB
          Andras Piros
        6. OOZIE-2339.006.patch
          777 kB
          Andras Piros
        7. OOZIE-2339.008.patch
          933 kB
          Andras Piros
        8. OOZIE-2339.010.patch
          1.05 MB
          Andras Piros
        9. OOZIE-2339.011.patch
          1.14 MB
          Andras Piros
        10. OOZIE-2339.012.patch
          1.17 MB
          Andras Piros
        11. OOZIE-2339.013.patch
          1.17 MB
          Andras Piros
        12. OOZIE-2339.014.patch
          1.19 MB
          Andras Piros
        13. OOZIE-2339.015.patch
          1.20 MB
          Andras Piros
        14. OOZIE-2339.016.patch
          1.21 MB
          Andras Piros
        15. OOZIE-2339.017.patch
          1.21 MB
          Andras Piros
        16. OOZIE-2339.018.patch
          1.21 MB
          Andras Piros
        17. OOZIE-2339.019.patch
          1.21 MB
          Andras Piros
        18. OOZIE-2339.020.patch
          1.21 MB
          Andras Piros
        19. OOZIE-2339.021.patch
          1.21 MB
          Andras Piros

          Issue Links

            Activity

              People

              • Assignee:
                andras.piros Andras Piros
                Reporter:
                rkanter Robert Kanter
              • Votes:
                1 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 252h
                  252h
                  Remaining:
                  Time Spent - 132h Remaining Estimate - 32h
                  32h
                  Logged:
                  Time Spent - 132h Remaining Estimate - 32h Time Not Required
                  132h