Minor Description
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in. Luigi is an alternative to Oozie and Azkaban. It is open-sourced here.
At Spotify, we run thousands of jobs through Luigi each day. Foursquare, Bit.ly and more use and contribute to it.
We already have simple smoke tests that tests whether Luigi can
- run workflow locally
- run workflow on the Hadoop cluster
and would be happy to contribute them, if there is an interest and possibility (Luigi is not Apache project ... yet).