[OODT-698] Adapt OODT Process Streaming Data

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1
• Component/s: file manager, resource manager, workflow manager
• Labels: streaming
• Skill Level: Don't Know (Unsure) - The default level
• Assignee: Michael Starch
• Reporter: Michael Starch
• Created: 28/May/14 21:29
• Updated: 15/Jul/17 04:04
• Resolved: 15/Jul/17 04:04

Description

Currently OODT excels at managing data, however; it is not as capable of processing really large data sets. The major drawbacks of OODT are: file-based data storage, and filesystem based io. Both of these drawbacks can be addressed by combining OODT with new stream-processing and cluster management technologies.

This effort is currently focused on combining OODT with the Berkley Data Analysis Stack to achieve these exact results.

Initial designs are captured in the attached slides. Focus on "Track 2" as this was decided to be the best approach.

Sub-Tasks

• Create Mesos Backend for Resource Manager: Resolved
• Create a Spark Runner for the Wengine: Closed
• Create Streaming Data Repository for Streaming OODT: Resolved
• Allow Spark Jobs to Run in Resource Manager: Resolved
• Create a Resource Manager Backend That Muliplexes: Closed
• Move Streaming Components to Seperate Top-Level Component: Resolved
• Create Deployment Scripts Using Python Fabric for Streaming Components: Resolved